In some applications, it’s critical to query and use huge amounts of data on the client. What if you want to visually display and work with a million rows of data? Using AMPS and the WPF DataGrid, you can build applications that do just that: filter and display over a million rows in a few seconds, and keep that display up to date with thousands of changes a second. In this blog post, we’ll show you how to do it.
Prerequisites
Before following the steps included in this document, ensure you have the following prerequisites installed and running:
- AMPS 3.3.1.0 (or greater), available from here.
- An appropriate x86_64 server to run AMPS.
- The AMPS C# client (version 3.2.0.0 or greater), available from here.
- Microsoft Visual Studio 2010
- Python (2.x)
Note: In this post, we’ve used an AMPS server with more than 100GB of memory; if you have less, scale back the initial size from 200 million records to 3 million records.
Sample Files
To get started with this app, you can download the example project here, which contains a zip file of an AMPS configuration, source code, and a utility for loading and updating test data in your AMPS server.
Configure AMPS
An example AMPS configuration file is provided in “amps-config” that matches the expectations of the rest of this example. Modify this configuration as necessary to represent your environment and start an ampServer instance once configured.
Run the Sample Workload
A python client to publish and update ORDERS data is provided in the “testdata” directory, “sample-workload.py”. For best performance in initial load, run sample-workload.py on the same computer as your AMPS instance. To publish 200 million records to the instance running on localhost:
./sample-workload.py init localhost 200After publishing the initial dataset, use the ‘update’ command to begin publishing updates to the base data. To update the 200 million records published before at a rate of 1,000 records per second:
./sample-workload.py update localhost 200 1000Build and Run the OrdView Application
Open the OrdView.sln solution located in this package. Add references from OrdView to the AMPS.Client.dll and AMPS.Client.XAML.dll assemblies located in the “bin” directory where the AMPS C# client is installed. Build and run the application, and you will see a window like the following:
In this window, the top bar contains the AMPS hostname and a filter expression, and the space below is dedicated to the result grid. The status bars below indicates the status of the subscription:
- Time to first: the time elapsed between the beginning of the query and the arrival of the first result row
- Time to last: the time elapsed between the beginning of the query and the arrival of the last result row.
- Rows in view: the number of rows displayed in the grid and the number of rows in the underlying ORDERS topic, once queried.
- Message Rate: the incoming message rate over the last second.
- Peak Rate: the peak message rate achieved during the last query. Change the Hostname to the name of your AMPS server, and type a filter expression that yields results. Press the Enter key (or click the Update button), and after a few moments, you will see a result:
This grid is “live”, and able to reflect thousands of updates every second to the underlying AMPS topic, even on top of a grid containing more than a million records.
Reviewing the Code
The code for this application makes extensive use of the data model classes provided in the AMPS.Client.XAML assembly of the AMPS C# client.
MainWindow.xaml
The XAML for this application is quite simple:
<Window x:Class="OrdView.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:data="clr-namespace:System.Data;assembly=System.Data"
xmlns:amps="clr-namespace:AMPS.Client.Xaml;assembly=AMPS.Client.Xaml"
Title="Orders View" Height="380" Width="524" Loaded="Window_Loaded">
<Window.Resources>
<amps:Subscription x:Key="MySubscription" Filter="1 = 0" Topic="ORDERS"/>
</Window.Resources>
<Grid DataContext="{StaticResource MySubscription}" HorizontalAlignment="Stretch" VerticalAlignment="Stretch">
<Grid.ColumnDefinitions>
<ColumnDefinition Width="Auto"/>
<ColumnDefinition Width="264*"/>
<ColumnDefinition Width="Auto"/>
<ColumnDefinition Width="194*" />
<ColumnDefinition Width="Auto"/>
</Grid.ColumnDefinitions>
<Grid.RowDefinitions>
<RowDefinition Height="Auto"/>
<RowDefinition Height="*"/>
<RowDefinition Height="Auto"/>
<RowDefinition Height="Auto"/>
</Grid.RowDefinitions>
<Label Grid.Column="0">Hostname</Label>
<TextBox Name="_server" VerticalContentAlignment="Center" Grid.Column="1">localhost</TextBox>
<Label Grid.Column="2">Filter</Label>
<TextBox Name="_filter" VerticalContentAlignment="Center" Text="{Binding Filter,UpdateSourceTrigger=Explicit}" Grid.Column="3"/>
<Button Grid.Column="4" Name="Update" Click="Update_Click" IsDefault="True" >Update</Button>
<StatusBar Grid.Row="3" Grid.ColumnSpan="5" HorizontalAlignment="Stretch">
<StatusBarItem>Rows in view:</StatusBarItem>
<StatusBarItem Foreground="Blue">
<Binding Path="TableRowCount"/>
</StatusBarItem>
<StatusBarItem>of</StatusBarItem>
<StatusBarItem Foreground="Blue">
<Binding Path="TopicMatches"/>
</StatusBarItem>
<StatusBarItem>Message Rate:</StatusBarItem>
<StatusBarItem Foreground="Blue">
<Binding Path="RecRate"/>
</StatusBarItem>
<StatusBarItem>Peak Rate:</StatusBarItem>
<StatusBarItem Foreground="Blue">
<Binding Path="PeakRate"/>
</StatusBarItem>
</StatusBar>
<StatusBar Grid.Row="2" Grid.ColumnSpan="5">
<StatusBarItem Foreground="Red">
<Binding Path="Error"/>
</StatusBarItem>
<StatusBarItem>Time to first:</StatusBarItem>
<StatusBarItem Foreground="Blue">
<Binding Path="FirstDT"/>
</StatusBarItem>
<StatusBarItem>Time to last:</StatusBarItem>
<StatusBarItem Foreground="Blue">
<Binding Path="LastDT"/>
</StatusBarItem>
</StatusBar>
<DataGrid Grid.Row="1" Grid.ColumnSpan="5" AutoGenerateColumns="False" ItemsSource ="{Binding Data}" Name="_table" DockPanel.Dock="Top">
<DataGrid.Columns>
<DataGridTextColumn Binding="{Binding Key}" Header="Key"/>
<DataGridTextColumn Binding="{Binding Symbol}" Header="Symbol"/>
<DataGridTextColumn Binding="{Binding Qty}" Header="Quantity"/>
<DataGridTextColumn Binding="{Binding UnitPrice}" Header="Price"/>
</DataGrid.Columns>
</DataGrid>
</Grid>
</Window>In this XAML, after declaring namespace references for the amps namespace, we create an amps:Subscription that connects and subscribe to an AMPS topic. The subscription is initialized with a filter, 1 = 0, that returns no rows, so that the data grid begins empty. Inside the code for the application, the Subscription is bound to a Server when Update is pressed.
Our window uses a Grid layout to lay out the status bars, hostname and filter text controls, update button, and data grid. Each of these controls are bound to properties of the subscription: the filter text box is directly bound to the Filter property, the standard WPF DataGrid to the Data property, and StatusBar items are bound to the various properties of the subscription, to indicate the state of the subscription to the user.
MainWindow.xaml.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using AMPS.Client.Xaml;
using System.Data;
using System.Threading;
using System.ComponentModel;
namespace OrdView
{
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
}
void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e)
{
MessageBox.Show(e.ExceptionObject.ToString(), "Unhandled exception");
}
// Since we have a large query to process, we don't
// want to run queries without an explicit command
private void Update_Click(object sender, RoutedEventArgs e)
{
_filter.GetBindingExpression(TextBox.TextProperty).UpdateSource();
this.Cursor = Cursors.Wait;
var sub = (Subscription)this.Resources["MySubscription"];
if (sub.Server == null || sub.Server.Hostname != _server.Text)
{
try
{
((Subscription)this.Resources["MySubscription"]).Server = new Server
{
Hostname = _server.Text,
Port = 9005,
MessageType = "nvfix"
};
}
catch (Exception ex)
{
this.Cursor = Cursors.Arrow;
MessageBox.Show(ex.Message, "Error", MessageBoxButton.OK, MessageBoxImage.Error);
}
}
this.Cursor = Cursors.Arrow;
}
}
}Most of the logic of this application is implemented declaratively through the XAML, so this part of the application is minimal. In the code here, the handler for the Update button forces an update to the Filter property on the Subscription. Next, the hostname in the text box is compared to the current Server’s hostname, and if they are different, a new Server is constructed and bound to the Subscription. Error handling is minimal; the application displays a message box with the content of any Exception thrown by constructing the Server. Setting the Cursor provides a nicer user experience, since connecting/not connecting to a server may take a few moments.
Summary
In this post, we’ve showed you how you can use AMPS as a large scale view server and build a simple query GUI that’s capable of displaying more than a million rows with thousands of updates per second. In our environment (details below), we were able to query 200 million records inside of AMPS and return and display a million records in under 1.5s. If your application requires a similar low-latency, high-scale user interface, try out this demo, and let us know what you find!
Configuration Details
Operating System:
- Type: Linux x86_64
- Kernel: 2.6.32-5
Processor: * 2 x Intel Xeon E5-2680
Memory: * 256 GB
Storage: * FusionIO ioDrive2 Duo 2.4TB * Driver Ver: 3.2.3
Network: * Intel 1Gb