AMPS 5.2 introduces a powerful new capability for subscribers to create custom aggregations and projections to AMPS SOW topics – with no configuration necessary! We call this functionality Aggregated subscriptions. Aggregated subscriptions are like private views for an individual subscription. You no longer have to reconfigure and restart AMPS to test a different calculation, or add a full view for a subscriber that needs different data – but only for a few days at the close of the month. When a subscriber has unique needs, aggregated subscriptions can give that subscriber a unique view.

Aggregated Subscriptions can be used with any command that queries a State of the World topic (for those of you familiar with AMPS, this includes the sow, sow_and_subscribe, and sow_and_delta_subscribe commands.)

To use Aggregated Subscriptions, configure one or more SOW topics on your AMPS instance, for example:

<SOW>
  <Topic>
    <Name>Orders</Name>
    <MessageType>json</MessageType>
    <Key>/order_key</Key>
    ...
  </Topic>
</SOW>

No additional configuration is required to support aggregated subscriptions; any topic in the SOW may be used with these options.

Aggregated Subscriptions specify a set of grouping fields and a set of projection fields when placing the subscription or issuing the SOW query. These serve the same purpose as the Grouping and Projection elements in the AMPS configuration when defining a View. However, instead of specifying these fields in a server configuration file, you provide these options through the AMPS Client you use, in the options field of the command.

The AMPS command-line tool spark supports providing an options field via the -opts argument, so we can use spark to quickly test new aggregations.

Examples

Suppose our Orders topic above has been seeded with a few sample messages:

{"order_key":1, "symbol":"MSFT", "price":62.30, "qty":100}
{"order_key":2, "symbol":"MSFT", "price":62.28, "qty":150}
{"order_key":3, "symbol":"IBM", "price":180.20, "qty":16}
{"order_key":4, "symbol":"FIZZ", "price":61.77, "qty":4000}
{"order_key":5, "symbol":"YUM", "price":64.07, "qty":123}

We can use the sow command with projection and grouping options to ask for custom aggregations to be built and returned. Suppose we’d like to know the average order price for each symbol, for example. We can use the command line utility spark to easily execute this query:

$ ~/spark sow -server localhost:9007 -topic Orders -opts "projection=[/symbol,avg(/price) as /avg_price],grouping=[/symbol]"                                                   
{"symbol":"FIZZ","avg_price":61.77}
{"symbol":"YUM","avg_price":64.07}
{"symbol":"MSFT","avg_price":62.29}
{"symbol":"IBM","avg_price":180.2}

Note the syntax of the projection and grouping options in the -opts argument. Both options take a list of fields. For the grouping option, this is a list of one or more fields you’d like to group your results by. The list of fields in projection is more flexible, and allows you to simply project a field through (e.g. /symbol), or use AMPS SQL-like syntax to compute a value you’d like projected (e.g. avg(/price) as /avg_price).

Customizing Output

The projection syntax allows us to do arbitrary computation and to call User Defined Functions as well. Imagine we’d like to compute and return the average order total by symbol, for example:

$ ~/spark sow -server localhost:9007 -topic Orders -opts "projection=[lower(/symbol) as /symbol,avg(/price*/qty) as /avg_total],grouping=[/symbol]" -orderby "/avg_total desc"
{"symbol":"fizz","avg_total":247080.0}
{"symbol":"yum","avg_total":7880.61}
{"symbol":"msft","avg_total":7786.0}
{"symbol":"ibm","avg_total":2883.2}

In this example we use an AMPS built-in function lower to convert the symbol names to lowercase; we also average on the order’s price multiplied by the order’s quantity, and sort the results on this new /avg_total field using -orderby.

Subscriptions

In addition to a one-time query, aggregated subscriptions can be placed which allows your application to see ongoing updates to the results of the aggregation as changes to underlying data arrive. For fast-moving underlying data, this may be combined with subscription conflation to reduce update frequency.

As an example, imagine we place this subscription in one session:

$ ~/spark sow_and_subscribe -server localhost:9007 -topic Orders -opts "projection=[lower(/symbol) as /symbol,avg(/price*/qty) as /avg_total],grouping=[/symbol],conflation=5s" -orderby "/avg_total desc"
{"symbol":"fizz","avg_total":247080.0}
{"symbol":"yum","avg_total":7880.61}
{"symbol":"msft","avg_total":7786.0}
{"symbol":"ibm","avg_total":2883.2}

spark keeps running, listening for more data. Our use of conflation=5s means AMPS will conflate messages it might send us on a 5 second interval. In another window, we quickly publish 4 new Orders for YUM:

{"order_key":10,"symbol":"YUM","price":70,"qty":10000}
{"order_key":11,"symbol":"YUM","price":70,"qty":8000}
{"order_key":12,"symbol":"YUM","price":70,"qty":9000}
{"order_key":13,"symbol":"YUM","price":70,"qty":7000}

Because we’ve specified conflation=5s, we see just one additional message published to our subscriber, a few seconds later:

{"symbol":"yum","avg_total":477576.122}

In addition to conflation, aggreagated subcscriptions may be combined with both content filters and with delta subscriptions to even further reduce the amount of data your subscriber must process.

Aggregated subscriptions are unique to each client, even when they contain the same projection fields and grouping clause, so note that additional system resources are used for each client that requests an ongoing aggregation subscription. The resources used by a client’s aggreagations count against the configurable byte limits for a client; a client may be disconnected by the server if this exceeds the configured limit.

Conclusion

AMPS 5.2’s Aggregated Subscriptions make AMPS much more flexible and allows you to build more responsive, customizable applications. This feature allows you to build richer experiences for your users, and to make changes to the aggreagations you offer without reconfiguring AMPS. For more information on these new capabilites, consult the User Guide.