Select Lists: Data Served Up As You Like It

burrito loaded with ingredients Select Lists is a new feature introduced in our 5.3 release of the AMPS server. This feature lets you declare a subset of fields for your application to receive when querying or subscribing to a topic. AMPS client applications no longer need to receive a full message when the application will only use part of the message. This feature, when combined with existing filter functionality, provides developers with new methods to efficiently manage the volume of data flowing back to their applications from the AMPS server.

Select lists can be used to replace prior work-arounds such as declaring views on top of underlying topics, simply to provide a method to project a subset of fields from the underlying messages, back to the client application. Likewise, if an application declares an aggregated subscription only for the sake of receiving a subset of fields, select lists is an easier and more efficient way to get the same result.

Choose Your Fields

Select lists introduces a new, expressive grammar to declare which fields an application would like to receive from AMPS.

The grammar used to declare a Select List specifies a new options keyword, select= followed by a comma-delimited list of XPath identifiers for fields that you wish to grant or remove access to. You express whether you are granting or removing access by prepending a + or - immediately preceding the XPath identifiers. Examples of each of these tokens are +/included and -/excluded for requesting that AMPS include the /included field and requesting that AMPS exclude the -/excluded field, respectively. For additional details, please refer to the AMPS User Guide.

Building A Better Burrito

Let’s look at a simple example. Suppose a message lists the ingredients of a burrito.

{"id":123, "menu_item": "basic burrito", "green": "lettuce",
           "protein": "chicken", "beans": "pinto", 
           "cheese":"cheddar mix", "topping": "sour cream", 
           "salsa": "chipotle", "tortilla": "flour"}

An application that wanted to know the full ingredients of the burrito could simply do a sow query for a burrito, as follows:

# Assume diner is a connected AMPS Client or HAClient

for m in diner.sow('foods', '/menu_item = "basic burrito"'):
   print m.get_data()

This would return the original message.

An application could use a select list, though, to simply see what protein and salsa is available on a basic burrito:

# Assume diner is a connected AMPS Client or HAClient

for m in diner.sow('foods', '/menu_item = "basic burrito"',
                   options='select=[-/,+/protein,+/salsa]'):
   print m.get_data()

This query explicitly removes all fields from the message with the directive -/, then adds back the protein and salsa fields. This select list produces the following result:

{ "protein": "chicken", "salsa": "chipotle" }

Notice that the message contains exactly the fields requested in the select list. No other information is included – not even the key field for the SOW or the field that the query filters on.

Following this pattern, you could also easily select other versions of the basic burrito. For example, you could make a vegan burrito:

# Assume diner is a connected AMPS Client or HAClient

# Vegan burrito
for m in diner.sow('foods', '/menu_item = "basic burrito"',
                   options='select=[-/protein,-/cheese,-/topping]'):
   print m.get_data()

{"id":123, "menu_item": "basic burrito", "green": "lettuce",
           "beans": "pinto", "salsa": "chipotle",
           "tortilla": "flour"}

You could also select a burrito without any beans or tortilla:

# Assume diner is a connected AMPS Client or HAClient

# Low carb burrito
for m in diner.sow('foods', '/menu_item = "basic burrito"',
                   options='select=[-/beans,-/tortilla]'):
   print m.get_data()

Keeping Result Sets Lean

Dashboard client applications are a perfect fit to use Select Lists in concert with paginated subscriptions, to optimize as responsive a UI as possible!

The following example highlights how a dashboard client application would query a quotes topic using a paginated sow_and_subscribe command to request 10 messages at a time and dynamically selecting the subset of fields it would like to receive. Before select lists, AMPS would either require the user to receive all fields from the topic (possibly many more than required), create a View that would project the desired subset of fields to be returned, or use dynamic aggregation (which created and calculated an aggregation, even when the only result was to provide a few fields.

For example, the dashboard might simply return the ticker, price and quantity fields while retrieving 10 messages at a time:

# Assumes dashboard is a connect AMPS Client or HAClient

dashboard.sow_and_subscribe('quotes', order_by='/ticker', \
          options='top_n=10,skip_n=10,select=[-/,+/ticker,+/price,+/qty]')

Securing Data with Select Lists

Select Lists have been built-in to all aspects of AMPS. Not only are subscribing clients able to specify a Select List for fields to be returned, but AMPS also provides select lists as part of the entitlement system. These select lists can be use to prevent users from retrieving or querying certain fields within a topic.

Below are some example Entitlement Select Lists that to demonstrate how they would be declared:

{ "userId": 456,
  "userName": "John Investor",
  "password": "NeverGonnaGuess",
  "secret": "Boston Red Sox fan",
  "optional": "empty"
}

In this example, the Entitlement Select List starts off by implicitly granting full access to all fields (explicitly, would specify +/) then specifying the removal of access to the /password and /secret expressions for those specific fields.

-/password,-/secret

For the message above, the result would be that the user would be able to see the userId, userName, and optional fields. Even better, if the user provides a filter that provides either the password field or the secret field, those fields will always be treated as NULL – exactly the same as if they did not exist in the original message.

Another way to use entitlement select lists is to configure an explicit removal of access to all fields, followed by granting of individual entitlement to specific fields. This expression would grant entitlement to the following fields: userId and userName.

-/,+/userId,+/userName

In this case, no matter what other fields were present in the message, this user would only be able to see the userId and userName fields. For that user, it is as though no other fields exist.

Choose Your Own Data (or Toppings)

As shown above, select lists are one more feature to help applications reduce bandwidth usage and processor consumption. Just like ordering a burrito, you can add or remove data as your application needs. Of course, also like ordering a burrito, you can only choose from the data available – you can’t order mango salsa if there’s no mango salsa available. So, for a basic burrito, you can include or remove an ingredient, but you can’t change the value of an ingredient. (If you do need to change data in a message, you can use a view, an aggregated subscription, or enrichment/preprocessing to do that, depending on your exact needs.)

With the integration into the entitlement system, select lists also provide another facet of an overall strategy for keeping data secure in AMPS. Combined with entitlement filters and the extensibility of AMPS user-defined functions, AMPS has extremely fine-grained control over access to data.

How do you plan to use select lists? How much bandwidth and processing power will you save? Let us know in the comments!

[Edited on 5/23/2019 to fix typos in text and code snippets.]