Unions

Unions let the server maintain a real-time, read-only SOW topic that reflects the combined contents of one or more existing SOW topics. AMPS subscribes to the underlying topic (or set of topics when you provide a regular expression) and immediately mirrors every insert, update, and delete into the union. Applications can then query or subscribe to the union just as they would to any other SOW topic—without having to maintain their own fan-in or conflation logic.

Use a Union when you need to:

• Present a consolidated “all customers” or “all regions” view that spans many SOW topics.
• Provide a stable topic name for downstream services even though the upstream data is spread across multiple SOW topics.
• Consolidate the output of a set of Views separately computing slices of an input topic.

Configuring a Union

Define each union inside either the <TopicMetaData> section or the <SOW> section of the AMPS configuration, using a <Union> element. The following configuration items are available:

Name (required)

Defines the name clients use when issuing SOW commands to the union. Provide the Name element (or Topic, which AMPS accepts for backward compatibility).

UnderlyingTopic (required)

Specifies the source topic or topics that feed the union. This value can be a literal SOW topic name or a regular expression. When you reference a topic defined with Pattern, provide the topic’s Name. AMPS subscribes to every SOW topic that matches this value and mirrors the records into the single union defined by Name.

MessageType (required)

Sets the message type shared by the union and its underlying topics. The union must use the same message type as the sources it unions.

HugePages

Enables huge page allocation for the union’s SOW store when the operating system supports huge pages.

HashIndex

Adds one or more hash indexes so queries against the union can match on fields without rescanning the entire topic. Provide a HashIndex element that contains one Key element per field in the index, just as you would for any SOW topic.

Example

<SOW>
  <Topic>
    <Name>/customers/001/orders</Name>
    <MessageType>json</MessageType>
    <Key>/order/id</Key>
  </Topic>
  <Topic>
    <Name>/customers/002/orders</Name>
    <MessageType>json</MessageType>
    <Key>/order/id</Key>
  </Topic>

  <Union>
    <Name>/orders/all</Name>
    <MessageType>json</MessageType>
    <UnderlyingTopic>^/customers/.+/orders$</UnderlyingTopic>
  </Union>
</SOW>

In this example, AMPS subscribes to every per-customer orders topic that matches the regular expression and mirrors the records into /orders/all. Applications can issue SOW queries against /orders/all to see the complete order book, or they can continue to query the individual per-customer topics.

Operating and Monitoring Unions

• Queries and subscriptions: Use the standard SOW verbs (sow, delta, sow_and_subscribe, and so on) against the union name. Deletions in the underlying topic are propagated as SOWDelete events in the union.
• Regex foundations: When the UnderlyingTopic value is a regular expression, AMPS subscribes to every matching topic and writes the combined results into the single union named by Name. Clients do not need to guess the per-topic naming scheme.
• Replication: Like Views and ConflatedTopics, Unions are derived data and are not replicated between instances. To reproduce the same union elsewhere, configure the union on each instance or replicate the underlying SOW topics and rebuild the union locally.
• Statistics: Unions do not emit dedicated statistics. Monitor the underlying SOW topics metrics to track performance.
• Resource planning: Because a union stores a full copy of the underlying data set, size the SOW directory to accommodate another full topic plus any indexes you add. Apply the same retention and checkpoint policies you use for other SOW topics.

Unions provide a server-managed way to create consolidated datasets without rewriting applications or adding per-client consolidation logic.