# 60East Technologies > 60East Technologies provides the highest-performance real-time streaming analytic database in the industry. This file contains all documentation content in a single document following the llmstxt.org standard. # AMPS Command Reference This guide includes a listing of all AMPS commands as well as the required and optional parameters. AMPS supports a consistent set of commands and options regardless of the protocol used to communicate with AMPS. This guide covers the semantics of the commands and options, but does not cover how those commands and options are represented in any particular protocol. Each protocol uses a different concrete format for messages, and that format is specific to the protocol. To use a command from your application, set the properties of the Command object as shown in this guide, then use the execute function to send the command. The AMPS client is responsible for interpreting the command and formatting the message to AMPS in the proper format for the specific protocol the client is using. This guide lists the available options and their usage. When using the AMPS client libraries (either with the named convenience methods or with the `Command` interface), your application only needs to specify a subset of options. The AMPS client itself will handle formatting the command, adding options necessary for submitting the command (such as requesting a response indicating if a subscription succeeded, and so forth). The Developer Guide for each language provides a cookbook for common commands to AMPS, and indicates which fields and options to set for those commands. :::tip The AMPS client libraries handle many of the details for an application, such as handling heartbeating, requesting acknowledgment on commands, and so on. The Developer Guides for the AMPS client libraries include a reference that lists the fields and options to set to produce a specific result. ::: --- # Commands to AMPS This section describes messages that applications send to AMPS. The AMPS Client libraries handle the details of formatting the message and sending it to AMPS. The application is only responsible for setting the necessary fields for the command to AMPS. This can be done either by providing parameters to a named method (such as `subscribe()`) or by creating a `Command` object, setting the parameters on that object, and then calling `execute()` or `executeAsync()/execute_async()` on the client object. --- # flush command Sends a command to AMPS that returns an acknowledgment when all previous commands from this client have been processed. This command helps applications that use AMPS determine when AMPS has received all of the messages that have been sent, making it safe for the client to exit. :::tip Notice that the `publishFlush` command available in the AMPS client libraries includes additional logic (when a `PublishStore` is present) to ensure that messages published to AMPS have been safely persisted. ::: ## Header Fields The following table contains the header fields available to a `flush` command. | Field | Description | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

The command to be executed.

Value: `flush`

| | `client_name` | A string identifier used to give a client a unique ID. | | `ack_type` |

Acknowledgment type for the given command.

Value is a comma separated list of one or more of the following: `none`, `completed` or `processed`.

| ## Returns A `flush` message specifying an `AckType` of `completed` or `processed` will receive an `ack` message when all previous messages from this client have been processed by AMPS. The following table contains the acknowledgment messages that can be returned by a `flush` command. | Acknowledgment | Description | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | | `none` |

No `ack` message is returned.

This is the default behavior.

| | `completed` | All previous commands have been processed by AMPS. | | `persisted` | All previous commands from this publisher have been persisted by AMPS (or have failed and produced the requested acknowledgments). | | `processed` | AMPS has processed the `flush` message. | | `received` | The `flush` command has been received. | | `stats` | Not supported at this time. | --- # heartbeat command Sends a command to AMPS that starts or refreshes a `heartbeat` timer. When a heartbeat timer is active, AMPS publishes periodic `heartbeat` messages to AMPS and expects the client to respond with a `heartbeat` message. If the client does not provide a `heartbeat` within the time specified, AMPS logs an error and disconnects the connection. The AMPS client libraries automatically manage creating and returning heartbeats. It is rarely, if ever, necessary for an application to manage heartbeats. See the _Developer Guide_ for the client library you are using for details. ## Header Fields The following table contains the header fields available to a `heartbeat` command. | Field | Description | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

The command to be executed.

Value: `heartbeat`

| | `opts` |

Specifies whether this command starts the timer or refreshes the timer.

Valid options are:

| ## Returns The `heartbeat` message does not typically request an acknowledgment, and therefore does not receive a response. The command can, however, request acknowledgments as listed below. | Acknowledgment | Description | | ------------------ | ------------------------------------------- | | `none` | Not supported at this time. | | `completed` | Not supported at this time. | | `parsed` | Not supported at this time. | | `persisted` | Not supported at this time. | | `processed` | AMPS has processed the `heartbeat` message. | | `received` | AMPS has received the `heartbeat` message. | | `stats` | Not supported at this time. | --- # logon command To help identify clients and users, it is recommended that clients send a `logon` command to the AMPS engine and specify a client name. By default, AMPS requires a `logon` command as the first command when a client connects. AMPS only allows a single `logon` command for each connection. The `logon` command must be the first command sent over a new connection. Otherwise, AMPS performs an implicit `logon`, causing any other `logon` commands for the connection to be rejected. In AMPS configurations where authentication is enabled, all connecting clients must issue a `logon` message with the `username` and `password` credentials specified in the command. Attempts to logon to an AMPS instance that do not contain the information required will be rejected and prohibited from issuing further commands until a successful `logon` has been placed. If an AMPS client is connected to an instance that has a transaction log enabled, the `ClientName` specified _must_ be unique for the instance. Only one client with the same name is allowed to connect to the instance at a given time. If an application logs on with the same `ClientName` and authenticated user name as an existing connection, AMPS assumes that the new logon is a reconnection from the existing connection and disconnects the existing connection. If an application logs on with the same `ClientName` as an existing connection but a different authenticated user name, the new logon will fail. It is recommended that all `logon` commands request that a `processed` acknowledgment message be requested in the `AckType` header of the `logon` message. This will allow AMPS to communicate the result of the `logon` command to the client, allowing the client to determine how to best proceed. :::info The `websocket` protocol uses a different mechanism that contains the same information rather than sending a `logon` command. ::: ## Header Fields The following table contains the header fields available to a `logon` command. | Field | Description | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

The command to be executed.

Value: `logon`

| | `client_name` |

A string identifier used to give a client a unique ID. AMPS does not limit the character set used in this name. However, the specific protocol may have character set limitations.

60East recommends that the client name is meaningful, short, human readable, and avoids using control characters, newline characters, or square brackets.

The client name must be unique across the set of instances that share messages through replication. Otherwise messages may be lost since the client name is included in the process AMPS uses to form the unique identifier for messages published to AMPS.

| | `ack_type` |

Acknowledgment type for the given command.

Value is a comma separated list of one or more of the following: `none`, `received` or `processed`.

| | `message_type` |

The message type for the connection.

Required if the `Transport` accepts any message type.

| | `user_id` | The username passed into the AMPS authentication and entitlement module. | | `password` | The password passed into the AMPS authentication and entitlement module. | | `correlation_id` |

A user-provided string that will be included in the log message recording this logon, and in the information provided for the connection in the administration interface.

AMPS does not interpret this string or use the string for any other purpose. If this header is not present, AMPS does not store a value for the correlation id for this connection. The contents of this header must consist of characters that are legal in Base64 encoding.

| | `version` | The client library version (typical includes the language and build number) of the client making the connection. This version number is logged for the connection, but does not otherwise affect the connection. | ## Returns A `logon` message specifying an `AckType` of `received` or `processed` will receive an `ack` message to acknowledge the message receipt. If a client requests an acknowledgment message, the header will also contain the `ClientName` which was part of the original `logon` message. When requested, the `logon` command will result in a `processed` acknowledgment message. This returned acknowledgment is used in determining if a client was successfully authenticated against a server which has an authentication module enabled. The following table contains the acknowledgment messages that can be returned by a `logon` command. | Acknowledgment | Description | | ------------------ | ------------------------------------------------------------------------------------------- | | `none` |

No `ack` message is returned.

This is the default behavior.

| | `completed` | Not supported at this time. | | `persisted` | Not supported at this time. | | `processed` | AMPS has processed the `logon` message. | | `received` | The `logon` command has been received. | | `stats` | Not supported at this time. | ### Options Field The following table contains a list of the `Options` available and their definitions when used in the AMPS `logon` command. | Option | Description | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `none` | This is the default `Options` type. | | `ack_conflation=interval` |

When provided, the interval at which AMPS conflates `persisted` acknowledgment messages for publishes sent on this connection.

AMPS conflates `persisted` acknowledgment messages when a transaction log is configured for the instance.

Default: `1s` if no option is provided.

| | `pretty` | When provided and set to `true`, AMPS returns a formatted representation of the contents of the built-in binary message types rather than the original data. | --- # Publishing ## Publishing to AMPS This section describes commands used to send data to AMPS. Notice that, for topics in the state of the world, AMPS treats these commands as inserting a new message if one does not already exist, and updating the existing message if a message already exists. For topics in the transaction log, each `publish` or `delta_publish` will store the current message in the transaction log, just as it would be delivered to subscribers. The table below lists the commands used to send messages to AMPS. | Command | Usage | | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`publish`](publishing/publish) | Send a message to AMPS. | | [`delta_publish`](publishing/delta-publish) |

Send a partial message to AMPS. If a previous state for the message exists in the SOW `Topic` this message is published to, update the fields that are not NULL in this message, ignore fields that are NULL (including missing fields) in this message.

For publishes to topics that are not defined as a SOW `Topic`, there is no difference between this command and `publish`.

| --- # delta_publish command The `delta_publish` command is a way of publishing an incremental update to a record. If a client uses `delta_publish` to publish an update, AMPS first extracts the key fields from the record and does a look up for the record in the SOW. AMPS will then apply the update to the SOW record overwriting any non-key field that has a value in the update and appending to the message any new fields that were not previously in the SOW message. If `delta_publish` is used on a record that does not currently exist in the SOW or if it is used on a topic that does not have a SOW topic store defined, then `delta_publish` will behave like a standard `publish` command. A `delta_publish` is transparent to other clients and the merged record will be forwarded to matching subscriptions. ## Header Fields The following table contains the header fields available to a `delta_publish` command. | Field | Description | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

Command to be executed.

Value: `delta_publish`

| | `topic` | The SOW topic to publish the message to. | | `ack_type` | Acknowledgment type for the given command. Value is a comma separated list of one or more of the following: `none`, `received`, `processed`, `completed` or `stats`. | | `cmd_id` | If specified with an AMPS command which requests an acknowledgment message, all non-conflated requested acknowledgment messages will contain the `command_id` in the `ack` response header. Notice that AMPS may conflate successful `persisted` acknowledgments for publishes. | | `expiration` |

An interval used to define the lifetime of a `delta_message` message.

Time period is in seconds.

| | `seq` | A monotonically increasing number used to identify published messages in a high availability environment. | | `correlation_id` | A user-provided string that will be passed, verbatim, to subscribers. If this header is not present, subscribers receive no value for the `correlation_id`. The contents of this header must consist of characters that are legal in Base64 encoding. | | `sow_key` | For SOW topics that use an explicit key, the SOW key to use for the message. The contents of this header must consist of characters that are legal in Base64 encoding. | ## Returns For a `delta_publish` message, AMPS will send acknowledgment messages for the following `AckType` fields: `received`, `processed` and `persisted` along with a populated `Status` header field describing the acknowledgment. The following table contains the acknowledgment messages that can be returned by a `delta_publish`. In general, 60East recommends only requesting `persisted` acknowledgments for `delta_publish` commands, since these acknowledgments can be conflated when commands succeed. | Acknowledgment | Description | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `none` |

No `ack` message is returned.

This is the default behavior.

| | `completed` |

AMPS has processed the message and, if necessary enqueued the message for persistence and replication.

This acknowledgment includes the steps acknowledged by the `processed` acknowledgment.

| | `persisted` |

When AMPS returns an acknowledgment message of `persisted`, it guarantees that:

  1. All downstream synchronous replications have acknowledged that the message(s) have been delivered to their respective SOW Topic(s).
  2. When the publish message has been sent to all available downstream asynchronous replications.

If no replication is configured for this topic, this acknowledgment applies only to local persistence.

If no persistence is configured for this topic, this acknowledgment is still provided when requested.

AMPS may conflate successful persisted acknowledgments to acknowledge multiple publishes at once. In this case, the `seq` provided means that the message with that sequence number, and all previous messages from this publisher, have been acknowledged.

| | `processed` |

AMPS has processed the message(s) to be published to the SOW. Any errors which occur in the message will be returned to the client in this acknowledgment message.

This includes processing entitlement checks, and parsing the message but does not guarantee that the message has been replicated or persisted.

| | `received` | The `delta_publish` message has been received. | | `stats` | Not supported at this time. | ## Errors Any errors that occur while processing this command will be returned in the status of a `processed` acknowledgment message and logged to the log file. Regardless of success or failure, the `processed` acknowledgment message will be returned only if requested by specifying `processed` in the `AckType` field. Errors that occur during the processing command or after processing the command will be returned in the status of a `persisted` acknowledgment message if one is requested. This acknowledgment is returned at the point that the message can be considered safely persisted in AMPS. Notice that AMPS may conflate successful `persisted` acknowledgments for publishes. See the AMPS User Guide for details. --- # publish command The `publish` command is the primary way to insert messages into the AMPS processing stream. A `publish` command received by AMPS will be forwarded to other connected clients with matching subscriptions. Depending on the server configuration, a `publish` command may also be used to update the State of the World, written to the transaction log, replicated to other instances, and so on. In AMPS, the publisher is only responsible for delivering the message to AMPS reliably. The AMPS server configuration manages routing the message to subscribers, determining if the message should be persisted, updating any state that the message affects, and so on. ## Header Fields The following table contains the header fields available to a `publish` command. | Field | Description | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

Command to be executed.

Value: `publish`

| | `topic` | The topic to publish the message to. | | `ack_type` |

Acknowledgment type for the given command.

Value is a comma separated list of one or more of the following: `none`, `received`, `persisted` or `processed`.

| | `cmd_id` | If specified with an AMPS command requesting an acknowledgment message in response to the `publish` command, all requested acknowledgment messages will contain the `CommandId` in the response header. | | `expiration` | An interval in seconds, used to define the lifetime of a `publish` message. | | `seq` | A monotonically increasing identifier used in high availability configurations to determine message uniqueness across replicas. | | `correlation_id` | A user-provided string that will be passed, verbatim, to subscribers. If this header is not present, subscribers receive no value for the `correlation_id`. The contents of this header must consist of characters that are legal in Base64 encoding. | | `sow_key` | For SOW topics that use an explicit key, the SOW key to use for the message. The contents of this header must consist of characters that are legal in Base64 encoding. | ## Returns A client which issues a `publish` can request a `processed` acknowledgment message; however this is not recommended as there is a significant performance overhead associated with this. The following table contains the `AckType` messages which can be returned by a `publish`. In general, 60East recommends only requesting `persisted` acknowledgments for `publish` commands, since these acknowledgments can be conflated when commands succeed. | Acknowledgment | Description | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `none` |

No `ack` message is returned.

This is the default behavior.

| | `completed` |

AMPS has processed the message and, if necessary, enqueued the message for persistence and replication.

This acknowledgment includes the steps acknowledged by the `processed` acknowledgment.

| | `persisted` |

When AMPS returns an acknowledgment message of `persisted`, it guarantees that:

  1. All downstream synchronous replications have acknowledged that the message(s) have been delivered to their respective SOW Topic(s).
  2. When the `publish` message has been sent to all available downstream asynchronous replications.

If no replication is configured for this topic, this acknowledgment applies only to local persistence.

If no persistence is configured for this topic, this acknowledgment is still provided when requested.

AMPS conflates successful persisted acknowledgments to acknowledge multiple publishes at once. In this case, the `seq` provided means that the message with that sequence number, and all previous messages from this publisher, have been acknowledged.

| | `processed` |

AMPS has processed the `publish` message. Any errors which occur while processing will be returned to the client in this acknowledgment message.

This includes processing entitlement checks, and parsing the message if parsing is necessary, but does not guarantee that the message has been replicated or persisted.

| | `received` | The `publish` message has been received. | | `stats` | Not supported at this time. | ## Errors Any errors that occur while processing this command will be returned in the status of a `processed` acknowledgment message and logged to the log file. Regardless of success or failure, the `processed` acknowledgment message will be returned only if requested by specifying `processed` in the `AckType` field. Errors that occur during the processing command or after processing the command will be returned in the status of a `persisted` acknowledgment message if one is requested. This acknowledgment is returned at the point that the message can be considered safely persisted in AMPS. Notice that AMPS may conflate successful `persisted` acknowledgments for publishes. See the [AMPS User Guide](/docs/amps-user-guide) for details. --- # Subscribing to and Querying Topics This section describes commands used to retrieve content messages from AMPS, including both queries and subscriptions. For details on the differences between these commands, see the _AMPS User Guide_. | Command | Usage | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `subscribe` | Receive a stream of messages from a topic. | | `sow` | Query a snapshot of the values within a topic in the State of the World (SOW). | | `sow_and_subscribe` | Query a snapshot of the values within a topic in the SOW and begin a subscription atomically at the point of query. | | `delta_subscribe` | Receive a stream of messages from a topic. If the topic is in the SOW, and it is possible to construct a partial message, receive only fields that have changed (fields that identify the message are also included by default). | | `sow_and_delta_subscribe` | Query a snapshot of the values within a topic in the SOW and begin a subscription atomically at the point of query. If the topic is in the SOW, and it is possible to construct a partial message, receive only fields that have changed (fields that identify the message are also included by default). | --- # delta_subscribe command The `delta_subscribe` command is like the `subscribe` command except that subscriptions placed through `delta_subscribe` will receive only messages that have changed between the SOW record and the new update. If `delta_subscribe` is used on a topic which does not have a SOW store defined, then `delta_subscribe` behaves like a `subscribe` command. ### Header Fields The following table contains the header fields available to a `delta_subscribe` command. | Field | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

Command to be executed.

Value: `delta_subscribe`

| | `topic` | Topic with which to place a subscription. | | `ack_type` |

Acknowledgment type for the `delta_subscribe` command.

Value is a comma separated list of one or more of the following: `none`, `received`, `processed`, `completed` or `stats`.

| | `cmd_id` | If specified with an AMPS command requesting an acknowledgment message, all requested acknowledgment messages will contain the `CommandId` in the acknowledgment response header. | | `data_only` |

A Boolean (`true` or `false`) used to determine the type of data sent to the subscriber.

A value of `true` will, for example, not include a SOAP envelope.

| | `filter` |

String which is used as a content filter expression.

When using XML, the filter must be wrapped in a `CDATA`.

| | `opts` |

A comma separated list of flags available to the `subscribe` command.

The [Options Field](#options-field) table below describes the `Options` available for use in the `delta_subscribe` command.

| | `send_empty` |

Boolean (`true` or `false`) value used to determine whether empty messages which are published will be forwarded to matching subscriptions.

Default: `true`

| | `send_matching_ids` | Boolean (`true` or `false`) subscription identifiers will not be sent for all matched messages if set to `false`. | | `sub_id` |

The subscription ID for this command. When provided with a new subscription, this is the identifier that AMPS will use for the subscription.

When provided with the `replace` option, this field specifies the subscription to replace.

When provided with a `pause` or `resume` option, this field specifies the subscriptions to pause or resume.

For a new subscription, the AMPS clients will generate a subscription ID if one is not provided.

| ### Options Field The following table contains a list of the `Options` available and their definitions when used in the AMPS `sow_and_delta_subscribe` command. | Option | Description | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `none` | This is the default `Options` type. | | `bookmark` |

Specifies that the subscription should return bookmarks, if available, on each publish message. Bookmarks are only available if the topic that is subscribed to is recorded in the transaction log.

Notice that this option does not set the starting point for a bookmark subscription: use the `Bookmark` header for that purpose.

This option is not required to receive bookmarks for a bookmark subscription. Those subscriptions always include bookmarks on each publish message.

| | `conflation=n` |

Specifies whether to conflate this subscription.

The value provided can be a time interval, `auto`, or `none`.

When present and set to a value other than none, enables conflation for the subscription.

Can also be set to `auto`, which requests that AMPS attempt to determine an appropriate conflation interval based on client consumption.

Recognizes the same time specifiers used in the AMPS configuration file (for example, `100ms` or `1s` or `1m`).

Default: `none`

| | `conflation_key=[key]` |

When conflation is enabled, specifies the fields to use to determine message uniqueness.

The format of this option is a comma-delimited list of XPath identifiers within brackets.

For example, to conflate based on the value of the `/tickerId` and `/customerId` within a message, the value of this option would be:

`[/tickerId,/customerId]`

Defaults to the SOW key fields for SOW topics.

No default for non-SOW topics. This option is required for non-SOW topics.

This option is not valid with the `oof` option unless the keys provided are identical to the keys for the topic.

| | `live` |

Tells AMPS to send messages to subscribing clients before they have been persisted to the transaction log.

This option has no effect on subscriptions that are not replays from the transaction log (that is, this option only applies to bookmark subscriptions).

| | `max_backlog=n` |

When subscribing to a queue, the number of unacknowledged messages the client is willing to accept at a time.

AMPS will not exceed this number, but may choose a smaller number depending on the queue configuration.

| | `no_empties` |

Tells AMPS not to send empty publish messages to matching subscriptions.

This can be useful for suppressing messages where no fields have changed.

| | `no_sowkey` | Tells AMPS not to send the AMPS-generated SowKey for messages. | | `non_regex_topic` | Specifies that the topic name should be a literal match, even if the topic name contains regular expression characters. | | `oof` |

Send an OOF message for records which have fallen out of focus from the original subscription.

When focus tracking is enabled, AMPS will also deliver the full message to a subscription when a previously out-of-focus message comes into focus.

| | `pause` |

Pause a bookmark subscription.

This option is only valid for bookmark subscriptions that do not use the `live` option. When this option is present, AMPS pauses the subscription or subscriptions provided in the `SubId` of the command.

| | `rate=n` |

Set the maximum message delivery rate for a bookmark subscription.

This option is only valid for bookmark subscriptions that do not use the `live` option.

The rate can be specified as either the number of messages per second (for example, `1000`), the number of bytes per second (for example, `100KB`), or a multiple of the original replay rate (for example, `1.5X`).

| | `replace` | Replace the subscription associated with `SubId` with another subscription. When provided as part of `sow_and_subscribe`, AMPS runs a SOW query for the new subscription. | | `resume` |

Resume a bookmark subscription. This option is only valid for bookmark subscriptions that do not use the `live` option.

When this option is present, AMPS resumes the subscription or subscriptions provided in the `SubId` of the command.

| | `send_keys` |

AMPS will send the SOW keys fields back with messages from the SOW.

Notice that without this option, messages will never contain these fields (since, by definition, they do not change from update to update).

| | `timestamp` | AMPS will include a header with the time at which this instance of AMPS processed the incoming publish command for this message. | ### Returns A `delta_subscribe` command returns the following command types: | Command | Description | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `publish` | A message published to AMPS. | | `oof` | Returned when the `oof` option is requested on the command, the subscription is to a topic in the SOW, and the subscription is not a bookmark subscribe. | | `ack` | Acknowledgments requested, as described in the following section. | For a `delta_subscribe` message, AMPS will send acknowledgment messages for the following `AckType` fields: `received`, `processed`, `persisted` and `stats` along with a populated `Status` header field describing the acknowledgment. The following table contains the `AckType` messages which can be returned by a `delta_subscribe`. | Acknowledgment | Description | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `none` |

No `ack` is returned.

This is the default behavior.

| | `completed` | When a bookmark is present on the subscribe request and this acknowledgment is requested, AMPS sends a `completed` acknowledgment message to indicate that bookmark replay is complete. Further messages on this subscription are from new publishes. | | `persisted` | When a bookmark is present and this acknowledgment is requested, AMPS periodically sends a `persisted` acknowledgment message to indicate the most recent bookmark in the server's transaction log. | | `processed` | AMPS has compiled the filters for the `delta_subscribe` message(s). | | `received` | The `delta_subscribe` message has been received. | | `stats` | Returns an acknowledgment message with `Matches`, `TopicMatches` and `RecordsReturned`. | ### Errors Any errors that occur during this command will be returned in the status of a `processed` acknowledgment and logged to the log file. Regardless of success or failure, the `processed` acknowledgment will be returned only if requested by including processed in the `AckType` field of the `delta_subscribe` message header. --- # sow_and_delta_subscribe command A `sow_and_delta_subscribe` command is used to combine the functionality of commands `sow` and a `delta_subscribe` in a single command. The `sow_and_delta_subscribe` command is used: (a) to query the contents of a SOW topic (this is the `sow` command); and (b) to place a subscription such that any messages matching the subscribed SOW topic and query filter will be published to the AMPS client (this is the `delta_subscribe` command). As with the `delta_subscribe` command, `publish` messages representing updates to SOW records will contain only the information that has changed. If a `sow_and_delta_subscribe` is issued on a record that does not currently exist in the SOW topic, or if it is used on a topic that does not have a SOW-topic store defined, then a `sow_and_delta_subscribe` will behave like a `sow_and_subscribe` command. ## Header Fields The following table contains the header fields supported by a `sow_and_delta_subscribe` command. | Field | Description | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

The command to be executed.

Value: `sow_and_delta_subscribe`

| | `topic` | The target SOW topic to query and subscribe to. | | `ack_type` |

Acknowledgment type for the given command.

Value is a comma separated string of one or more of the following: `none`, `received`, `processed`, `completed` or `stats`.

| | `batch_size` |

Number of records to return in a single sow query results message.

While the default value is 1, it is recommended to use a higher value, as even small increases can yield greater performance in query result delivery.

| | `cmd_id` | If specified with an AMPS command requesting an acknowledgment message, all `ack` messages will contain the `CommandId` in the acknowledgment message. | | `data_only` |

If `true`, send only raw data to subscriber for a matching publish message.

For example, this will remove the SOAP envelope in an XML message.

| | `filter` | Content filter expression. | | `opts` |

A comma separated list of flags available to the `sow_and_delta_subscribe` command.

The [Options Field](#options-field) table below describes the `Options` available for use in the `sow_and_delta_subscribe` command.

| | `orderby` |

Return the SOW results sorted by the specified fields.

Fields are a comma-delimited list of AMPS identifiers, and may optionally include a sort specifier, `ASC` or `DESC`.

| | `query_id` |

Identifier used to identify the client's SOW topic query.

This identifier will be added to all messages that represent a response to the `sow_and_delta_subscribe` command.

| | `send_empty` |

If set to `true`, empty published messages are forwarded to matching subscriptions.

Default: `true`

| | `send_oof` |

Messages that have fallen out of focus from the subscription are sent to the client.

Default: `false`

| | `send_keys` | Option to instruct AMPS that the client would like to receive the `SowKey` back. | | `send_matching_ids` | If `true` subscription identifiers will be sent for a matched message. | | `sow_keys` |

A comma-delimited list of `SowKeys` that identify the messages to return from the query.

For example, you might send a query with the SowKeys value `42,100,3467` which would return records with those `SowKey` values, if any exist in the SOW.

| | `sub_id` |

The subscription ID for this command. When provided with a new subscription, this is the identifier that AMPS will use for the subscription.

When provided with the `replace` option, this field specifies the subscription to replace.

When provided with a `pause` or `resume` option, this field specifies the subscriptions to pause or resume.

For a new subscription, the AMPS clients will generate a subscription ID if one is not provided.

| | `top_n` | Return up to the number of messages specified from the SOW query. | ## Options Field The following table contains a list of the `Options` available and their definitions when used in the AMPS `sow_and_delta_subscribe` command. | Option | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `none` | This is the default `Options` type. | | `bookmark` |

Specifies that the subscription should return bookmarks, if available, on each publish message. Bookmarks are only available if the topic that is subscribed to is recorded in the transaction log.

Notice that this option does not set the starting point for a bookmark subscription: use the `Bookmark` header for that purpose.

This option is not required to receive bookmarks for a bookmark subscription. Those subscriptions always include bookmarks on each publish message.

| | `conflation=n` |

Specifies whether to conflate this subscription.

The value provided can be a time interval, `auto`, or `none`.

When present and set to a value other than none, enables conflation for the subscription.

Can also be set to `auto`, which requests that AMPS attempt to determine an appropriate conflation interval based on client consumption.

Recognizes the same time specifiers used in the AMPS configuration file (for example, `100ms` or `1s` or `1m`).

Default: `none`

| | `conflation_key=[key]` |

When conflation is enabled, specifies the fields to use to determine message uniqueness.

The format of this option is a comma-delimited list of XPath identifiers within brackets. For example, to conflate based on the value of the `/tickerId` and `/customerId` within a message, the value of this option would be:

`[/tickerId,/customerId]`

Defaults to the SOW key fields for SOW topics. No default for non-SOW topics. This option is required for non-SOW topics.

This option is not valid with the `oof` option unless the keys provided are identical to the keys for the topic.

| | `grouping=[keys]` |

For use with aggregated subscriptions.

The format of this option is a comma-delimited list of XPath identifiers within brackets.

For example, to aggregate entries based on their `/description` (producing one record in the aggregation for each distinct value in `/description`), the value of this option would be:

`[/description]`

This option must contain an entry for every field in the aggregated message. If there is no entry for a field in this option, that field will not appear in the aggregated message, even if the field is in the underlying message.

When this option is provided, a `projection` must also be provided.

This option cannot be used with a bookmark.

| | `live` | Tells AMPS to send messages to subscribing clients before they have been persisted to the transaction log. | | `no_empties` |

Tells AMPS not to send empty publish messages to matching subscriptions.

This can be useful for suppressing messages where no fields have changed.

| | `no_sowkey` | Tells AMPS not to send the AMPS-generated `SowKey` for messages. | | `non_regex_topic` | Specifies that the topic name should be a literal match, even if the topic name contains regular expression characters. | | `oof` |

Send an `OOF` message for records that have fallen out of focus from the original subscription.

When focus tracking is enabled, AMPS will also deliver the full message to a subscription when a previously out-of-focus message comes into focus.

| | `projection=[fields]` |

For use with aggregated subscriptions.

Specifies a comma-delimited set of fields to project, within brackets. Each entry has the format described in the AMPS User Guide.

This option must contain an entry for every field in the aggregated message. If there is no entry for a field in this option, that field will not appear in the aggregated message, even if the field is in the underlying message.

There is no default for this option. When this option is provided, a `grouping` must also be provided. This option cannot be used with a bookmark.

The maximum size of this option is 64KB.

| | `replace` | Replace the subscription associated with `SubId` with another subscription. | | `top_n=n` |

AMPS will provide at most `n` records, starting at the beginning of the result set as defined by the `OrderBy` header.

This option is equivalent to providing a `TopN` header.

| | `skip_n=n` |

AMPS will skip the specified number of records in the result set before returning results.

This option is used with the `top_n` option to create a paginated subscription.

| | `send_keys` | AMPS will send the SOW keys (that is, the data fields used to identify unique messages in the SOW) back with matching messages from the SOW. | | `select=[fields]` |

Specifies the fields to include in messages provided on this subscription.

The contents of this option are a comma-delimited list of inclusion specifiers.

| | `timestamp` | AMPS will include a header with the time at which this instance of AMPS processed the incoming publish command for this message. | ## Returns A `sow_and_delta_subscribe` command returns the following command types: | Command | Description | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `group_begin` | Indicates the beginning of results from a SOW query. | | `group_end` | Indicates the end of results from a SOW query. | | `sow` | Indicates a record returned from a SOW query. | | `publish` | A new message or update published to AMPS. | | `oof` | Returned when the `oof` option is requested on the command, the subscription is to a topic in the SOW, and the subscription is not a bookmark subscribe. | | `ack` | Acknowledgments requested, as described in the following section. | AMPS will send acknowledgment messages for the following `AckType` fields: `received` and `processed`, along with a populated `Status` header field describing the acknowledgment message. If the `sow_and_delta_subscribe` command is successful, AMPS will return a `group_begin` message to notify the client that a group of messages is being returned as part of the `sow` portion of the command. For more information about SOW topic query behavior, see the chapter on [Querying the State of the World](../../../amps-user-guide/sow-queries) in the _AMPS User Guide_. The following table contains the `AckType` messages which can be returned by a `sow_and_delta_subscribe`. | Acknowledgment | Description | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `none` |

No `ack` message is returned.

This is the default behavior.

| | `completed` | The `sow_and_delta_subscribe` message has completed the `sow` portion of the command, and all future messages will be updated based on publishes. | | `persisted` | Not supported at this time. | | `processed` | AMPS has compiled the filter(s) for the `sow_and_delta_subscribe` message(s). | | `received` | The `sow_and_delta_subscribe` message has been received. | | `stats` | Returns an `ack` message with `Matches`, `TopicMatches` and `RecordsReturned`. | The `stats` acknowledgment message includes three values in the header, the `Matches`, `TopicMatches` and the `RecordsReturned`. These are defined below: ### TopicMatches The total number of records compared across all matching SOW topics. ### Matches The number of records returned that match the topic regular expression and the content filter. This value can be greater than `RecordsReturned` in the case where the number of returned records is limited by `TopN`. ### RecordsReturned The total number of records returned to the client, which can be limited by the `TopN` header value. ## Errors Errors for a `sow_and_delta_subscribe` query are either returned in the `Status` field if an `AckType` has been defined, or the errors may be inserted into the AMPS log. --- # sow_and_subscribe command A `sow_and_subscribe` command is used to combine the functionality of `sow` and a `subscribe` command in a single command. The `sow_and_subscribe` command is used: (a) to query the contents of a SOW topic (this is the `sow` command); and (b) to place a subscription such that any messages matching the subscribed SOW topic and query filter will be published to the AMPS client (this is the `subscribe` command). As with the `subscribe` command, `publish` messages representing updates to SOW records are provided for records that match the filter when there is an update published to that record. ## Header Fields The following table contains the header fields supported by a `sow_and_subscribe` command. | Field | Description | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

The command to be executed.

Value: `sow_and_subscribe`

| | `topic` | The target SOW topic to query and subscribe. | | `ack_type` |

Acknowledgment type for the given command.

Value is a comma separated string of one or more of the following: `none`, `received`, `processed`, `completed` or `stats`.

| | `batch_size` |

Number of records to return in a single sow query results message.

The AMPS server default value is 1.

It is recommended to use a higher value, as even small increases can yield greater performance in query result delivery. Current AMPS client libraries provide a batch size of 10 by default.

The `BatchSize` header only applies to the sow query.

| | `bookmark` |

A bookmark specifying the historical state of the SOW to return results from. For SOW topics where historical query is enabled, AMPS returns the saved state of the SOW as of that bookmark. For SOW topics where historical query is not enabled, the only valid bookmark is NOW.

If the topic is enabled for historical query and AMPS has a transaction log that covers the topic, AMPS returns the saved state of the SOW as of that bookmark and starts a bookmark subscription at a point in the transaction log immediately after the point at which the SOW state was saved.

In other words, if the granularity of the historical SOW preserves the state of the SOW at 11:30:10 AM and 11:30:50 AM, a request for a bookmark at 11:30:20 AM will provide the SOW state as of 11:30:10 AM, and begin the replay immediately after that SOW state. This ensures no messages are missed, but means that the subscription may begin before the bookmark.

| | `cmd_id` | If specified with an AMPS command requesting an acknowledgment message, all `ack` messages will contain the `CommandId` in the acknowledgment message. | | `data_only` |

Only send raw data to subscriber for a matching `publish` message if `true`.

For example, this will remove the SOAP envelope in an XML message.

| | `filter` | Content filter expression. | | `opts` |

A comma separated list of flags available to the `sow_and_subscribe` command.

The [Options Field](#options-field) table below describes the `Options` available for use in the `sow_and_subscribe` command.

| | `orderby` |

Return the SOW results sorted by the specified fields.

Fields are a comma-delimited list of AMPS identifiers, and may optionally include a sort specifier, `ASC` or `DESC`.

| | `query_id` |

Identifier used to identify the client's SOW topic query.

This identifier will be added to all messages representing a response to the `sow_and_subscribe` command.

| | `send_oof` |

Messages that have fallen out of focus from the subscription are sent to the client.

Default: `false`

| | `send_keys` | Option to instruct AMPS that the client would like to receive the `SowKey` back. | | `send_matching_ids` | If `true`, subscription identifiers will be sent for a matched message. | | `sow_keys` | A comma-delimited list of `SowKeys` that identify the messages to return from the query. | | `sub_id` |

The subscription ID for this command. When provided with a new subscription, this is the identifier that AMPS will use for the subscription.

When provided with the `replace` option, this field specifies the subscription to replace.

When provided with a `pause` or `resume` option, this field specifies the subscriptions to pause or resume.

For a new subscription, the AMPS clients will generate a subscription ID if one is not provided.

| | `top_n` | Return up to the number of messages specified from the SOW query. | ## Returns A `sow_and_subscribe` command returns the following command types: | Command | Description | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `group_begin` | Indicates the beginning of results from a SOW query. | | `group_end` | Indicates the end of results from a SOW query. | | `sow` | Indicates a record returned from a SOW query | | `publish` | A new message or update published to AMPS. | | `oof` | Returned when the `oof` option is requested on the command, the subscription is to a topic in the SOW, and the subscription is not a bookmark subscribe. | | `ack` | Acknowledgments requested, as described in the following section. | AMPS will send acknowledgment messages for the following `AckType` fields: `received`, `processed` along with a populated `Status` header field describing the acknowledgment message. If the `sow_and_subscribe` command is successful, AMPS will return a `group_begin` message to notify the client that a group of messages is being returned as part of the `sow` portion of the command. For more information about SOW topic query behavior, see the chapter on [Querying the State of the World](../../../amps-user-guide/sow-queries) in the _AMPS User Guide_. The following table contains the `AckType` messages that can be returned by a `sow_and_subscribe`. | Acknowledgment | Description | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `none` |

No `ack` message is returned.

This is the default behavior.

| | `bookmark` |

Specifies that the subscription should return bookmarks, if available, on each publish message. Bookmarks are only available if the topic that is subscribed to is recorded in the transaction log.

Notice that this option does not set the starting point for a bookmark subscription: use the `Bookmark` header for that purpose.

This option is not required to receive bookmarks for a bookmark subscription. Those subscriptions always include bookmarks on each publish message.

| | `completed` | The `sow_and_subscribe` message has completed the `sow` portion of the command, and all future messages will be updated based on publishes. | | `persisted` | Not supported at this time. | | `processed` | AMPS has completed the work necessary to register the subscription and begin the SOW query. | | `received` | The `sow_and_subscribe` message has been received. | | `stats` | Returns an `ack` message with `Matches`, `TopicMatches` and `RecordsReturned`. | The `stats` acknowledgment message includes three values in the header: `Matches`, `TopicMatches` and the `RecordsReturned`. These are defined below: ### TopicMatches The total number of records compared across all matching SOW topics. ### Matches The number of records returned that match the topic regular expression and the content filter. This value can be greater than `RecordsReturned` in the case where the number of returned records is limited by `TopN`. ### RecordsReturned The total number of records returned to the client, which can be limited by the `TopN` header value. ## Options Field The following table contains a list of the `Options` available and their definitions when used in the AMPS `sow_and_subscribe` command. | Option | Description | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `none` | This is the default `Options` type. | | `conflation=n` |

Specifies whether to conflate this subscription.

The value provided can be a time interval, `auto`, or `none`.

When present and set to a value other than none, enables conflation for the subscription.

Can also be set to `auto`, which requests that AMPS attempt to determine an appropriate conflation interval based on client consumption.

Recognizes the same time specifiers used in the AMPS configuration file (for example, `100ms` or `1s` or `1m`).

Default: `none`

| | `conflation_key=[key]` |

When conflation is enabled, specifies the fields to use to determine message uniqueness. The format of this option is a comma-delimited list of XPath identifiers within brackets.

For example, to conflate based on the value of the `/tickerId` and `/customerId` within a message, the value of this option would be:

`[/tickerId,/customerId]`

Defaults to the SOW key fields for SOW topics.

No default for non-SOW topics. This option is required for non-SOW topics.

This option is not valid with the `oof` option unless the keys provided are the same as the keys for the underlying topic.

| | `grouping=[keys]` |

For use with aggregated subscriptions.

The format of this option is a comma-delimited list of XPath identifiers within brackets.

For example, to aggregate entries based on their `/description` (producing one record in the aggregation for each distinct value in `/description`), the value of this option would be:

`[/description]`

This option must contain an entry for every field in the aggregated message. If there is no entry for a field in this option, that field will not appear in the aggregated message, even if the field is in the underlying message.

When this option is provided, a `projection` must also be provided.

This option cannot be used with a bookmark.

| | `live` |

Tells AMPS to send messages to subscribing clients before they have been persisted to the transaction log.

This option is only valid for bookmark subscriptions.

| | `no_sowkey` | Tells AMPS not to send the AMPS-generated `SowKey` for messages. | | `non_regex_topic` | Specifies that the topic name should be a literal match, even if the topic name contains regular expression characters. | | `oof` | Send on `OOF` message for records which have fallen out of focus from the original subscription. | | `pause` |

Pause a bookmark subscription.

This option is only valid for bookmark subscriptions that do not use the `live` option. When this option is present, AMPS pauses the subscription or subscriptions provided in the `SubId` of the command.

| | `projection=[fields]` |

For use with aggregated subscriptions.

Specifies a comma-delimited set of fields to project, within brackets. Each entry has the format described in the AMPS User Guide.

This option must contain an entry for every field in the aggregated message. If there is no entry for a field in this option, that field will not appear in the aggregated message, even if the field is in the underlying message.

There is no default for this option. When this option is provided, a `grouping` must also be provided. This option cannot be used with a bookmark.

The maximum size of this option is 64KB.

| | `rate=n` |

Set the maximum message delivery rate for a bookmark subscription.

This option is only valid for bookmark subscriptions that do not use the `live` option.

The rate can be specified as either the number of messages per second (for example, `1000`), the number of bytes per second (for example, `100KB`), or a multiple of the original replay rate (for example, `1.5X`).

| | `replace` |

Replace the subscription associated with `SubId` with another subscription.

When provided as part of `sow_and_subscribe`, AMPS runs a SOW query for the new subscription.

| | `resume` |

Resume a bookmark subscription.

This option is only valid for bookmark subscriptions that do not use the `live` option.

When this option is present, AMPS resumes the subscription or subscriptions provided in the `SubId` of the command.

| | `top_n=n` |

AMPS will provide at most `n` records, starting at the beginning of the result set as defined by the `OrderBy` header.

This option is equivalent to providing a `TopN` header.

| | `skip_n=n` |

AMPS will skip the specified number of records in the result set before returning results.

This option is used with the `top_n` option to create a paginated subscription.

| | `send_keys` | AMPS will send the SOW keys (that is, the data fields used to identify unique messages in the SOW) back with matching messages from the SOW. | | `select=[fields]` |

Specifies the fields to include in messages provided on this subscription.

The contents of this option are a comma-delimited list of inclusion specifiers.

| | `timestamp` | AMPS will include a header with the time at which this instance of AMPS processed the incoming publish command for this message. | ## Errors Errors for a `sow_and_subscribe` query are returned in the `Status` field if an `AckType` has been defined. Errors will also be logged in the errors and events log (subject to logging configuration). --- # sow command The `sow` command is used to query the contents of a previously defined topic in the SOW (including also views, queues, and conflated topics). A `sow` command can be used to query an entire SOW `Topic`, or a filter can be used to further refine the results from the SOW. For more information, see the [State of the World](../../../amps-user-guide/sow) and [SOW Queries](../../../amps-user-guide/sow-queries) chapters in the _AMPS User Guide._ ## Header Fields | Field | Description | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

Command to be executed.

Value: `sow`

| | `topic` | The SOW topic from which the records will be queried. | | `ack_type` |

Acknowledgment type for the given command.

Value is a comma separated list of one or more of the following: `none`, `received`, `processed`, `completed` or `stats`.

| | `batch_size` |

Number of records to return in a single `sow` query result message.

The AMPS server default value is 1.

It is recommended to use a higher value, as even small increases can yield greater performance in query result delivery. Current AMPS client libraries provide a `BatchSize` of 10 by default.

The `BatchSize` header only applies to a sow query.

| | `bookmark` |

A bookmark specifying the historical state of the SOW to return results from.

For SOW topics where historical query is enabled, AMPS returns the saved state of the SOW as of that bookmark.

For SOW topics where historical query is not enabled, AMPS ignores this parameter.

| | `cmd_id` | If specified with an AMPS command requesting an acknowledgment message, all requested acknowledgment messages will contain the `CommandId` in the `ack` response header. | | `filter` |

Content filter expression.

See the Content Filtering chapter in the AMPS User Guide for more information on using content filters.

| | `order_by` |

Return the SOW results sorted by the specified fields.

Fields are a comma-delimited list of AMPS identifiers, and may optionally include a sort specifier, `ASC` or `DESC`.

| | `query_id` | Unique identifier which is returned as part of the response delivered back to the client. | | `sow_keys` | A comma-delimited list of `SowKeys` that identify the messages to return from the query. | | `top_n` | Return up to the number of messages specified from the SOW query. | ## Options Field The following table contains a list of the `Options` available and their definitions when used in the AMPS `sow` command. | Option | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `none` | This is the default `Options` type. | | `no_sowkey` | Tells AMPS not to send the AMPS-generated `SowKey` for messages. | | `grouping=[keys]` |

For use with aggregated SOW queries.

The format of this option is a comma-delimited list of XPath identifiers within brackets.

For example, to aggregate entries based on their `/description` (producing one record in the aggregation for each distinct value in `/description`), the value of this option would be:

`[/description]`

This option must contain an entry for every field in the aggregated message. If there is no entry for a field in this option, that field will not appear in the aggregated message, even if the field is in the underlying message.

When this option is provided, a `projection` must also be provided.

When the topic has `History` enabled, this option can be used with a bookmark to aggregate the historical state of the SOW.

| | `oof` | Send on `OOF` message for records which have fallen out of focus from the original subscription. | | `projection=[fields]` |

For use with aggregated SOW queries.

Specifies a comma-delimited set of fields to project, within brackets. Each entry has the format described in the AMPS User Guide.

This option must contain an entry for every field in the aggregated message. If there is no entry for a field in this option, that field will not appear in the aggregated message, even if the field is in the underlying message.

There is no default for this option. When this option is provided, a `grouping` must also be provided.

When the topic has `History` enabled, this option can be used with a bookmark to aggregate the historical state of the SOW.

The maximum size of this option is 64KB.

| | `replace` |

Replace the subscription associated with `SubId` with another subscription.

When provided as part of `sow_and_subscribe`, AMPS runs a SOW query for the new subscription.

| | `skip_n=n` |

Skips the number of messages specified before returning results.

A command that provides this option must also provide a `top_n` option (or header) and an `OrderBy` header.

| | `top_n=n` | Return up to the number of messages specified from the SOW query. | | `select=[fields]` |

Specifies the fields to include in messages provided on this subscription.

The contents of this option are a comma-delimited list of inclusion specifiers.

| | `send_keys` | AMPS will send the SOW keys (that is, the data fields used to identify unique messages in the SOW) back with matching messages from the SOW. | | `timestamp` | AMPS will include a header with the time at which AMPS processed the incoming publish command for this message. | ## Returns When a `sow` message is received, AMPS can return a `received` message as notification that the message has arrived. When the message filter has been processed, AMPS will return the `processed` acknowledgment message along with any errors that might have occurred. The results returned by a SOW are put into a `sow` record group by first sending a `group_begin` message, followed by the matching SOW records. A `group_end` message is used to denote the close of query results processing. The following table contains a listing of the acknowledgment messages supported by the `sow` command. | Acknowledgment | Description | | ------------------ | -------------------------------------------------------------------- | | `none` | No acknowledgment message is returned. This is the default behavior. | | `completed` | The `sow` command has completed. | | `persisted` | Not supported at this time. | | `processed` | AMPS has compiled the filter(s) for the `sow` message. | | `received` | The `sow` command has been received. | | `stats` | Returns statistics related to the state of the SOW query results. | The stats message include three values in the header: `Matches`, `TopicMatches`, and the `RecordsReturned`. These are defined below: ### TopicMatches The total number of records compared across all matching SOW topics. ### Matches The number of records returned that match the topic regular expression and the content filter. This value can be greater than `RecordsReturned` in the case where the number of returned records is limited by `TopN`. ### RecordsReturned The total number of records returned to the client, which can be limited by the `TopN` header value. ## Errors Any errors which occur during a `sow` command are returned in the `processed` acknowledgment message. The error is identified in the `Status` header field in the acknowledgment message, and the reason given in the `Reason` header field. :::tip The ordering of records returned by a SOW query is undefined unless the `OrderBy` header on the command is provided. ::: --- # subscribe command The `subscribe` command is the primary way to retrieve messages from the AMPS processing stream. A client can issue a `subscribe` command on a topic to receive all published messages to that topic in the future. Additionally, content filtering can be used to choose which messages the client is interested in receiving. ### Header Fields | Field | Description | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

Command to be executed.

Value: `subscribe`

| | `topic` | Topic to place a subscription against. | | `ack_type` |

Acknowledgment type for the given command.

Value is a comma separated list of one or more of the following: `none`, `received`, `processed` or `completed`.

| | `bookmark` |

A bookmark specifying the point in the transaction log at which to start the subscription.

If the topic provided is not recorded in a transaction log, AMPS enters the subscription without replaying messages. You can provide a single bookmark, or a comma-delimited list of bookmarks. When a list is provided, AMPS starts the subscription at the earliest bookmark in the list.

| | `cmd_id` | If specified with an AMPS command requesting an acknowledgment message, all requested acknowledgment messages will contain the `CommandId` in the `ack` response header. | | `data_only` |

A Boolean value (`true` or `false`) which, if `true`, will send only raw data to subscriber for a matching publish message.

In the case where the message type is XML, the SOAP envelope will not be included.

| | `filter` | A CDATA wrapped string, used as a content filter expression. | | `opts` |

A comma separated list of flags available to the `subscribe` command.

The [Options Field](#options-field) table below describes the `Options` available for use in the `subscribe` command.

| | `send_matchng_ids` | Boolean (`true` or `false`) that tells if `true` requests AMPS to send subscription identifiers with a matched message. | | `sub_id` |

The subscription ID for this command. When provided with a new subscription, this is the identifier that AMPS will use for the subscription.

When provided with the `replace` option, this field specifies the subscription to replace.

When provided with a `pause` or `resume` option, this field specifies the subscriptions to pause or resume.

For a new subscription, the AMPS clients will generate a subscription ID if one is not provided.

| | `top_n` |

The maximum number of messages to provide from a bookmark subscription.

This parameter is only valid for replay from the transaction log.

This parameter is not valid if no bookmark is provided, if the provided bookmark is `0\|1\|` (start from now), or if the command includes the `live` option.

| ### Options Field The following table contains a list of the `Options` available and their definitions when used in the AMPS `subscribe` command. | Option | Description | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `none` | This is the default `Options` type. | | `bookmark` |

Specifies that the subscription should return bookmarks, if available, on each publish message. Bookmarks are only available if the topic that is subscribed to is recorded in the transaction log.

Notice that this option does not set the starting point for a bookmark subscription: use the `Bookmark` header for that purpose.

This option is not required to receive bookmarks for a bookmark subscription. Those subscriptions always include bookmarks on each publish message.

| | `bookmark_not_found` |

Specifies how AMPS should interpret a failure to locate one of the bookmarks provided in the starting point for the bookmark subscription.

This option should be one of three parameters:

If this option is not specified in a bookmark subscription, the default value of `now` is used.

This option is unsupported on subscriptions that are not bookmark replays.

| | `conflation=n` |

Specifies whether to conflate this subscription.

The value provided can be a time interval, `auto`, or `none`.

When present and set to a value other than none, enables conflation for the subscription.

Can also be set to `auto`, which requests that AMPS attempt to determine an appropriate conflation interval based on client consumption.

Recognizes the same time specifiers used in the AMPS configuration file (for example, `100ms` or `1s` or `1m`).

Default: `none`

| | `conflation_key=[keys]` |

When conflation is enabled, specifies the fields to use to determine message uniqueness.

The format of this option is a comma-delimited list of XPath identifiers within brackets.

For example, to conflate based on the value of the `/tickerId` and `/customerId` within a message, the value of this option would be:

`[/tickerId,/customerId]`.

Defaults to the SOW key fields for SOW topics.

No default for non-SOW topics. This option is required for non-SOW topics.

This option is not valid with the `oof` option unless the keys provided are identical to the keys for the topic.

| | `fully_durable` |

Tells AMPS to send messages to subscribing clients only after they have been persisted to the local transaction log and acknowledged by all downstream instances that use synchronous replication.

This option is only valid for bookmark subscriptions.

| | `live` |

Tells AMPS to send messages to subscribing clients before they have been persisted to the transaction log.

This option has no effect on subscriptions that are not replays from the transaction log (that is, this option only applies to bookmark subscriptions).

| | `max_backlog=n` |

When subscribing to a queue, the number of unacknowledged messages the client is willing to accept at a time.

AMPS will not exceed this number, but may choose a smaller number depending on the queue configuration.

This option does not apply for subscriptions that are *not* subscriptions to a queue, local queue, or group local queue.

| | `non_regex_topic` | Specifies that the topic name should be a literal match, even if the topic name contains regular expression characters. | | `no_sowkey` | Tells AMPS not to send the AMPS-generated `SowKey` for messages. | | `oof` | Provide out of focus notifications. Only supported if the topic for the subscription is a Topic in the State of the World, a View, or a Conflated Topic. | | `pause` |

Pause a bookmark subscription.

This option is only valid for bookmark subscriptions that do not use the `live` option. When this option is present, AMPS pauses the subscription or subscriptions provided in the `SubId` of the command.

| | `rate=n` |

Set the maximum message delivery rate for a bookmark subscription.

This option is only valid for bookmark subscriptions that do not use the `live` option.

The rate can be specified as either the number of messages per second (for example, `1000`), the number of bytes per second (for example, `100KB`), or a multiple of the original replay rate (for example, `1.5X`).

| | `rate_max_gap=n` | When entering a bookmark subscribe that is rate-limited, specify the maximum amount of time that AMPS will wait between messages. | | `replace` | Replace the subscription associated with `SubId` with another subscription. | | `resume` |

Resume a bookmark subscription.

This option is only valid for bookmark subscriptions that do not use the `live` option.

When this option is present, AMPS resumes the subscription or subscriptions provided in the `SubId` of the command.

| | `select=[fields]` |

Specifies the fields to include in messages provided on this subscription.

The contents of this option are a comma-delimited list of inclusion specifiers.

| | `send_keys` | Not supported by this command type. | | `timestamp` | AMPS will include a header with the time at which this instance of AMPS processed the incoming publish command for this message. | ### Returns A `subscribe` command returns the following command types: | Command | Description | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `publish` | A message published to AMPS. | | `oof` | Returned when the `oof` option is requested on the command, the subscription is to a topic, view, or conflated topic in the SOW, and the subscription is not a bookmark subscribe. | | `ack` | Acknowledgments requested, as described in the following section. | It is possible to specify a `processed` acknowledgment be sent back to the client that issued the `subscribe` command. Within this `processed` acknowledgment, a client can get back the result of placing the subscription (success or failure) and the `SubscriptionId`, which uniquely identifies the subscription within AMPS. Keeping track of the `SubscriptionId` is useful for unsubscribing from subscriptions and issuing SOW queries. The following table contains a list of the supported acknowledgment messages available to the `subscribe` command. | Acknowledgment | Description | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `none` | No acknowledgment message is returned. This is the default behavior. | | `completed` |

Used by bookmark subscriptions to identify the point at which replay is completed.

When a bookmark is present on the subscribe request and this acknowledgment is requested, AMPS sends a `completed` acknowledgment message to indicate that bookmark replay is complete and the subscription has reached the point in the transaction log at which the subscribe command was received.

Messages delivered after the `completed` ack are from new publishes.

| | `processed` |

AMPS has completed the work necessary to register the subscription.

When a bookmark is present and this acknowledgment is requested, this acknowledgment indicates that AMPS is about to begin replay.

| | `persisted` | When a bookmark is present and this acknowledgment is requested, AMPS periodically sends a `persisted` acknowledgment message to indicate the most recent fully-persisted bookmark in the server's transaction log. See [Bookmark Subscriptions and Persisted Acknowledgements](../../../amps-user-guide/acks/bookmark-subscriptions-and-persisted-acknowledgments). | | `received` | The `subscribe` message has been received. | ### Errors Any errors that occur during this command will be returned in the status of a `processed` acknowledgment and logged to the log file. Regardless of success or failure, the processed acknowledgment will only be returned if requested by including `processed` in the `AckType` field. --- # unsubscribe command The `unsubscribe` command allows a client to notify AMPS that it no longer wishes to receive messages related to a previous subscription. There are two ways that a client can unsubscribe from an existing subscription: 1. Adding the `all` keyword to the `SubId` header field in the `unsubscribe` message will unsubscribe the client from all AMPS SOW topic subscriptions. 2. With each `subscription` command issued, AMPS will return a `SubId` with the `processed` acknowledgment message. Issuing an `unsubscribe` command using the same `SubId` header field which was returned as part of the original `subscribe` command's `processed` acknowledgment message will unsubscribe a client from a single subscription. ## Header Fields | Field | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

Command to be executed.

Value: `unsubscribe`

| | `sub_id` |

Subscription ID entered in AMPS by the client when the original `subscription` was placed.

AMPS accepts a single subscription ID or a comma-delimited list of subscription ID.

The keyword `all` can also be used to unsubscribe from all current subscriptions for the client.

When both `SubId` and `QueryId` are provided, AMPS removes all matching subscriptions and SOW queries.

This command requires at least one of the `SubId` or `QueryId` fields to be set.

| | `query_id` |

To cancel an in-progress SOW query, the unsubscribe command accepts the Query ID entered in AMPS by the client when the original `sow` command was placed.

AMPS accepts a single query ID or a comma-delimited list of query IDs.

When both `SubId` and `QueryId` are provided, AMPS removes all matching subscriptions and SOW queries.

This command requires at least one of the `SubId` or `QueryId` fields to be set.

| | `ack_type` |

Acknowledgment type for the given command.

Value is a comma separated list of one or more of the following: `none`, `received` or `persisted`.

| | `cmd_id` | If specified within an AMPS command requesting an acknowledgment message, all requested acknowledgment messages will contain the `CommandId` in the `ack` response header. | ## Returns The `unsubscribe` command supports the `received` and `processed` acknowledgment message types, as described in the following table. | Acknowledgment | Description | | ------------------ | -------------------------------------------------------------------- | | `none` | No acknowledgment message is returned. This is the default behavior. | | `completed` | Not supported at this time. | | `processed` | AMPS has processed the `unsubscribe` message(s). | | `persisted` | Not supported at this time. | | `received` | The `unsubscribe` message has been received. | | `stats` | Not supported at this time. | --- # Removing Messages (SOW/Topic or Message Queue) In AMPS, there are three different ways to remove records from the SOW. The first method is to construct a `publish` message that matches the message to be removed, with the `Command` field set to be a `sow_delete` message. This has the net effect of causing AMPS recreate the `SowKey` for the particular message, then look up the `SowKey` message in the SOW and finally remove it. The other method to remove messages from the SOW is to construct a `sow_delete` message and pass in a comma separated list of `SowKey`s in the message header which will cause all of the messages to be removed from the SOW Topic. The third way to remove records from the SOW is similar to the manner in which a `sow` query command with a `filter` is performed. In this case, instead of returning the results of the `sow` command, those records that match the filter will be deleted from the SOW. The `sow_delete` command is also used to acknowledge messages from a queue. With this form of `sow_delete`, a client sends one or more bookmarks (as a comma-delimited list) that specify the messages to acknowledge. This form can only be used for acknowledging messages from a queue topic. ## Header Fields The following table contains the header fields supported by a `sow_delete`. | Field | Description | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

Command to be executed.

Value: `sow_delete`

| | `topic` | The SOW topic from which to delete the messages(s). | | `ack_type` |

Acknowledgment type for the given command.

Value is a comma separated list of one or more of the following: `none`, `received`, `processed`, `persisted`, `completed` and `stats`.

| | `cmd_id` | If specified with an AMPS command requesting an `ack`, all requested acknowledgment messages will contain the command ID in the acknowledgment message header. | | `sow_keys` |

A comma separated list of unique ids to be deleted.

AMPS uses these IDs to locate and remove the specified records. Notice that these values are the internal ID used by AMPS -- the `sow_key` -- and not the value of a field in the message.

To use the values of fields in the message to locate the records to remove, use a `filter` or `Data`.

| | `filter` |

Content filter expression.

See the Content Filtering chapter in the AMPS User Guide for more information on using content filters.

When provided, AMPS removes the matching records.

| | `data` |

Message data that identifies the record to be removed.

When provided, AMPS uses this `data` to look up the record that would be updated were this command a publish. AMPS then deletes that record.

| | `bookmark` |

Processed when the `sow_delete` command is acknowledging a message from a queue.

When this option is used, the message must have been provided from a message queue, and the `SowKeys` and `Filter` headers may not be used.

| | `opts` |

Available when the `sow_delete` command is acknowledging a message from a queue.

When a value of `cancel` is provided in this field, the message is returned to the queue and made available to other subscribers.

When a value of `expire` is provided in this field, the message is automatically expired by AMPS, removing the message from the queue and invoking any actions configured to listen for message expirations.

| :::warning The `sow_keys`, `filter`, `data` and `bookmark` header fields cannot be used together. They are mutually exclusive. Using them together in the same `sow_delete` command will cause indeterminate results. The `bookmark` header can only be used to acknowledge messages from queue topics. ::: ## Returns For a `sow_delete` message, AMPS will send acknowledgment message, `completed` and `stats` for the following acknowledgment message types: `received`, `processed` and `persisted` along with a populated `Status` header field describing the acknowledgment. | Acknowledgment | Description | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `none` |

No acknowledgment message is returned.

This is the default behavior.

| | `completed` |

Supported for a `sow_delete` with a `Filter` defined.

The `completed` acknowledgment message is returned when the query portion of the command has completed.

| | `persisted` |

When an AMPS engine returns an acknowledgment message of `persisted` this guarantees that:

  1. All downstream synchronous replication(s) have acknowledged that the message(s) have been deleted from their respective SOW topic(s).
  2. The `sow_delete` message has been sent to all available downstream asynchronous replications.
| | `processed` | AMPS has compiled the filter(s) for the `sow_delete` messages. | | `received` | The `sow_delete` message has been received. | | `stats` | Returns an acknowledgment message with `Matches`, `TopicMatches` and `RecordsDeleted`. | The `stats` acknowledgment message include three values in the header, `Matches`, `TopicMatches` and the `RecordsDeleted`. These are defined below: ### TopicMatches The total number of records compared across all matching SOW topics. ### Matches The number of records returned that match the topic regular expression and the content filter. ### RecordsDeleted The total number of records deleted. ## Errors Errors that occur during a `sow_delete` are returned as part of the `processed` acknowledgment message and recorded to the log. Typical errors involved a missing topic, or a missing/invalid `SowKey`. --- # Command Cookbook This section lists the headers and options that are set for common commands. It is not intended to be exhaustive. Instead, this section is intended to provide a quick reference. For detailed information on a specific command, including a full list of available options, see the reference section for that specific command. --- # Cookbook: Delta Publishing This section presents common recipes for publishing to a topic in AMPS using the `Command` or `Message` interfaces. This section provides information on how to configure the request to AMPS. You can adapt this information to your application and the specific interface you are using. *Command:* `delta_publish` ## Basic Delta Publish In its simplest form, a delta publish needs only the topic to publish to and the data to publish. The AMPS client automatically constructs the necessary AMPS headers and formats the full `delta_publish` command. In many cases, a publisher only needs to use the basic delta publish command. | Header | Comment | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to publish to. The topic specified must be a literal topic name. Regular expression characters in the topic name are not interpreted. Some topics in AMPS, such as views and conflated topics, cannot be published to directly. Instead, a publisher must publish to the underlying topics. | | `data` | The data to publish to the topic. The AMPS client does not interpret, escape, or validate this data: the data is provided to the server verbatim. | ## Delta Publish with CorrelationId AMPS provides publishers with a header field that can be used to contain arbitrary data, the `correlation_id`. A delta publish message can be used to update the `correlation_id` as well as the data within the message. | Header | Comment | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to publish to. The topic specified must be a literal topic name. Regular expression characters in the topic name are not interpreted. Some topics in AMPS, such as views and conflated topics, cannot be published to directly. Instead, a publisher must publish to the underlying topics. | | `data` | The data to publish to the topic. The AMPS client does not interpret, escape, or validate this data: the data is provided to the server verbatim. | | `correlation_id` | The `correlation_id` to provide on the message. AMPS provides the `correlation_id` to subscribers. The `correlation_id` has no significance for AMPS. The `correlation_id` may only contain characters that are valid in base-64 encoding. | ## Delta Publish with Explicit SOW Key When publishing to a SOW topic that is configured to require an explicit SOW Key, the publisher needs to set the `sow_key` header on the message. | Header | Comment | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to publish to. The topic specified must be a literal topic name. Regular expression characters in the topic name are not interpreted. Some topics in AMPS, such as views and conflated topics, cannot be published to directly. Instead, a publisher must publish to the underlying topics. | | `data` | The data to publish to the topic. The AMPS client does not interpret, escape, or validate this data: the data is provided to the server verbatim. | | `sow_key` | The SOW Key to use for this message. This header is only supported for publishes to a topic that requires an explicit SOW Key. | --- # Cookbook: Delta Subscribe This section presents common recipes for subscribing to a topic in AMPS using the `Command` or `Message` interfaces. This section provides information on how to configure the request to AMPS. You can adapt this information to your application and the specific interface you are using. _Command:_ `delta_subscribe` ## Basic Delta Subscription In its simplest form, a delta subscription needs only the topic to subscribe to. | Header | Comment | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

`topic`

(required)

| Sets the topic to subscribe to. All messages from the topic will be delivered on this subscription. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | ## Delta Subscription with Options In its simplest form, a delta subscription needs only the topic to subscribe to. To add options to the subscription, set the options (`opts`) header on the command. | Header | Comment | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

`topic`

(required)

| Sets the topic to subscribe to. All messages from the topic will be delivered on this subscription. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` | A comma-delimited set of options for this command. See the [`delta_subscribe`](../commands-to-amps/query-subscribe/delta-subscribe) section of this guide for a description of supported options. | ## Delta Subscription with Content Filter To provide a content filter on a delta subscription, set the `filter` property on the command. The _AMPS User Guide_ provides details on the filter syntax. | Header | Comment | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `filter` | Sets the content filter to be applied to the subscription. Only messages that match the content filter will be provided to the subscription. | --- # Cookbook: Publishing This section presents common recipes for publishing to a topic in AMPS using the `Command` or `Message` interfaces. This section provides information on how to configure the request to AMPS. You can adapt this information to your application and the specific interface you are using. *Command:* `publish` The AMPS server does not return a stream of messages in response to a `publish` command. :::info AMPS `publish` commands do not return a stream of messages. A publish command must be used with asynchronous message processing and should typically pass an empty message handler. ::: ## Basic Publish In its simplest form, a publish needs only the topic to publish to and the data to publish. The AMPS client automatically constructs the necessary AMPS headers and formats the full `publish` command. In many cases, a publisher only needs to use the basic `publish` command. | Header | Comment | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to publish to. The topic specified must be a literal topic name. Regular expression characters in the topic name are not interpreted. Some topics in AMPS, such as views and conflated topics, cannot be published to directly. Instead, a publisher must publish to the underlying topics. | | `data` | The data to publish to the topic. The AMPS client does not interpret, escape, or validate this data: the data is provided to the server verbatim. | ## Publish with CorrelationId AMPS provides publishers with a header field that can be used to contain arbitrary data, the `correlation_id`. | Header | Comment | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to publish to. The topic specified must be a literal topic name. Regular expression characters in the topic name are not interpreted. Some topics in AMPS, such as views and conflated topics, cannot be published to directly. Instead, a publisher must publish to the underlying topics. | | `data` | The data to publish to the topic. The AMPS client does not interpret, escape, or validate this data: the data is provided to the server verbatim. | | `correlation_id` | The correlation ID to provide on the message. AMPS provides the correlation ID to subscribers. The `correlation_id` header has no significance for AMPS. The `correlation_id` may only contain characters that are valid in base-64 encoding. | ## Publish with Explicit SOW Key When publishing to a SOW topic that is configured to require an explicit SOW key, the publisher needs to set the `sow_key` header on the message. | Header | Comment | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to publish to. The topic specified must be a literal topic name. Regular expression characters in the topic name are not interpreted. Some topics in AMPS, such as views and conflated topics, cannot be published to directly. Instead, a publisher must publish to the underlying topics. | | `data` | The data to publish to the topic. The AMPS client does not interpret, escape, or validate this data: the data is provided to the server verbatim. | | `sow_key` | The SOW Key to use for this message. This header is only supported for publishes to a topic that requires an explicit SOW Key. | --- # Cookbook: SOW and Delta Subscribe This section presents common recipes for atomic SOW and Delta Subscribe in AMPS using the `Command` or `Message` interfaces. This section provides information on how to configure the request to AMPS. You can adapt this information to your application and the specific interface you are using. _Command:_ `sow_and_delta_subscribe` ## Basic SOW and Delta Subscribe In its simplest form, a SOW and Delta Subscribe needs only the topic to query and subscribe to. | Header | Comment | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | ## SOW and Delta Subscribe with Options In its simplest form, a SOW and Delta Subscribe needs only the topic to query and subscribe to. To add options to the subscription, set the `Options` header on the `Command`. | Header | Comment | | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` |

A comma-delimited set of options for this command. See the [`sow_and_delta_subscribe`](/docs/amps-command-reference/commands-to-amps/query-subscribe/sow-and-delta-subscribe) section for a full description of supported options. The most common options for this command are:

| ## SOW and Delta Subscribe with Content Filter To provide a content filter on a SOW and Delta Subscribe, set the `Filter` property on the command. The _AMPS User Guide_ provides details on the filter syntax. | Header | Comment | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `filter` | Sets the content filter to be applied to the query. Only messages that match the content filter will be returned in response to the query. | ## Paginated SOW and Delta Subscribe To request a paginated subscription for a SOW and Delta Subscribe, set the `Options` property on the command to specify the number of records to return and the number of records to skip before returning records. Most paginated subscriptions also specify the `oof` option to be notified when a record is out of focus. | Header | Comment | | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `orderby` | Specifies how to order the results within the paginated result set. If the `OrderBy` is not provided, the results are ordered by the `SowKey` for the messages. | | `opts` |

A comma-separated list of options for the command. Set the number of results returned with `top_n`, and the starting point within the result set with `skip_n`. For example, to display records 31-50 of the result set, you could provide the following option:

`top_n=20,skip_n=30,oof`

This tells AMPS to skip the first 30 records of the result set, and then provide the top 20 records from the remaining results. Out of focus notifications will be provided.

| ## Aggregated SOW and Delta Subscribe An Aggregated SOW and Delta Subscribe is a command that provides aggregation options. To add these options to the command, set the `Options` header on the `Command`. | Header | Comment | | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` |

A comma-delimited set of options for this command. For an aggregated SOW and delta subscribe, the options provide the `projection` and `grouping` for the command. A SOW and delta subscribe can provide options in addition to the `projection` and `grouping`. The most common additional options are:

| --- # Cookbook: SOW and Subscribe This section presents common recipes for atomic SOW and Subscribe in AMPS using the `Command` or `Message` interfaces. This section provides information on how to configure the request to AMPS. You can adapt this information to your application and the specific interface you are using. *Command:* `sow_and_subscribe` ## Basic SOW and Subscribe In its simplest form, a SOW and Subscribe command needs only the topic to query and subscribe to. | Header | Comment | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics.| ## SOW and Subscribe with Options In its simplest form, a SOW and Subscribe command needs only the topic to query and subscribe to. To add options to the subscription, set the `Options` header on the `Command`. | Header | Comment | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` | A comma-delimited set of options for this command. See the [sow_and_subscribe](/docs/amps-command-reference/commands-to-amps/query-subscribe/sow-and-subscribe) section for a full description of supported options. The most common options for this command are: | ## SOW and Subscribe with Select List In its simplest form, a SOW and Subscribe command needs only the topic to query and subscribe to. To add options to the subscription, set the `Options` header on the `Command`. To use a select list, include the specifier for the select list in the options provided. | Header | Comment | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` | A comma-delimited set of options for this command. See the [sow_and_subscribe](/docs/amps-command-reference/commands-to-amps/query-subscribe/sow-and-subscribe) section for a full description of supported options. For example, to remove all fields except for `id` and `ticker`, and also request out of focus notifications, you might use an options string of: `oof,select=[-/,+/id,+/ticker]`. | ## SOW and Subscribe with Content Filter To provide a content filter on a SOW and Subscribe command, set the `Filter` property on the command. The *AMPS User Guide* provides details on the filter syntax. | Header | Comment | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `filter` | Sets the content filter to be applied to the query. Only messages that match the content filter will be returned in response to the query. | ## Conflated SOW and Subscribe To request conflation on the subscription for a SOW and Subscribe command, set the `Options` property on the command to specify the conflation interval. When the topic has a SOW (including a view or a conflated topic), there is no need to provide a `conflation_key`. | Header | Comment | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` | A comma-separated list of options for the command. Set the conflation interval in the options. For example, to set a conflation interval of 250 milliseconds, provide the following option: `conflation=250ms`. To set the conflation interval to 1 minute, provide an option of: `conflation=1m `. | ## Paginated SOW and Subscribe To request a paginated subscription for a SOW and Subscribe command, set the `Options` property on the command to specify the number of records to return and the number of records to skip before returning records. Most paginated subscriptions also specify the `oof` option to be notified when a record is out of focus. | Header | Comment | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to query and subscribe to. The topic specified must be a literal topic name. Pagination is not supported with regular expression subscriptions. | | `orderby` | Specifies how to order the results within the paginated result set. If the `OrderBy` is not provided, the results are ordered by the `SowKey` for the messages. | | `opts` | A comma-separated list of options for the command. Set the number of results returned with `top_n`, and the starting point within the result set with `skip_n`. For example, to display records 31-50 of the result set, you could provide the following option: `top_n=20,skip_n=30,oof`. This tells AMPS to skip the first 30 records of the result set, and then provide the top 20 records from the remaining results. Out of focus notifications will be provided. | ## Historical SOW and Subscribe To create a historical SOW query with a subscription, set the `Bookmark` property on the command. The property can be either a specific bookmark or a timestamp. The *AMPS User Guide* provides details on creating timestamps. This command is only supported on SOW topics that are recorded in an AMPS transaction log. When `History` is enabled for the SOW topic, the `Bookmark` provided must be a timestamp or a specific bookmark value, including the special values `NOW` (`0|1|`) or `EPOCH` (`0`). If the `Bookmark` provided is a value other than `NOW` (`0|1|`), the SOW topic must have `History` enabled. | Header | Comment | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `bookmark` | Sets the historical point in the SOW at which to query. The query returns the saved state of the records in the SOW as of the point in time specified in this header. | ## Historical SOW and Subscribe with Content Filter To create a historical SOW query with a subscription, set the `Bookmark` property on the command. The property can be either a specific bookmark or a timestamp. The *AMPS User Guide* provides details on creating timestamps. This command is only supported on SOW topics that are recorded in an AMPS transaction log. If the `Bookmark` provided is a value other than NOW (`0|1|`), the SOW topic must have `History` enabled. | Header | Comment | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `bookmark` | Sets the historical point in the SOW at which to query. The query returns the saved state of the records in the SOW as of the point in time specified in this header. | | `filter` | Sets the content filter to be applied to the query. Only messages that match the content filter will be provided to the query. | ## Aggregated SOW and Subscribe An aggregated SOW and Subscribe command is a command that provides aggregation options. To add these options to the `sow_and_subscribe` command, set the `Options` header on the `Command`. | Header | Comment | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `topic`

(required) | Sets the topic to query and subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` | A comma-delimited set of options for this command. For an aggregated subscription, the options provide the `projection` and `grouping` for the command. An aggregated SOW and Subscribe can include options in addition to `projection` and `grouping`. The most common additional options are: | --- # Cookbook: SOW Delete This section presents common recipes for sending a `sow_delete` command using the `Command` or `Message` interfaces. This section provides information on how to configure the request to AMPS. You can adapt this information to your application and the specific interface you are using. *Command:* `sow_delete` ## Delete All Records in a SOW To delete all records in a SOW, provide a filter that evaluates to TRUE for every record in the SOW. By convention, 60East recommends `1=1` for the filter. | Header | Comment | | ----------------------------- | ------------------------------------------------------------------------------------------------------ | | `topic`

(required) | Sets the topic from which to remove records. | | `filter`

(required) | A filter specifying the messages to remove. By convention, use `1=1` to remove all records in the SOW. | ## Delete SOW Records Matching a Filter To delete the records that match a particular filter, provide the filter in the `sow_delete` command. | Header | Comment | | ----------------------------- | -------------------------------------------- | | `topic`

(required) | Sets the topic from which to remove records. | | `filter`

(required) | A filter specifying the messages to remove. | ## Delete a Specific Message by Data To delete a specific message, provide the data for the message to delete. With this form of SOW delete, AMPS deletes the message that would have been updated if the data were provided as a publish message. Notice that this form of `sow_delete` relies on the `Key` definition in the SOW configuration, and is not generally useful with explicitly-keyed SOW topics. | Header | Comment | | ---------------------------- | -------------------------------------------- | | `topic`

(required) | Sets the topic from which to remove records. | | `data`

(required) | The message to remove. | ## Delete Specific Messages using Keys To delete specific messages using SOW keys, provide the SOW keys for the message to delete. | Header | Comment | | ------------------------------ | ----------------------------------------------------------------------- | | `topic`

(required) | Sets the topic from which to remove records. | | `sow_keys`

(required) | A comma-delimited list of SOW Keys that specify the messages to remove. | ## Acknowledge Messages from a Queue To acknowledge messages from an AMPS queue, provide the bookmarks for the messages to acknowledge. Notice that this is the only form of the `sow_delete` command that can acknowledge messages from a queue, and that this form of `sow_delete` is not accepted for topics that are not queue topics. | Header | Comment | | ------------------------------- | ----------------------------------------------------------------------------- | | `topic`

(required) | Sets the topic that contains the messages to acknowledge. | | `bookmark`

(required) | A comma-delimited list of Bookmarks that specify the messages to acknowledge. | --- # Cookbook: SOW This section presents common recipes for querying a SOW topic in AMPS using the `Command` or `Message` interfaces. This section provides information on how to configure the request to AMPS. You can adapt this information to your application and the specific interface you are using. _Command:_ `sow` ## Basic SOW Query In its simplest form, a SOW query needs only the topic to query. | Header | Comment | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`Topic`

(required)

| Sets the topic to query. The SOW query returns all messages in the SOW. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | ## SOW Query with Options In its simplest form, a SOW query needs only the topic to query. To add options to the query, set the `Options` header on the `Command`. | Header | Comment | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`Topic`

(required)

| Sets the topic to query. The SOW query returns all messages in the SOW. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `Options` | A comma-delimited set of options for this command. See the [`sow`](../commands-to-amps/query-subscribe/sow) section for a description of supported options. | ## SOW Query with Ordered Results In its simplest form, a SOW query needs only the topic to query. To return the results in a specific order, provide an ordering expression in the `OrderBy` header. | Header | Comment | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`Topic`

(required)

| Sets the topic to query. The SOW query returns all messages in the SOW. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `OrderBy` |

Orders the results returned as specified. Requires a comma-separated list of identifiers of the form:

`/field/[ASC \| DESC]`

For example, to sort in descending order by `orderDate` so that the most recent orders are first, and ascending order by `customerName` for orders with the same date, you might use a specifier such as:

`/orderDate DESC, /customerName ASC`

| ## SOW Query with TopN Results In its simplest form, a SOW query needs only the topic to query. To return only a specific number of records, provide the number of records to return in the `top_n` option. | Header | Comment | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`Topic`

(required)

| Sets the topic to query. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `Options` | The `top_n` option specifies the maximum number of records to return from the query. AMPS uses the `OrderBy` header to determine the order of the records. If no `OrderBy` header is provided, records are returned in an indeterminate order. In most cases, using an `OrderBy` header when you use the `TopN` header will guarantee that you get the records of interest. For example, to specify that the first ten records are returned, you would use the option `top_n=10` | | `OrderBy` | Orders the results returned as specified. Requires a comma-separated list of identifiers of the form: `/field/[ASC \| DESC]`. For example, to sort in descending order by `orderDate` so that the most recent orders are first, and ascending order by `customerName` for orders with the same date, you might use a specifier such as: `/orderDate DESC, /customerName ASC` | ## SOW Query with Content Filter To provide a content filter on a SOW query, set the `Filter` property on the command. The _AMPS User Guide_ provides details on the filter syntax. | Header | Comment | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | |

`Topic`

(required)

| Sets the topic to query. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `Filter` | Sets the content filter to be applied to the query. Only messages that match the content filter will be returned in response to the query. | ## Historical SOW Query To create a historical SOW query, set the `Bookmark` property on the command. The property can be either a specific bookmark or a timestamp. The _AMPS User Guide_ provides details on creating timestamps. This command is only supported on SOW topics that have `History` enabled. | Header | Comment | | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`Topic`

(required)

| Sets the topic to query. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `Bookmark` | Sets the historical point in the SOW at which to query. The query returns the saved state of the records in the SOW as of the point in time specified in this header. | ## Historical SOW Query with Content Filter To create a historical SOW query, set the `Bookmark` property on the command. The property can be either a specific bookmark or a timestamp. The _AMPS User Guide_ provides details on creating timestamps. To add a filter to the query, set the `Filter` property on the command. The _AMPS User Guide_ provides details on the filter syntax. This command is only supported on SOW topics that have `History` enabled. | Header | Comment | | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`Topic`

(required)

| Sets the topic to query. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `Bookmark` | Sets the historical point in the SOW at which to query. The query returns the saved state of the records in the SOW as of the point in time specified in this header. | | `Filter` | Sets the content filter to be applied to the query. Only messages that match the content filter will be provided to the query. | ## SOW Query for Specific Records AMPS allows a consumer to query for specific records as identified by a set of `SowKeys`. For topics where AMPS assigns the `SowKey`, the `SowKey` for the record is the AMPS-assigned identifier. For topics configured to require a user-provided `SowKey`, the `SowKey` for the record is the original key provided when the record was published. The _AMPS User Guide_ provides more details on SOW keys. | Header | Comment | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`Topic`

(required)

| Sets the topic to query. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `SowKeys` | A comma-delimited list of `SowKey` values. AMPS returns only the records specified in this list. For example, a valid format for a list of keys would be: `1853097931817257202,104027799402 01650075,22363879930342650852` | ## SOW Query with Pagination AMPS allows a consumer to page through records in the SOW using the `top_n` and `skip_n` options. With this approach, the application uses the `top_n` option to limit the number of records returned to a single page worth of records. The application uses the `skip_n` option to set the number of records to skip ahead to get to the page to display, and sets the `OrderBy` header to specify the ordering for the records. For example, if 10 records fit on a page, and the pages are ordered by the `ClientName` field of the records, to display the fourth page, the application would set `top_n` to `10`, `skip_n` to `30` (to skip the first three pages of records), and `OrderBy` to `/ClientName`. | Header | Comment | | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`Topic`

(required)

| Sets the topic to query. The topic specified must be a literal topic name. Pagination is not supported with regular expression subscriptions. | | `OrderBy` | Orders the results returned as specified. Requires a comma-separated list of identifiers of the form: `/field/[ASC \| DESC]` . For example, to sort in descending order by `orderDate` so that the most recent orders are first, and ascending order by `customerName` for orders with the same date, you might use a specifier such as: `/orderDate DESC, /customerName ASC` | | `Options` | An options string that sets the `top_n` and `skip_n` values for this query. For example, to skip 100 records and return the next 10 records, use an options string such as: `top_n=10,skip_n=100` | ## Aggregated SOW Query An aggregated SOW query is a SOW query that provides aggregation options in the options field. To add these options to the subscription, set the `Options` header on the `Command`. | Header | Comment | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to query. The SOW query returns all messages in the SOW. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` | A comma-delimited set of options for this command. For an aggregated SOW query, the options provide the `projection` and `grouping` for the aggregation. See the [`sow`](../commands-to-amps/query-subscribe/sow) section for a description of supported options. | --- # Cookbook: Subscribe This section presents common recipes for subscribing to a topic in AMPS using the `Command` or `Message` interfaces. This section provides information on how to configure the request to AMPS. You can adapt this information to your application and the specific interface you are using. _Command:_ `subscribe` ## Basic Subscription In its simplest form, a subscription needs only the topic to subscribe to. | Header | Comment | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

`topic`

(required)

| Sets the topic to subscribe to. All messages from the topic will be delivered on this subscription. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | ## Subscription with Options In its simplest form, a subscription needs only the topic to subscribe to. To add options to the subscription, set the options (`opts`) header on the `Command`. | Header | Comment | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

`topic`

(required)

| Sets the topic to subscribe to. All messages from the topic will be delivered on this subscription. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` | A comma-delimited set of options for this command. See the [`subscribe`](../commands-to-amps/query-subscribe/subscribe) section for a description of supported options. | ## Subscription with Select List In its simplest form, a subscription needs only the topic to subscribe to. To use a select list with the subscription, set the options (`opts`) header on the `Command` to the select list specifier. | Header | Comment | | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to subscribe to. All messages from the topic will be delivered on this subscription. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` | A comma-delimited set of options for this command. See the [`subscribe`](../commands-to-amps/query-subscribe/subscribe) section for a description of supported options. For example, to remove all fields except for `/id` and `/ticker`, you might use an option string of `select=[-/,+/id,+/ticker]`. | ## Subscription with Content Filter To provide a content filter on a subscription, set the `filter` property on the command. The _AMPS User Guide_ provides details on the filter syntax. | Header | Comment | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | `topic` (required) | Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `filter` | Sets the content filter to be applied to the subscription. Only messages that match the content filter will be provided to the subscription. | ## Conflated Subscription to a SOW Topic To request conflation on a subscription, set the options (`opts`) property on the command to specify the conflation interval. When the topic has a SOW (including a view or a conflated topic), there is no need to provide a `conflation_key`. | Header | Comment | | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` | A comma-separated list of options for the command. Set the conflation interval in the options. For example, to set a conflation interval of 250 milliseconds, provide the following option: `conflation=250ms`. To set the conflation interval to 1 minute, provide an option of: `conflation=1m` . | ## Conflated Subscription to a Topic with No SOW To request conflation on a subscription, set the options (`opts`) property on the command to specify the conflation interval. When the topic is not in the State of the World, you must provide a `conflation_key`. | Header | Comment | | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` | A comma-separated list of options for the command. Set the conflation interval and the fields that determine a unique message in the options. For example, to set a conflation interval of 250 milliseconds for messages that have the same value for `/id`, provide the following option: `conflation=250ms,conflation_key= [/id]`. To set the conflation interval to 1 minute for messages that are the same, as determined by a combination of `/customerId` and `/issueNumber`, provide an option of: `conflation=1m, conflation_key=[/customerId,/issueNumber]`. | ## Bookmark Subscription To create a bookmark subscription, set the `bookmark` property on the command. The value of this property can be either a specific bookmark, a timestamp, or one of the client-provided constants. The _AMPS User Guide_ provides details on creating timestamps. Notice that the `MOST_RECENT` constant tells the AMPS client to find the appropriate message in the client bookmark store and begin the subscription at that point. In this case, the client sends that bookmark value to AMPS. The `Bookmark` option is only supported for topics that are recorded in an AMPS transaction log. | Header | Comment | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `Bookmark` | Sets the point in the transaction log at which the subscription will begin. The bookmark provided can be a specific AMPS bookmark, a timestamp, or one of the client-provided constants. AMPS also accepts a comma-delimited list of bookmarks. In this case, AMPS begins the subscription from whichever of the bookmarks is earliest in the transaction log. | ## Rate Controlled Bookmark Subscription To create a bookmark subscription, set the `bookmark` property on the command. The value of this property can be either a specific bookmark, a timestamp, or one of the client-provided constants. The _AMPS User Guide_ provides details on creating timestamps. Notice that the `MOST_RECENT` constant tells the AMPS client to find the appropriate message in the client bookmark store and begin the subscription at that point. In this case, the client sends that bookmark value to AMPS. The `bookmark` option is only supported for topics that are recorded in an AMPS transaction log. To manage the message delivery rate to a bookmark subscription, set the `rate` option specifying the rate at which to replay messages. | Header | Comment | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `bookmark` | Sets the point in the transaction log at which the subscription will begin. The bookmark provided can be a specific AMPS bookmark, a timestamp, or one of the client-provided constants. AMPS also accepts a comma-delimited list of bookmarks. In this case, AMPS begins the subscription from whichever of the bookmarks is earliest in the transaction log. | | `opts` | A comma-separated list of options for the command. To control the rate at which AMPS delivers messages, the options for the command must include a rate specifier. For example, to specify a limit of 750 messages per second, include `rate=750` in the options string. | ## Rate Controlled Bookmark Subscription with Maximum Gap To create a bookmark subscription, set the `bookmark` property on the command. The value of this property can be either a specific bookmark, a timestamp, or one of the client-provided constants. The _AMPS User Guide_ provides details on creating timestamps. Notice that the `MOST_RECENT` constant tells the AMPS client to find the appropriate message in the client bookmark store and begin the subscription at that point. In this case, the client sends that bookmark value to AMPS. The `bookmark` option is only supported for topics that are recorded in an AMPS transaction log. The `rate` option specifies the rate at which to replay messages, while the `rate_max_gap` option specifies the maximum amount of time for AMPS to allow between messages. | Header | Comment | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

`topic`

(required)

| Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `bookmark` | Sets the point in the transaction log at which the subscription will begin. The bookmark provided can be a specific AMPS bookmark, a timestamp, or one of the client-provided constants. AMPS also accepts a comma-delimited list of bookmarks. In this case, AMPS begins the subscription from whichever of the bookmarks is earliest in the transaction log. | | `opts` | A comma-separated list of options for the command. To control the rate at which AMPS delivers messages, the options for the command must include a rate specifier. For example, to specify that AMPS replays no faster than twice the original rate, include `2X` in the options string. To specify that AMPS will go no more than 3 seconds without producing a message, regardless of the original replay timing, use a `rate_max_gap` of `3s`. To provide the options described above, you would use the options string: `rate=2X,rate_max_gap=3s`. | ## Bookmark Subscription with Completed Acknowledgment To create a bookmark subscription, set the `bookmark` property on the command. The value of this property can be either a specific bookmark, a timestamp, or one of the client-provided constants. The _AMPS User Guide_ provides details on creating timestamps. Notice that the `MOST_RECENT` constant tells the AMPS client to find the appropriate message in the client bookmark store and begin the subscription at that point. In this case, the client sends that bookmark value to AMPS. The `bookmark` option is only supported for topics that are recorded in an AMPS transaction log. To receive acknowledgment messages to a bookmark subscription, set the `ack_type` on the command specifying the type of acknowledgments to receive. In this case, be sure to include the `completed` type. | Header | Comment | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

`topic`

(required)

| Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `bookmark` | Sets the point in the transaction log at which the subscription will begin. The bookmark provided can be a specific AMPS bookmark, a timestamp, or one of the client-provided constants. AMPS also accepts a comma-delimited list of bookmarks. In this case, AMPS begins the subscription from whichever of the bookmarks is earliest in the transaction log. | | `ack_type` | Sets the acknowledgment messages that AMPS returns. The AMPS clients typically request a `processed` acknowledgment for a subscription to verify whether the subscription succeeded. To receive a `completed` acknowledgment, which indicates the point at which replay is complete for the subscription, include `completed` in the set of acknowledgments requested. AMPS will return a message with a command type of `ack` and an ack type of `completed` at that point in the subscription. | ## Bookmark Subscription with Content Filter To create a bookmark subscription, set the `bookmark` property on the command. The property can be either a specific bookmark, a timestamp, or one of the client-provided constants. The _AMPS User Guide_ provides details on creating timestamps. Notice that the `MOST_RECENT` constant tells the AMPS client to find the appropriate message in the client bookmark store and begin the subscription at that point. In this case, the client sends that bookmark value to AMPS. To add a filter to a bookmark subscription, set the `filter` property on the command. The _AMPS User Guide_ provides details on the filter syntax. | Header | Comment | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `bookmark` | Sets the point in the transaction log at which the subscription will begin. The bookmark provided can be a specific AMPS bookmark, a timestamp, or one of the client-provided constants. AMPS also accepts a comma-delimited list of bookmarks. In this case, AMPS begins the subscription from whichever of the bookmarks is earliest in the transaction log. | | `filter` | Sets the content filter to be applied to the subscription. Only messages that match the content filter will be provided to the subscription. | ## Entering a Bookmark Subscription In the Paused State To pause a bookmark subscription, you must provide the subscription ID and the `pause` option on a `subscribe` command. This option is used to enter a number of subscriptions at the same point in the transaction log. The subscriptions can then be resumed together and will progress through the transaction log together. | Header | Comment | | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | |

`sub_id`

(required)

| The subscription ID to enter in a paused state. | |

`opts`

(required)

| A comma-delimited list of options for the command. To enter a paused subscription, the options must include `pause`. | ## Starting One or More Paused Bookmark Subscriptions Starting bookmark subscriptions that are currently paused requires you to provide the subscription ID and a `resume` option on a `subscribe` command. | Header | Comment | | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | |

`SubId`

(required)

| A comma-delimited list of subscription IDs to start. | | `Options` | A comma-delimited list of options for the command. To resume a subscription, the options must include `resume`. | ## Replacing the Filter on a Subscription To replace the content filter on a subscription, provide the `sub_id` of the subscription to be replaced, add the `replace` option, and set the `filter` property on the command with the new filter. The _AMPS User Guide_ provides details on the filter syntax. | Header | Comment | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | |

`sub_id`

(required)

| The identifier for the subscription to update. The `sub_id` is the `sub_id` of the original `subscribe` command (or the `cmd_id` if a `sub_id` was not provided). | | `opts` | A comma-separated list of options. To replace the filter on a subscription, include `replace` in the list of options. | | `filter` | Sets the content filter to be applied to the subscription. Only messages that match the content filter will be provided to the subscription after the filter is replaced. | ## Subscribing to a Queue and Requesting a max\_backlog To subscribe to a queue and request a `max_backlog` greater than `1`, use the options (`opts`) field of the `subscribe` command to set the requested `max_backlog`. | Header | Comment | | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

`topic`

(required)

| Sets the topic to subscribe to. The topic specified can be the literal topic name, or a regular expression that matches multiple topics. | | `opts` | A comma-separated list of options. To request a value for the `max_backlog`, pass the value in the options as follows: `max_backlog= NN`. For example, to request a `max_backlog` of 7, your application would pass the following option: `max_backlog=7`. | --- # Responses from AMPS This section describes messages that AMPS returns to applications. The AMPS Client libraries handle the details of parsing the returned message and delivering it to the application as needed. The application is only responsible for interpreting the parsed message and responding as necessary. ### Content Messages AMPS provides three types of message that contain message content: * [`publish`](/docs/amps-command-reference/messages-from-amps/content-publish) messages return data from a topic as it is published, in order, whether the data is being published live, or is the result of a replay. * [`sow`](/docs/amps-command-reference/messages-from-amps/content-sow) messages return data from a SOW query. These messages return the state of messages that are current as of the time for the query. By default, the messages are returned without regard to the order in which the messages were published. A query can specify the order of the returned messages based on the data within the message by including the `OrderBy` header on the SOW query. * [`oof`](/docs/amps-command-reference/messages-from-amps/oof) messages indicate that a content message no longer matches a subscription. These messages are sent to a client in order. ## Ack Messages AMPS provides information to the application and AMPS client about the status of commands using acknowledgment, or `ack`, messages. A command to AMPS must explicitly request acknowledgment to receive a response from AMPS. By default, the AMPS client libraries request acknowledgments as needed to detect failures: see the _AMPS User Guide_ chapter on [Command Acknowledgments](/docs/amps-user-guide/acks) and the _Developer Guide_ for each client language for details. In some cases, an application may want additional information about the state of a command, and may request an `ack` explicitly. For example, an application may want to know how many records would be returned by a particular query. In that case, a common technique is to request a `stats` (statistics) `ack`, while setting the number of data messages to be returned to `0` (using the `top_n=0` option). The `ack` messages section describes the contents of the `ack` messages returned by AMPS. ## Query Result Set Delimiters AMPS provides a pair of delimiters, [`group_begin`](/docs/amps-command-reference/messages-from-amps/group-begin-end) and [`group_end`](/docs/amps-command-reference/messages-from-amps/group-begin-end), that indicate when a query result set begins and ends. --- # ack: Status from Server The `ack` message returns status information from AMPS. AMPS does not create `ack` messages unless an acknowledgment is specifically requested. The exact meaning and content of ack messages depends on the command the requests the message. AMPS supports the following types of `ack` messages with the general semantics described below: | ack Type | Meaning | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `completed` |

An operation has completed.

For example, subscriptions that replay from the transaction log can produce a `completed` acknowledgment to indicate when transaction log replay has finished and further messages for the subscription are the result of new publishes.

| | `persisted` | Data has been persisted. | | `processed` |

AMPS has processed the command.

Notice that, depending on the command, AMPS may not have executed the command when this acknowledgment is produced.

| | `received` | AMPS has received the command, but has not yet processed it. | | `stats` |

Statistics for the command.

This acknowledgment is typically produced after the command has fully completed.

| ## Header Fields The following table contains the header fields returned in an `ack` message: | Field | Description | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` | Type of message. Always `ack`, as encoded by the protocol. | | `ack_type` | The type of acknowledgment. One of `completed`, `persisted`, `processed`, `received` or `stats`. | | `cmd_id` |

The command ID that this ack refers to. Clients can use this field to correlate the `ack` returned with the command being acknowledged.

This field will not be returned for acknowledgments that are conflated (`persisted` acknowledgments for publish commands when AMPS has a transaction log configured).

| | `status` | The status of the command. | | `reason` | The reason for the status, most often returned with a `failure` status to provide more detailed information about why the command failed. | ### `logon` Acknowledgment: Additional Fields When the `ack` message is produced in response to a `logon` command, the following additional header fields may be set: | Field | Description | | ------------ | ---------------------------------------------------------------------------------------------------------- | | `client_name` | The name of the client provided with the command. | | `seq` | The last message sequence number fully processed (and, if possible, safely persisted) for this client, as identified by the `client_name`. | | `bookmark` | The last bookmark from this client. | | `user_id` | `user-id` to use when the status is retry. | | `password` | `Password` to use when the status is retry. | | `version` | The version of the AMPS server. | ### `publish`, `delta_publish`: Additional Fields When the `ack` message is produced in response to a `publish` or `delta_publish` command, the following additional header fields may be set: | Field | Description | | ------------ | ---------------------------------------------- | | `seq` | The last message sequence number processed (and, if possible, safely persisted) for this client. | | `bookmark` | The last Bookmark persisted for this client. | ### `subscribe`, `delta_subscribe`: Additional Fields When the `ack` message is produced in response to a `subscribe` or `delta_subscribe` command, the following additional header fields may be set: | Field | Description | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sub_id` |

The subscription ID sent with the command, or the ID generated by AMPS if no `sub_id` was provided.

This field is not returned in `processed` acks.

| | `opts` |

Returned when the command is a `subscribe` to a queue.

Contains the following options:

`max_backlog` - Indicates the effective maximum backlog that the server has assigned for this subscription.

| | `bookmark` |

For a `completed` acknowledgment on a bookmark subscription, this indicates the point in the transaction log where the acknowledgment message was generated.

Notice that the message that corresponds to this bookmark need not be a message matched by the subscription.

| ### `unsubscribe`: Additional Fields When the `ack` message is produced in response to an `unsubscribe`, AMPS does not provide additional header fields. ### `sow`, `sow_and_subscribe`, `sow_and_delta_subscribe`: Additional Fields When the `ack` message is produced in response to a `sow`, `sow_and_subscribe`, or `sow_and_delta_subscribe`, the following additional header fields may be set: | Field | Description | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sub_id` | The `sub_id` sent with the `sow` command. | | `query_id` | The `query_id` sent with the `sow` command. | | `records_returned` |

The number of records returned by a SOW query.

This header field is present on `stats` acknowledgments.

| | `topic_matches` |

The total number of records compared across all matching SOW topics.

This header field is present on `stats` acknowledgments.

| | `matches` |

The number of records that match the topic regular expression and content filter.

This header field is present on `stats` acknowledgments.

| ### `sow_delete`: Additional Fields When the `ack` message is produced in response to a `sow_delete` the following additional header fields may be set: | Field | Description | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `query_id` | The `QueryId` sent with the `sow_delete` command. | | `records_deleted` |

The number of records deleted by the command.

This header field is present on `stats` acknowledgments.

| | `topic_matches` |

The total number of records compared across all matching SOW topics.

This header field is present on `stats` acknowledgments.

| | `matches` |

The number of records that match the topic regular expression and content filter.

This header field is present on `stats` acknowledgments.

| --- # publish: Content from Server AMPS returns a `publish` message to a client when a new message is published to AMPS that matches one of the subscriptions requested by the client. There are two ways that AMPS can generate publish messages: * _Single-origin_ Messages - For subscriptions to topics where AMPS can identify a single source for a publish message, AMPS provides information from that publish message to the subscriber. This applies to subscriptions to unpersisted topics, SOW topics, and conflated topic replicas. This does not include subscriptions to views (or conflated topics based on views), since views provide the ability to join multiple topics and aggregate over multiple messages. For conflated topics, the header information provided is the information provided with the message published to the subscriber. For messages produced by delta publish, AMPS will use the information provided on the delta publish except as noted in the table below. * _Synthetic_ Messages - In some cases, AMPS must provide a message that is constructed by the server. This happens for views, and for status messages from AMPS. AMPS provides different values in the header fields depending on the origin of the publish message. For synthetic messages, AMPS does not provide information on the origin of the message, since there may be multiple sources of the message or, in the case of status messages, no external source. Likewise, AMPS does not provide a `correlation_id`, since that header is set by the publisher for a specific message. ## Header Fields The following table contains the header fields returned in a `publish` message. Notice that the wire format of the message may abbreviate these field names or encode them differently: | Field | Description | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

Type of message.

Always `publish`, as encoded by the protocol.

| | `topic` | The topic the message was published to. | | `correlation_id` |

A publisher-provided string that is passed, verbatim, to subscribers.

If this header is not present, or the message is a synthetic message as described above, subscribers receive no value for the `correlation_id`.

The contents of this header must consist of characters that are legal in Base64 encoding.

For delta publishes, AMPS uses the `correlation_id` of the delta publish if one is present. If no `correlation_id` is present on the publish, AMPS uses the `correlation_id` of the existing message, if one is present.

If there is no `correlation_id` on the publish, and there is no `correlation_id` for the existing message, AMPS does not provide a `correlation_id`.

| | `user_id` |

The `user_id` of the client that published the message.

An authentication module may choose whether to allow subscribers to receive this information.

| | `sids` |

The set of subscription IDs that produced this message.

When a message matches multiple subscriptions, AMPS may produce a list of subscription IDs for all matching subscriptions.

This header is provided by AMPS. The AMPS Clients process this list and provide a single `sub_id` for each message provided to message handlers.

| | `bookmark` | The bookmark assigned to this message, if the message was persisted to a transaction log. | | `timestamp` |

An ISO-8601 datetime that notes the time the message was processed by this instance of AMPS.

The header is included if the client requested a `timestamp` for the subscription. The value returned has microsecond resolution. Fractions of a second are represented as decimal values.

| | `leaseperiod` | For messages received from a queue, the ISO-8601 datetime that indicates when the lease expires. | | `msg_len` | For messages that contain a message body (message data), the length of the message body. | | `sow_key` | If the message was from a topic that uses a SOW, the message includes the `sow_key` that AMPS uses to uniquely identify the message within the SOW. | --- # sow: Content from Server The `sow` message returns a record from the SOW. For more information, see the [State of the World (SOW) Topics](../../amps-user-guide/sow) and [Querying the State of the World (SOW)](../../amps-user-guide/sow-queries) chapters in the _AMPS User Guide._ ## Header Fields The following table contains the header fields returned in a `sow` message. Notice that the wire format of the message may abbreviate these field names or encode them differently: | Field | Description | | ----------- | ------------------------------------------------------------------------------------------ | | `cmd` |

Type of message.

Always `sow`, as encoded by the protocol.

| | `topic` | The topic from which the records were produced. | | `sow_key` | An AMPS-created identifier for this message. | | `batch_size` | The number of records returned in a single `sow` batch. | | `timestamp` | The time at which AMPS processed the message. This is returned in ISO-8601 format. This timestamp has microsecond precision. Fractions of a second are represented as a decimal value. | | `query_id` | The `QueryId` of the query that produced this message. | | `msg_len` | The length of the first SOW message in the data portion of this message. | ## Data The `sow` message contains data. The data for the message consists of up to `batch_size` messages, formatted as expected by the protocol. Each message contains its own header, with the following fields: | Field | Description | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sow_key` | An AMPS-created identifier for this message. | | `correlation_id` |

A user-provided string that will be passed, verbatim, to subscribers.

If this header is not present on the SOW record, subscribers receive no value for the `CorrelationId`.

The contents of this header must consist of characters that are legal in Base64 encoding.

| | `msg_len` | The length of the next SOW message in the data portion of this message. | --- # group_begin / group_end: Result Set Delimiters This section describes the query delimiters that indicate where the results of a query begin and end. ## group\_begin message The `group_begin` message marks the beginning of a set of records returned by a SOW query. For more information, see the [State of the World (SOW) Topics](../../amps-user-guide/sow) and [Querying the State of the World (SOW)](../../amps-user-guide/sow-queries) chapters in the _AMPS User Guide._ ### Header Fields The following table contains the header fields returned in a `group_begin` message. Notice that the wire format of the message may abbreviate these field names or encode them differently: | Field | Description | | --------- | -------------------------------------------------------------------------------------------------- | | `cmd` |

Type of message.

Always `group_begin`, as encoded by the protocol.

| | `query_id` | The `query_id` of the query that produced this message. If no explicit query ID was submitted, this will be the command ID of the command that ran the query. | ## group\_end message The `group_end` message marks the end of a set of records returned by a SOW query. For more information, see the [State of the World (SOW) Topics](../../amps-user-guide/sow) and [Querying the State of the World (SOW)](../../amps-user-guide/sow-queries) chapters in the _AMPS User Guide._ ### Header Fields The following table contains the header fields returned in a `group_end` message: | Field | Description | | --------- | ------------------------------------------------------------------------------------------------ | | `cmd` |

Type of message.

Always `group_end`, as encoded by the protocol.

| | `query_id` | The `query_id` of the query that produced this message. If no explicit query ID was submitted, this will be the command ID of the command that ran the query. | --- # oof: Content from Server The `oof` message indicates that a message in the SOW that previously matched the subscription is no longer in focus. For more information, see the [Out of Focus Messages (OOF)](../../amps-user-guide/oof) chapter in the _AMPS User Guide._ ## Header Fields The following table contains the header fields returned in an `oof` message. Notice that the wire format of the message may abbreviate these field names or encode them differently: | Field | Description | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmd` |

Type of message.

Always `oof`, as encoded by the protocol.

| | `topic` | The topic which contained the message that has gone out of focus. | | `msg_len` | The length of the message body. | | `sow_key` | An AMPS-created identifier for the message that has gone out of focus. | | `reason` |

The reason the message has gone out of focus.

Valid reasons include `deleted`, `expired`, `match`, and `entitlement`.

| | `sids` |

The subscription IDs of the subscriptions that produced this message.

The AMPS clients will provide this message to the handler registered for each of the subscriptions specified.

| | `correlation_id` |

A user-provided string that will be passed, verbatim, to subscribers.

If this header is not present on the SOW record that was deleted, subscribers receive no value for the `correlation_id`.

The contents of this header must consist of characters that are legal in Base64 encoding.

| ## Reason The `reason` field of the `oof` message explains why the message no longer matches the subscription. Possible values are: | Reason | Description | | ------------- | ------------------------------------------------------------------------------------------------------- | | `deleted` | The message was deleted. | | `expired` | The message expired. | | `match` | The message no longer matches the filter for the subscription or (for paginated subscriptions) no longer matches the pagination window. | | `entitlement` | The access to the messaged changed, such that the user no longer has permission to receive the message. | ## Data The `oof` message contains the updated message that caused the message to go out of focus when the reason is `match`. Otherwise, this connection does not have permission to see the updated message (if the reason is `entitlement`) or there is no updated message to provide (if the reason is `deleted` or `expired`). The previous message will be provided for reasons other than `match`. --- # Protocol Reference This section contains information on how different protocols represent AMPS headers. The AMPS clients handle constructing and parsing AMPS headers. However, understanding the format of command can be useful when inspecting trace level logs or network traffic captures. :::tip The `amps` protocol is the recommended protocol for all application development. The other protocols are included for legacy compatibility. While they will remain supported for their current functionality, the legacy protocols will not be enhanced further. These protocols have limitations in current versions of AMPS, and future features of AMPS may require the `amps` protocol. (Notice that websocket connections use the websocket transport framing, but interact with AMPS using the `amps` protocol.) ::: --- # AMPS Protocol The following table describes how the headers in the `amps` protocol are processed by AMPS. Current releases of AMPS use a simplified JSON-like syntax for the `amps` protocol. #### AMPS Message Header - Sorted by Name | AMPS Header Field | Abbreviation | Name | | ----------------- | ------------ | --------------------- | | ack\_type | a | `AckType` | | password | pw | `Password` | | bookmark | bm | `Bookmark` | | batch\_size | bs | `BatchSize` | | client\_name | | `ClientName` | | cmd | c | `Command` | | cmd\_id | cid | `CommandId` | | correlation\_id | x | `CorrelationId` | | data\_only | | `DataOnly` | | expiration | e | `Expiration` | | filter | f | `Filter` | | gseq | | `GroupSequenceNumber` | | heartbeat | | `Heartbeat` | | leaseperiod | lp | `LeasePeriod` | | matches | | `Matches` | | msg\_len | l | `MsgLen` | | max\_msgs | | `MaximumMessages` | | opts | o | `Opts` | | orderby | | `OrderBy` | | query\_id | | `QueryID` | | reason | | `Reason` | | records\_deleted | | `RecordsDeleted` | | records\_inserted | | `RecordsInserted` | | records\_returned | | `RecordsReturned` | | records\_updated | | `RecordsUpdated` | | seq | s | `Sequence` | | send\_empty | | `SendEmpty` | | send\_keys | | `SendKeys` | | send\_oof | | `SendOutOfFocus` | | sow\_key | k | `SowKey` | | sow\_keys | | `SowKeys` | | status | | `Status` | | sub\_id | | `SubscriptionId` | | sids | | `SubscriptionIds` | | src | | `Src` | | timeout\_interval | | `TimeoutInterval` | | timestamp | ts | `TransmissionTime` | | top\_n | | `TopNRecordsReturned` | | topic\_matches | | `TopicMatches` | | topic | t | `Topic` | | use\_ns | | `UseNamespaces` | | user\_id | | `UserId` | | version | v | `Version` | ### Header Fields - Reference This section provides a reference to the header fields that have been used in AMPS messages. Not all headers are present on all outgoing messages, and not all headers are processed on incoming commands. See the detailed message descriptions in this guide for descriptions of which fields are used on a given command or a given response from AMPS. Further, although this table lists the header fields defined in every protocol, current versions of AMPS may not use every header defined in this table (see the detailed message descriptions for which fields are relevant for a given command or response). | Name | Type | Definition | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AckType` | string | Acknowledgment type for the given command. | | `BatchSize` |

integer

Default is 1 when not present.

| Specifies the number of messages that are batched together when returning a query result. | | `Bookmark` | string | A client-originated identifier used to mark a location in journaled messages. | | `ClientName` | string |

Used to identify a client.

Useful for publishers that wish to identify the source of a publish, client status messages and for client heartbeats.

Can be set with logon command.

| | `Command` |

One of:

| Command to be executed. | | `CommandId` | string |

Client-specified command ID.

The CmdId is returned by the engine in responses to commands to allow the client to correlate the response to the command.

| | `CorrelationId` | string, base64 encoded characters only | Opaque token set by an application and returned with the message. | | `DataOnly` | Boolean (`true` or `false`) | If `true`, only send raw data to subscriber for a matching publish message, i.e. do not include FIX/NVFIX envelope. | | `Expiration` | integer (seconds) | SOW expiration time if used in `publish`. | | `Filter` | string, should wrap in CDATA | Content filter expression. | | `GracePeriod` | integer (milliseconds) | Grace period after heartbeat interval is exceeded before client is considered in error state. | | `GroupSequenceNumber` | integer | Group Sequence Number returned with each batch message of a SOW response. | | `Heartbeat` |

One of:

| Heartbeat command. | | `LeasePeriod` | timestamp | For messages from a queue, the time at which the lease expires. | | `LogLevel` |

One of:

|

Set the log level.

Deprecated

Not used in current versions of AMPS.

| | `Matches` | integer | Returned in the acknowledgment to a SOW query that indicates number of matches. | | `MaximumMessages` | integer greater than zero | Specifies the maximum number of messages within a batch publish. | | `MessageID` | string, e.g. `MAMPS–XYZ` | Set by AMPS engine to tag every incoming message. | | `MessageLength` | integer | Sent with messages that have data (a message body) to indicate the number of bytes in the message body. | | `MessageType` |

string

One of the configured message types in AMPS.

| Message type. | | `MsgLen` | integer | Sent with messages that have data (a message body) to indicate the number of bytes in the message body. | | `Opts` | string | A comma-delimited list of options on a specific command. | | `Password` | string | Password used to authenticate with an AMPS server. | | `QueryID` | string | SOW query identifier set by client to identify a query. | | `Reason` | string | The failure message that appears when an acknowledgment returns a `status` or `failure`. | | `RecordsDeleted` | integer | Used in conjunction with the `stats` acknowledgment, this is the number of records deleted from the SOW with a `sow_delete` command. | | `RecordsInserted` | integer | Used in conjunction with the `stats` acknowledgment, this is the number of records inserted into the SOW. | | `RecordsUpdated` | integer | Used in conjunction with the `stats` acknowledgment, this is the number of records updated in the SOW. | | `RecordsReturned` | integer | Returned in the acknowledgment to an SOW query that indicates number of records in the store. | | `SendEmpty` | Boolean (`true` or `false`); default is `true` | If `true`, empty messages that are published will be forwarded to matching subscriptions. | | `SendKeys` | Boolean (`true` or `false`) | Option to instruct AMPS that a client would like to receive the `SowKey`(s) back. | | `SendOutOfFocus` | Boolean (`true` or `false`) | If `true`, Out of Focus messages are sent for the SOW query. | | `SendSubscriptionIDs` | Boolean (`true` or `false`) | If `false`, subscription identifiers will not be sent for a matched message. | | `Sequence` | integer greater than zero |

An integer that corresponds to the publish message sequence number.

For more information see the [Replicating Messages Between Instances](../../amps-user-guide/replication) chapter in the AMPS User Guide.

| | `SowKey` |

string containing the digits of an unsigned long for AMPS-generated SOW keys

arbitrary string in the base64 character set for user-provided SOW keys

|

A SowKey will accompany each message returned in an SOW batch.

A SowKey may also be added to messages coming in on a subscription when the published message matches a record in the SOW.

A publish command may contain a SOW key if the SOW for the topic is configured to accept user-provided SOW keys.

| | `SowKeys` | comma-separated list of `SowKey` values | Comma-separated list of `SowKey` values. | | `Status` |

One of:

| Used to indicate client status when client is monitored for heartbeats. | | `SubscriptionId` | string, e.g. `SAMPS-XYZ` | The subscription identifier set by server when processing a subscription. | | `SubscriptionIds` | string | Comma-separated list of `SubIds` sent from AMPS engine to identify which client subscriptions match a given publish message. | | `TimeoutInterval` | integer | Used in conjunction with the heartbeat interval to set the timeout interval for a publisher. | | `TopNRecordsReturned` | unsigned integer |

The number of records to return.

Note: If TopN is not equally divisible by the BtchSz, then more records will be returned so that the total number of records is equally divisible by the BtchSz setting.

| | `Topic` | string | Topic | | `TopicMatches` | integer | Returned in the acknowledgment to an SOW query that indicates number of topic matches. | | `TransmissionTime` | ISO-8601 date-time | Timestamp - Optionally set to time the message was processed by the server. | | `UseNamespaces` | Boolean (`true` or `false`) | Use SOAP XML namespaces in all messages from the AMPS engine. | | `UserId` | string | Used to identify the user ID of a command. | | `Version` | string | Contains the version of the AMPS server. | --- # Legacy Protocols Reference ### FIX/NVFIX Protocol The following table describes how the headers in the `fix` and `nvfix` protocols are processed by AMPS. #### FIX/NVFIX Message Header - Sorted by Value | FIX/NVFIX Header Field | AMPS Protocol Equivalent | | -------------------------- | ------------------------------------------------------------------------------- | | 20000 | `cmd` | | 20001 | `cmd_id` | | 20002 | `client_name` | | 20003 | `user_id` | | 20004 | `timestamp` | | 20005 | `topic` | | 20006 | `filter` | | 20007 | `message_type` | | 20008 | `ack_type` | | 20009 | `sub_id` | | 20011 | `version` | | 20012 | `expiration` | | 20013 |

N/A -- obsolete header

(`SendSubscriptionIDs`)

| | 20014 | `data_only` | | 20015 | `heartbeat` | | 20016 | `timeout_interval` | | 20017 | `lease_period` | | 20018 | `status` | | 20019 | `query_id` | | 20020 | `send_oof` | | 20021 |

N/A -- obsolete header

(`LogLevel`)

| | 20022 | `use_ns` | | 20023 | `batch_size` | | 20025 | `top_n` | | 20029 | `send_empty` | | 20031 | `max_msgs` | | 20032 | `sow_keys` | | 20033 | `send_keys` | | 20034 | `src` | | 20035 | `correlation_id` | | 20036 | `seq` | | 20037 | `bookmark` | | 20038 | `password` | | 20039 | `opts` | | 20052 | `records_inserted` | | 20053 | `records_updated` | | 20054 | `records_deleted` | | 20055 | `records_returned` | | 20056 | `topic_matches` | | 20057 | `matches` | | 20058 | `msg_len` | | 20059 | `sow_key` | | 20060 | `gseq` | | 20061 | `sids` | | 20062 | `reason` | | 20063 |

N/A -- obsolete header

(`MessageID`)

| | 20074 |

`correlation_id`

(used for AMPS/ClientStatus messages)

| #### FIX/NVFIX Message Header - Sorted by Name | FIX/NVFIX Header Field | AMPS Protocol Field | | -------------------------- | ----------------------------------------------------------------------- | | 20008 | `ack_type` | | 20037 | `bookmark` | | 20023 | `batch_size` | | 20002 | `client_name` | | 20000 | `cmd` | | 20001 | `cmd_id` | | 20035 | `correlation_id` | | 20014 | `data_only` | | 20012 | `expiration` | | 20006 | `filter` | | 20017 | `lease_period` | | 20060 | `gseq` | | 20015 | `heartbeat` | | 20017 | `leaseperiod` | | 20021 |

N/A -- obsolete header

(`LogLevel`)

| | 20057 | `matches` | | 20063 |

N/A -- obsolete header

(`MessageID`)

| | 20058 | `msg_len` | | 20007 | `message_type` | | 20031 | `max_msgs` | | 20039 | `opts` | | 20038 | `password` | | 20019 | `query_id` | | 20062 | `reason` | | 20054 | `records_deleted` | | 20053 | `records_inserted` | | 20055 | `records_returned` | | 20036 | `seq` | | 20029 | `send_empty` | | 20033 | `send_keys` | | 20020 | `send_oof` | | 20013 |

N/A -- obsolete header

(`SendSubscriptionIDs`)

| | 20059 | `sow_key` | | 20032 | `sow_keys` | | 20034 | `src` | | 20018 | `status` | | 20009 | `sub_id` | | 20061 | `sids` | | 20016 | `timeout_interval` | | 20025 | `top_n` | | 20056 | `topic_matches` | | 20005 | `topic` | | 20004 | `timestamp` | | 20022 | `use_ns` | | 20003 | `user_id` | ### XML Protocol The following table describes how the headers in the `xml` protocol are processed by AMPS. #### XML Message Header - Sorted by Name | XML Header Field | AMPS Protocol Equivalent | | -------------------- | ---------------------------------------------------------------- | | AckTyp | `ack_type` | | BkMrk | `bookmark` | | BtchSz | `batch_size` | | ClntName | `client_name` | | Cmd | `cmd` | | CmdId | `cmd_id` | | DatOnly | `data_only` | | Expn | `expiration` | | Fltr | `filter` | | GrcPrd |

N/A -- obsolete

(`GracePeriod`)

| | GrpSqNum | `gseq` | | Hrtbt | `heartbeat` | | LeasePeriod | `lease_period` | | LogLvl |

N/A -- obsolete

(`LogLevel)`

| | Matches | `matches` | | MsgId | `MessageID` | | MsgLen | `msg_len` | | MsgTyp | `message_type` | | MxMsgs | `max_msgs` | | Opts | `opts` | | PW | `password` | | QId | `query_id` | | Reason | `reason` | | RecordsDeleted | `records_deleted` | | RecordsReturned | `records_returned` | | Seq | `seq` | | SndEmpty | `send_empty` | | SndKeys | `send_keys` | | SndOOF | `send_oof` | | SndSubIds |

N/A -- obsolete

(`SendSubscriptionIDs)`

| | SowKey | `sow_key` | | SowKeys | `sow_keys` | | Status | `status` | | SubId | `sub_id` | | SubIds | `sids` | | TmIntvl | `timeout_interval` | | TopN | `top_n` | | TopicMatches | `topic_matches` | | Tpc | `topic` | | TxmTm | `timestamp` | | UseNS | `use_ns` | | UsrId | `user_id` | | Version | `version` | --- # Publishing --- # AMPS Evaluation Guide Thank you for choosing the Advanced Message Processing System (AMPS) from 60East for evaluation. This guide provides information to help you evaluate AMPS for your application. This guide covers the following topics: | Topic | Description | | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | [Introduction](intro) | General introduction to evaluating AMPS. | | [Evaluation and Development With AMPS](evaluation\_and\_development) | Description of the suggested evaluation process and how to get started with developing applications with AMPS. | | [Tips on Measuring Performance](perf) | Performance measurement guidance and considerations. | | [Next Steps](next-steps) | Suggested paths after reading this guide. | --- # Evaluation and Development with AMPS AMPS runs on any 64-bit Linux system. For best performance in a development environment, 60East recommends that the system have a minimum of 4GB of memory available. For basic functional evaluation and development, AMPS runs well in a virtual machine, in a container, or in a WSL2 shell on Windows, as well as on a Linux host. The [Introduction to AMPS](/docs/intro-guide/intro) includes information on how to set up a basic development environment for AMPS. ## Product Overview AMPS is designed to help you quickly and easily develop and deploy data-intensive applications with demanding requirements for low latency and high performance. AMPS takes a nontraditional approach to messaging, storage, and analytics that is designed from the ground up for streaming data and highly-parallelized multicore systems. AMPS is based on an incredibly fast messaging engine that supports multiple messaging paradigms, as well as providing persistent current value caching (effectively, an integrated database), content filtering and continuous query, historical replay, aggregation and analytics, message enrichment, focus tracking, partial updates and change tracking, and more. Furthermore, AMPS is designed and engineered specifically for next generation computing environments. The architecture, design and implementation of AMPS allows the exploitation of parallelism inherent in emerging multi-socket, multi-core commodity systems and the low-latency, high-bandwidth of 10Gb Ethernet and faster networks. AMPS is designed to detect and take advantage of the capabilities of the hardware of the system on which it runs. AMPS was designed to improve performance and reduce latency in real-world messaging deployments by focusing on the entire lifetime of a message from the message's origin to the time at which a subscriber takes action on the message. AMPS considers the full message lifetime, rather than just the "in flight" time, and allows you to optimize your applications to conserve network bandwidth and subscriber CPU utilization -- typically the first elements of a system to reach the saturation point in real messaging systems. ## Understanding AMPS Features and Scenarios For an overview of the features of AMPS, see the [Overview of AMPS](/docs/intro-guide/product\_overview) in the Introduction to AMPS. To understand which features are most commonly used for a given scenario or application pattern, see the [Scenario and Feature Reference](/docs/intro-guide/feature\_guide/). This provides a quick guide to help you focus on learning the features that are most relevant to the problem at hand. For example, to distribute work across a set of independent processors, you would use AMPS message queues, whereas if your application requires a content-aware last value cache, you would use a Topic in the AMPS State of the World. ## Evaluation Process Outline 60East provides access to technical support during the evaluation process. To prepare to evaluate AMPS, 60East recommends the following process: * Engage 60East support with a description of the evaluation goals, and to put in place any agreements necessary to make evaluation go more smoothly (such as mutual non-disclosure agreements). * Define the detailed goals of the feasibility phase of the evaluation. Typically, these break down into: * _Functional Capability -_ This represents what the evaluation project needs to be able to do. (For example: accurately receive NVFIX messages and deliver them to the appropriate subscriber or subscribers while maintaining the ability to replay 30 days of history at any point.) * _Performance Goals -_ This represents the service level for the evaluation. (For example: Maximum latency to reach client processing of 250ms for current messages, no more than 1s to first message for beginning a replay at an arbitrary depth in history.) * _Capacity Goals -_ This represents the total volume of work being evaluated. (For example: The system needs to reach capability and performance goals while processing 10,000 messages per second ingestion.) * Develop initial design and testing plan. During this process, teams use the AMPS documentation to understand how to use AMPS to meet the goals of the evaluation. Teams engage 60East support as necessary to resolve any questions that emerge or get advice on tradeoffs and options to achieve the evaluation goals. * Once a design and testing plan is complete, review the design and testing plan with the 60East engineering team, adjusting as necessary. * Implement the design and tests. If questions or issues emerge, consult with 60East support to resolve the issues. * Test and review the results with 60East. * Evaluate the deployment and maintenance phase of the evaluation. Typically, this involves: * _Operations and Performance at Scale -_ Evaluate the application in a production-like environment at scales at or near production volumes. * _Maintenance Goals -_ Develop and test the maintenance, support, and upgrade plan as described in the 60East deployment checklist. * Follow up on any open issues and complete the evaluation. --- # Introduction ## Welcome to 60East Technologies AMPS Thank you for choosing to evaluate the Advanced Message Processing System (AMPS) from 60East Technologies! AMPS is designed to make it easy to develop and deploy data-intensive applications with demanding requirements for low latency and high performance. AMPS takes a nontraditional approach to messaging, storage, and analytics that is designed from the ground up for streaming data and highly-parallelized multicore systems. AMPS is more than a publish and subscribe system. It is a feature-rich platform that enables you to easily build data intensive applications that provide previously unattainable low latency and high performance. AMPS combines a set of capabilities that cut across traditional component divisions. 60East designed the capabilities based on the needs of some of the most demanding data-intensive applications on the planet, and engineered the capabilities to work together seamlessly and provide the kind of performance and latency that those applications demand. AMPS isn't a traditional database or messaging product. This guide presents a brief introduction to AMPS and contains information on evaluating AMPS. ### Documentation Roadmap This guide is designed to be use alongside other parts of the AMPS documentation. | Guide | Purpose | | -------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | [Introduction to AMPS](/docs/amps-user-guide/intro) | Overview of AMPS features and capabilities, intended as a starting point for learning AMPS. | | [AMPS User Guide](/docs/amps-user-guide) | Description of AMPS server-side features and configuration options. | The AMPS client distributions also contain guides discussing the programming model for applications and how to use the client libraries effectively. In addition, 60East maintains an FAQ on the 60East support site at: | Site | Purpose | | -------------------------------------------------------------------------------------------|-----------------------------------------------------| | [https://crankuptheamps.com/support](/support) | Frequently asked questions about the AMPS product. | For evaluation purposes, 60East recommends starting with the [Introduction to AMPS](/docs/intro-guide/intro) for an overview of the features of AMPS, including a cross-reference as to which features are most commonly used together in particular application scenarios. ## AMPS Software Requirements The AMPS server is supported on the following platforms: * Linux 64-bit (2.6 kernel or later) on x86\_64 compatible processors :::tip While 2.6 is the minimum kernel version supported, AMPS will select the most efficient mechanisms available to it and thus reaps greater benefit from more recent kernel and CPU versions. ::: The AMPS distribution contains all of the supporting libraries and dependencies needed to run on a typical Linux server installation: no further software is required. Some utilities provided with the AMPS server have additional dependencies. These utilities are not required to run the server, but can make it easier to troubleshoot and test on the system that hosts the AMPS instance: * `spark`, a basic command line client that supports a subset of AMPS functionality, requires Java 1.7 or later. * The utilities for inspecting AMPS files (`amps_sow_dump`, `amps_clients_ack_dump`, and so on) require a Python installation. * `amps-grep` requires a Python installation. * `amps-sqlite3` requires a Python installation and the sqlite3 package for your distribution (often, but not always, installed by default). ## Obtaining Evaluation Licenses and the AMPS Server For existing customers, evaluation and development licenses are typically covered in the existing licensing agreement. Contact the team that manages the license agreement or 60East for details. For new customers, you can register for your evaluation, obtain an evaluation license, and receive instructions for downloading AMPS from the [Evaluate AMPS](https://www.crankuptheamps.com/evaluate) page of the [60East Website](https://www.crankuptheamps.com). The registration process covers the terms of the evaluation license. You can use the support website, as described in[ Obtaining Evaluation Support](intro#obtaining-evaluation-support), for any questions that arise during your evaluation, including both licensing questions and technical questions. ## Obtaining Evaluation Support For existing customers, an outline of your specific support benefits and policies is available in your 60East Technologies License Agreement. Support contracts can be purchased through your 60East Technologies account representative. Existing customers will also typically already have a non-disclosure agreement in place, allowing development teams to discuss the details of their applications with 60East for the purposes of troubleshooting issues or answering questions about AMPS. For new customers, contact support (as described in the following sections) for any issues that emerge during your evaluation. Support for evaluation purposes is typically available for new customers without a support contract. If your company requires a non-disclosure agreement with 60East before discussing technical information, please contact support to begin the process of creating and executing an agreement. ### Support Steps You can save time if you complete the following steps before you contact 60East Technologies Support: 1. _**Check the documentation**_ The problem may already be solved and documented in the _User Guide_ for the product. 60East Technologies also provides answers to frequently asked support questions on the support web site at [http://crankuptheamps.com/support](/support). 2. _**Isolate the problem**_ If you require Support Services, please isolate the problem to the smallest test case possible. Capture erroneous output into a text file along with the commands used to generate the errors. 3. _**Collect your information**_ * Your product version number. * Your operating system and its kernel version number. * The expected behavior, observed behavior and all input used to reproduce the problem. * Submit your request. * If you have a minidump file, be sure to include that in your email to [crash@crankuptheamps.com](mailto:crash@crankuptheamps.com). The AMPS version number used when reporting your product version number follows a format listed below. The version number is composed of the following: ``` MAJOR.MINOR.FEATURE.HOTFIX.TIMESTAMP.TAG ``` ### AMPS Versioning and Certification Each AMPS version number component has the following breakdown: | Component | Description | Minimum Verification | | -------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------| | `MAJOR` |

Increments when there are any backward-incompatible changes in functionality, file formats, client network formats or configuration; or when deprecated functionality is removed.

May introduce major new functionality or include internal improvements that introduce major behavioral changes.

| Megacert | | `MINOR` |

Increments when functionality is added in a backwards-compatible way, or when functionality is deprecated.

May include internal improvements, including internal improvements that introduce minor behavioral changes or changes to network formats used only by the AMPS server (such as replication).

| Megacert | | `FEATURE` |

Increments for previews of new features.

May introduce behavioral changes to fix incorrect behavior, enable new functionality or to enhance performance.

May include internal enhancements that do not introduce behavioral changes.

Note: A feature level of `0` indicates a long-term stable release. A feature level above zero indicates the current feature level (a preview of the next long-term stable release).

| Kilocert | | `HOTFIX` |

A release for a critical defect impacting a customer. A hotfix release is designed to be 100% compatible with the release it fixes (that is, a release with same `MAJOR.MINOR.FEATURE` version).

May introduce behavioral changes to fix incorrect behavior. May document previously undocumented features or extend surface area to improve usability for existing features.

| Cert | | `TIMESTAMP` |

Proprietary build timestamp.

| (does not affect verification level) | | `TAG` |

Identifier that corresponds to precise code used in the release.

| (does not affect verification level) | The certification levels are defined in the following table. Notice that, in all cases, 60East will certify at a higher level if time permits or if a change involves a critical part of AMPS (such as replication or internal utility classes that are widely used). | Certification Level | Description | Time to Certify | | ---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------| | Megacert |

Performance and long-haul testing.

Full regression suite and stress-testing suite, including replication testing and application scenario tests.

Full unit testing suite, including new unit tests to verify correct behavior of bugfixes in this release.

| less than 2 weeks | | Kilocert |

Full regression suite and stress-testing suite, including replication testing and application scenario tests.

Full unit testing suite, including new unit tests to verify correct behavior of bugfixes in this release.

| less than 1 week | | Cert |

Full unit testing suite, including new unit tests to verify correct behavior of bugfixes in this release.

Replication testing suite if release affects replication code.

| 4 hours | ### Contacting 60East Technologies Support Please contact 60East Technologies Support Services according to the terms of 60East Technologies License Agreement (whether that is an existing license or evaluation license). Support is offered through the United States: | | | |-----------|-------------------------------------------------------------------------| | Web: | [http://www.crankuptheamps.com](http://www.crankuptheamps.com) | | E-mail: | [support@crankuptheamps.com](mailto:support@crankuptheamps.com) | | Support: | [http://crankuptheamps.com/support](/support) | Other support options may be available for existing customers, depending on the terms of the support agreement. --- # Next Steps Once you have done a basic evaluation of AMPS, there are two typical paths forward in usage of the product: * On one path, you may want to learn how to configure, deploy, and administer an instance of AMPS. For this path, see the _AMPS User Guide_, which provides complete information for system administrators who are responsible for the deployment, availability and management of data to other users. * Alternatively, you may need to develop an application to work with AMPS, using one of the Developer Guides for Java, Python, C++, or C#. For this path, download one of the client distributions from the AMPS developer page at [https://www.crankuptheamps.com/develop/](https://www.crankuptheamps.com/develop/). The client distributions include a set of examples and an AMPS server configuration that works with the examples. The following sections provide more information about each of these paths and also briefly describes some use cases for AMPS. ## Operation and Deployment In preparing to deploy your instance of AMPS, you must size your host environment according to multiple dimensions: memory, storage, CPU, and network. The [Operation and Deployment](../amps-user-guide/operation) chapter in the [AMPS User Guide](../amps-user-guide/) provides guidelines and best practices for configuring the host environment. The chapter also specifies recommended settings for running AMPS on a Linux operating system. ## Application Development Each language-specific _Development Guide_ explains how to install, configure, and develop applications that use AMPS. In order to develop applications using an AMPS client, you must understand the basic concepts of AMPS, such as _topics_, _subscriptions_, _messages_ and _SOW_. You will also need an installed and running AMPS server to use the product. Typically, a team will use a server licensed for evaluation during the initial stages of development, then transition to a full license as the evaluation completes and the team prepares to deploy the application. --- # Tips on Measuring Performance One of the most common questions during evaluation of AMPS is how best to measure and quantify the overall performance of the application that uses AMPS. There are several factors that are included in any meaningful discussion of performance: ## Performance of the Underlying Hardware AMPS is designed to use the underlying hardware as efficiently as possible, and does not suffer from artificial bottlenecks that limit performance. The implications of this, though, are that the performance available from an installation of AMPS depends on the capacity of the underlying hardware. In particular, pay attention to: * Storage device speed and bandwidth (for applications that persist data) * Memory speed * Network speed and capacity Often, you can come up with the theoretical maximum performance of a system based on the underlying hardware. For example, storage that can only write 80MB/s would be unsuitable for a system that needs to retain messages that arrive at a sustained rate of 100MB/s. Likewise, a system with 64GB of memory would see reduced performance for lookups on a 128GB data set, so benchmarking an application that retains 128GB of active data on a system with 64GB of memory will produce very different results than the same benchmark run on a system with 256GB of memory. ## Operating System Performance Most Linux distributions and installations are, by default, tuned for interactive desktop usage. This is convenient when developing applications, but can produce reduced performance as compared with a well-tuned server. [Linux OS Settings](../amps-user-guide/operation/linux-configuration) in the [AMPS User Guide](../amps-user-guide/) discusses the Linux settings that are most often configured in a way that limits the performance of AMPS on a host. Before taking final benchmarks, tune the Linux host according to those guidelines. ## Realistic Data Complexity and Volumes AMPS is designed for high-throughput, low latency messaging. This means that AMPS typically performs better with a realistic workload than with a very small number of messages. It is typically not useful to run a performance test with a small number of messages and then attempt to extrapolate the performance at scale. As an example, imagine a test that deploys a Docker container from scratch, starts AMPS, sends and receives a single message, and then shuts down the container and uses the elapsed time from the start of the test to the time that the container shuts down as the "single message throughput time". That number will be orders of magnitude slower than the actual time that it takes for AMPS to deliver the message: most of the time in the test is consumed by overhead unrelated to delivering an individual message. Although it would be unlikely that anyone would create a test with as much overhead as the scenario above, it is not uncommon to have hidden overhead in a test. Likewise, there are often "economies of scale" that the system (including AMPS) can take advantage of production-level usage that is not available at unrealistically low messaging rates. A realistic test should avoid measuring overhead that would not be present in a production environment. If the requirement of the application is to have latency within a certain threshold when AMPS is processing messages at a sustained rate from a dozen publishers, the results of a test will be more accurate the more closely the test approximates that scenario. In particular, as much as possible, build your tests to: * _Have similar use of connections as the production application._ If a given application will have multiple subscribers in production, do not use a single subscriber in performance testing and assume that parallel processing offers no benefit. * _Have similar message volumes as the production application._ Do not assume that you can use a rate of 100 messages per second to predict latency or processing time of an application that will need to process 1000 or 10000 messages per second. * _Have similar message sizes as the production application._ Do not assume that a 1MB message size in test will have the same performance characteristics as a 250KB message (or a 5MB message) in production. ## Compare Equivalent Work When benchmarking different implementation ideas, compare equivalent work. In some cases, having the AMPS server do additional work does not add noticeable latency due to the efficiencies (and parallel processing) in AMPS. In other cases, having the server do additional work may add more latency. In either case, accurately measuring throughput and latency must measure the cost of doing equivalent work in the application. For example, if your application will use AMPS delta subscriptions (that is, have AMPS automatically calculate the differences between an update to a message and the current state of the message), rather than comparing throughput for a subscription that uses that option to a subscription that does not use that option based solely on when messages arrive at the client, compare the differences between having AMPS calculate the difference versus having the application calculate the difference, and evaluate this difference based on the total throughput numbers for a realistic number of subscribers. ## Use AMPS Capabilities AMPS is carefully designed to include functionality that reduces end-to-end throughput in the system, and to provide server-side capability where performing those functions on the server improves overall performance. When evaluating performance, take advantage of those capabilities to get an accurate measure of how an application would perform in a production environment. For example, if your application needs to append a calculated field to every published message, use message enrichment (or the AMPS delta publish functionality) rather than a process that extracts, rewrites, and updates the full message. Likewise, if your application will only process a subset of messages to the topic, use AMPS content filtering to ensure that AMPS only provides actionable messages rather than oversubscribing and discarding messages in your application. If you have questions on whether your application is using the built-in capabilities of AMPS in the most effective way possible, contact 60East support for an engineer to review your design. --- # Updating AMPS Client Connections ## Client Connection Strings Use the URI that matches the proxy and AMPS transport configuration: - Secure TCP connections: `tcps://localhost:443/tcps/amps/json?http_preflight=true` - Secure WebSocket connections: `wss://localhost:443/wss/amps/json` - Standard non-SSL TCP connections: `tcp://localhost:80/tcp/amps/json?http_preflight=true` - Standard non-SSL WebSocket connections: `ws://localhost:80/ws/amps/json` The code examples below all use the secure URLs. For a non-SSL deployment, replace `tcps` with `tcp`, `wss` with `ws`, port `443` with `80`, and the `/tcps/` and `/wss/` paths with `/tcp/` and `/ws/`. WebSocket connections don't require the HTTP Preflight option as they use an HTTP Upgrade mechanism to upgrade to the WebSocket protocol by default. For more information on HTTP Preflight, see [HTTP Preflight](/docs/amps-user-guide/transports/http-preflight) in the AMPS User Guide. ## Example Client Connections ### JavaScript For install and usage details, see [Obtaining and Installing the AMPS Client](/clients/amps-client-javascript/installing). ```javascript showLineNumbers const { Client } = require('amps') // required when the CA certificates are self-signed process.env.NODE_TLS_REJECT_UNAUTHORIZED = 0; async function main() { // Connect and logon const client = new Client('javascript-proxy-client') // HTTP preflight is not required for standard websocket connections await client.connect('wss://localhost:443/wss/amps/json') console.log('connected!') // No difference in usage once the client is connected client.publish("test", "{\"hello\":\"world\"}") client.disconnect() } main() ``` This example creates an AMPS Client and establishes a secure WebSocket connection to AMPS through the Apache reverse proxy. ### Python For install and usage details, see [Python Quickstart](/clients/amps-client-python/quickstart). ```python showLineNumbers import AMPS client = AMPS.Client('python-proxy-client') # Connect and logon using HTTP preflight client.connect('tcps://localhost:443/tcps/amps/json?http_preflight=true') client.logon() print('connected!') # No difference in usage once the client is connected client.publish("test", "{\"hello\":\"world\"}") client.disconnect() client.close() ``` This example creates an AMPS Client and uses HTTP Preflight to establish an upgraded TCPS connection to AMPS through the Apache reverse proxy. ### C++ For install and usage details, see [Obtaining and Installing the AMPS Client](/clients/amps-client-cpp/installing). ```cpp showLineNumbers #include #include int main() { AMPS::Client client("cpp-proxy-client"); // Connect and logon using HTTP preflight client.connect("tcps://localhost:443/tcps/amps/json?http_preflight=true"); client.logon(); std::cout << "Connected!" << std::endl; // No difference in usage once the client is connected client.publish("test", "{\"hello\":\"world\"}"); client.disconnect(); return 0; } ``` This example creates an AMPS Client and uses HTTP Preflight to establish an upgraded TCPS connection to AMPS through the Apache reverse proxy. ### Java For install and usage details, see [Obtaining and Installing the AMPS Client](/clients/amps-client-java/installing). :::note SSL connections in Java require a properly configured truststore so the client can verify and trust the server’s certificate during the TLS handshake. You can create a truststore by running the following: ```bash keytool -importcert -file /etc/pki/tls/certs/localhost.crt -keystore amps-truststore.p12 -storepass 123456 ``` ::: ```java showLineNumbers import com.crankuptheamps.client.Client; public class Example { public static void main(String[] args) { // required for providing the proper certs for ssl connections System.setProperty("javax.net.ssl.trustStore", "amps-truststore.p12"); System.setProperty("javax.net.ssl.trustStorePassword", "123456"); Client client = new Client("java-proxy-client"); try { // Connect and logon using HTTP preflight client.connect("tcps://localhost:443/tcps/amps/json?http_preflight=true"); client.logon(); System.out.println("Connected!"); // No difference in usage once the client is connected client.publish("test", "{\"hello\":\"world\"}"); } catch (Exception e) { System.err.println("Exception: " + e); } finally { client.close(); } } } ``` This example creates an AMPS Client and uses HTTP Preflight to establish an upgraded TCPS connection to AMPS through the Apache reverse proxy. For additional SSL guidance, see [Providing SSL Certificates to the AMPS Java Client](/clients/amps-client-java/advanced-topics#providing-ssl-certificates-to-the-amps-java-client). ### C#/.NET For install and usage details, see [Obtaining and Installing the AMPS Client](/clients/amps-client-csharp/installing). ```cs showLineNumbers using System; using AMPS.Client; using AMPS.Client.Exceptions; try { // Connect and logon using HTTP preflight and the proxy endpoint Client client = new Client("csharp-proxy-client"); client.connect("tcps://localhost:443/tcps/amps/json?http_preflight=true"); client.logon(); Console.WriteLine("Connected!"); // No difference in usage once the client is connected client.publish("test", "{\"hello\":\"world\"}"); client.close(); } catch (AMPSException exception) { Console.WriteLine(exception); } ``` This example creates an AMPS Client and uses HTTP Preflight to establish an upgraded TCPS connection to AMPS through the Apache reverse proxy. :::tip If the clients fail to connect after following the above steps on **Fedora/Red Hat**, try running the following command: ```bash sudo setsebool -P httpd_can_network_connect 1 ``` This command modifies a security setting in **SELinux** (Security-Enhanced Linux) to allow the **httpd** (web server) process to make network connections. It also ensures that this setting persists across system reboots. If you use the Fedora `localhost` certificate shown in this guide, connect to `localhost` rather than `127.0.0.1`, or replace the certificate with one that matches the hostname clients will use. ::: --- # Configuring AMPS for Use via a Proxy ## Update the AMPS Configuration First, we need to define the AMPS transports that the proxy will forward connection requests to. For `TLS/SSL` transports, set `Type` to `tcps` and add the `Certificate` and `PrivateKey` elements, like so: ```xml any-tcps tcps amps 9007 /etc/pki/tls/certs/localhost.crt /etc/pki/tls/private/localhost.key any-wss tcps websocket 9008 /etc/pki/tls/certs/localhost.crt /etc/pki/tls/private/localhost.key ``` For a standard non-SSL configuration, you can simply omit the `Certificate` and `PrivateKey` elements from the above example and use a `Type` of `tcp`: ```xml any-tcp tcp amps 9007 any-ws tcp websocket 9008 ``` Second, update the `Admin` configuration with the `SQLTransport` element to specify which websocket `Transport` Galvanometer should use to submit queries and subscriptions: ```xml 8085 any-wss /etc/pki/tls/certs/localhost.crt /etc/pki/tls/private/localhost.key ``` The `Certificate` and `PrivateKey` elements are only necessary when the Admin interface itself is served over TLS/SSL. For a non-SSL deployment, use the non-secure websocket transport name (`any-ws`) and omit the TLS configuration. Lastly, include the `ExternalInetAddr` element in the `Admin` configuration to specify the value that should be used for connections to the Admin interface. `ExternalInetAddr` is not intended to override the `InetAddr` parameter or change the network addresses that the Admin server uses, but instead to provide an externally visible address that will reach the `InetAddr`. For example, a proxy on address `proxy_host_address` that exposes an `/admin` endpoint for the Admin interface: ```xml proxy_host_address/admin ``` Additionally, you can include the `SQLTransportInetAddr` element in the `Admin` configuration to directly provide the URI that the Galvanometer will use for submitting queries and subscriptions to AMPS. For example, if the proxy exposes a secure websocket transport at `proxy_host_address/wss`, use: ```xml showLineNumbers proxy_host_address/wss ``` ## Start the AMPS Server ```bash .//ampServer config.xml ``` If you need help installing and starting AMPS, see the [Getting Started With AMPS](/docs/intro-guide/getting_started) guide for instructions. You can find additional information on [Using AMPS with a Proxy](/docs/amps-user-guide/operation/proxy) in the AMPS User Guide. --- # Install and Configure Apache ## Prerequisites Before proceeding with this guide, make sure that you have access to the following on your development machine: - Fedora/Red Hat Environment - Sudo Privileges ## Install Apache HTTPD The Apache HTTPD package and its dependencies can be installed via the `dnf` package manager: ```bash sudo dnf install httpd mod_ssl ``` Additionally, the Apache Web Server (httpd) service must be enabled and started: ```bash sudo systemctl enable --now httpd # enable: Configures httpd to start automatically at system boot. # --now: Starts the httpd service immediately ``` ## Configuring Apache HTTPD as a Reverse Proxy :::info This guide focuses on three kinds of proxy traffic: - Galvanometer over HTTP or HTTPS - TCP or TCPS connections using HTTP Preflight - WS or WSS connections using WebSocket upgrade For connections over SSL, use `tcps://` and `wss://` on the client side, `/tcps/` and `/wss/` on the Apache side, and `https://` backend targets for the secure AMPS transports. For non-SSL connections, use `tcp://` and `ws://` on the client side, `/tcp/` and `/ws/` on the Apache side, and `http://` backend targets. If you are unsure which connection type to use, see [TLS/SSL Transports](/docs/amps-user-guide/transports/configuring-transports#tlsssl-transports) in the AMPS User Guide. ::: Create a virtual host configuration file under `/etc/httpd/conf.d/` such as `/etc/httpd/conf.d/amps-proxy.conf` (`sudo` access is required). The first step in configuring Apache is to define a `VirtualHost` and configure the address `*:443` to listen for all HTTPS requests. For a non-SSL configuration, we will use `*:80` to listen for all HTTP requests. Next, we have to enable the SSL functionality in Apache by including `SSLEngine` and `SSLProxyEngine`, and by defining the necessary certificates. For this demonstration, we will use the Fedora localhost certificates located at `/etc/pki/tls/certs/localhost.crt` and `/etc/pki/tls/private/localhost.key`. Because the backend AMPS services also use those localhost certificates, this example disables backend certificate verification with `SSLProxyVerify`, `SSLProxyCheckPeerName`, and `SSLProxyCheckPeerExpire`. :::warning The Fedora `localhost` certificate is suitable only for local development. If clients or browsers connect using a different hostname, the certificate must match that hostname. For browser-based access to Galvanometer, the certificate must also be trusted by the browser. ::: To support HTTP upgrade requests, we will also include the `Header`, `RequestHeader`, and `ProxyPreserveHost` directives so Apache can forward the requests properly. Start with the base `VirtualHost` and SSL configuration: ```apache showLineNumbers ServerName ApacheReverseProxyExample SSLEngine On SSLCertificateFile /etc/pki/tls/certs/localhost.crt SSLCertificateKeyFile /etc/pki/tls/private/localhost.key SSLProxyEngine On SSLProxyVerify none SSLProxyCheckPeerName off SSLProxyCheckPeerExpire off Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains" ProxyPreserveHost On RequestHeader set X-Forwarded-Proto "https" ``` Now we can define the proxy configuration by adding `ProxyPass` and `ProxyPassReverse` directives for each endpoint we want Apache to forward. We can also add a `RedirectMatch` directive for tools like the Galvanometer which expect the URL with a trailing slash. :::note This guide assumes the AMPS server is running on the same host as the reverse proxy. If this is not the case, replace `127.0.0.1` with the address of your AMPS server. ::: Add the following directives to forward `/admin/` traffic to the backend service on port `8085`: ```apache showLineNumbers ProxyPass "/admin/" "https://127.0.0.1:8085/" ProxyPassReverse "/admin/" "https://127.0.0.1:8085/" ProxyPassReverseCookiePath "/" "/admin/" RedirectMatch 301 ^/admin$ /admin/ ``` Next, add a `/tcps/` block that forwards TCPS connections to port `9007` with `upgrade=tcps`: ```apache showLineNumbers ProxyPass "/tcps/" "https://127.0.0.1:9007/" timeout=86400 upgrade=tcps ProxyPassReverse "/tcps/" "https://127.0.0.1:9007/" ``` Finally, add a block for `/wss/` that forwards Secure WebSocket traffic to port `9008` with `upgrade=websocket`: ```apache showLineNumbers ProxyPass "/wss/" "https://127.0.0.1:9008/" timeout=86400 upgrade=websocket ProxyPassReverse "/wss/" "https://127.0.0.1:9008/" ``` The complete Apache configuration for SSL connections: ```apache showLineNumbers ServerName ApacheReverseProxyExample SSLEngine On SSLCertificateFile /etc/pki/tls/certs/localhost.crt SSLCertificateKeyFile /etc/pki/tls/private/localhost.key SSLProxyEngine On SSLProxyVerify none SSLProxyCheckPeerName off SSLProxyCheckPeerExpire off Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains" ProxyPreserveHost On RequestHeader set X-Forwarded-Proto "https" # ----------------------------------------- # Galvanometer UI: /admin -> :8085 # Provides access to https://proxy_host_address:443/admin and /admin/ # ----------------------------------------- ProxyPass "/admin/" "https://127.0.0.1:8085/" ProxyPassReverse "/admin/" "https://127.0.0.1:8085/" ProxyPassReverseCookiePath "/" "/admin/" # Handle /admin without trailing slash RedirectMatch 301 ^/admin$ /admin/ # ------------------------- # TCPS Connections: /tcps -> :9007 # Client connection string: tcps://proxy_host_address:443/tcps/...?http_preflight=true # ------------------------- ProxyPass "/tcps/" "https://127.0.0.1:9007/" timeout=86400 upgrade=tcps ProxyPassReverse "/tcps/" "https://127.0.0.1:9007/" # ------------------------- # Secure WebSockets: /wss -> :9008 # Client connection string: wss://proxy_host_address:443/wss/... # ------------------------- ProxyPass "/wss/" "https://127.0.0.1:9008/" timeout=86400 upgrade=websocket ProxyPassReverse "/wss/" "https://127.0.0.1:9008/" ``` The standard non-SSL deployment Apache configuration: ```apache showLineNumbers ServerName ApacheReverseProxyExample ProxyPreserveHost On RequestHeader set X-Forwarded-Proto "http" # ----------------------------------------- # Galvanometer UI: /admin -> :8085 # Provides access to http://proxy_host_address:80/admin and /admin/ # ----------------------------------------- ProxyPass "/admin/" "http://127.0.0.1:8085/" ProxyPassReverse "/admin/" "http://127.0.0.1:8085/" ProxyPassReverseCookiePath "/" "/admin/" # Handle /admin without trailing slash RedirectMatch 301 ^/admin$ /admin/ # ------------------------- # TCP: /tcp -> :9007 # Client connection string: tcp://proxy_host_address:80/tcp/...?http_preflight=true # ------------------------- ProxyPass "/tcp/" "http://127.0.0.1:9007/" timeout=86400 upgrade=tcp ProxyPassReverse "/tcp/" "http://127.0.0.1:9007/" # ------------------------- # WebSockets: /ws -> :9008 # Client connection string: ws://proxy_host_address:80/ws/... # ------------------------- ProxyPass "/ws/" "http://127.0.0.1:9008/" timeout=86400 upgrade=websocket ProxyPassReverse "/ws/" "http://127.0.0.1:9008/" ``` ## Validate and Apply the Apache Configuration Validate the Apache configuration file: ```bash sudo apachectl configtest ``` Restart the Apache HTTP server to pick up the new configuration file: ```bash sudo systemctl restart httpd ``` Verify the Apache HTTP server is running: ```bash sudo systemctl status httpd ``` The examples above show both SSL and non-SSL Apache reverse proxy configurations for AMPS TCP, WebSocket, and Galvanometer connections. --- # Apache Proxy Configuration Guide An Apache reverse proxy sits in front of AMPS and forwards client connections to the appropriate backend transports, simplifying network routing, and providing a single entry point for AMPS traffic. This guide covers SSL (`tcps`, `wss`) and non-SSL (`tcp`, `ws`) proxy setups, as well as the corresponding AMPS transport configuration. It also demonstrates how AMPS Clients that use `TCP/SSL` support connection through an HTTP proxy when the HTTP Preflight option is provided in the connection string. For an overview of AMPS and instructions for setting up your development environment, refer to the [Introduction to AMPS](/docs/intro-guide/intro) guide. For additional information on accessing AMPS through a proxy, see the [Using AMPS with a Proxy](/docs/amps-user-guide/operation/proxy) section. For more information on HTTP Preflight, see the [HTTP Preflight](/docs/amps-user-guide/transports/http-preflight) section in the AMPS User Guide. For an overview of Apache reverse proxy features, see the [Apache Reverse Proxy Guide](https://httpd.apache.org/docs/2.4/howto/reverse_proxy.html). :::tip This guide assumes that you have a development environment for configuring a reverse proxy (including sudo privileges), and access to an AMPS server. ::: --- # Builder The builder pattern is used to construct new AMPS Sink instances. ## Required The following methods in the builder are required to construct a valid AMPS Sink instance. ### Connection to AMPS One of the following must be used for the AMPS Sink to connect to AMPS. | Method | Description | | ------ | ----------- | | setUri | Sets the URI that will be used to connect to AMPS. | | setServerChooserSupplier | Sets the supplier used to get a server chooser to connect to AMPS. The supplier must be serializable. | ### Topic for AMPS The following must be used to determine what topic the AMPS Sink should publish to. | Method | Description | | ------ | ----------- | | setTopic | Sets the topic that will be used by the publish command. | ### Serializing Data for AMPS The following method is overloaded to allow either a `SerializationSchema` or an `AMPSSerializationSchema` to be used by an AMPS Sink. | Method | Description | | ------ | ----------- | | setSerializationSchema | Sets the serialization schema that will be used to serialize messages for AMPS. | ### Building the AMPS Sink The following method must be used to build the AMPS Sink based on the set builder methods. | Method | Description | | ------ | ----------- | | build | Returns the constructed AMPS Sink. | ## Optional - AMPS Client The following methods in the builder are optional for functionality related to the AMPS client. | Method | Description | | ------ | ----------- | | setClientName | Sets the base client name. | | setPublishStoreFunction | Sets the `SerializableFunction` that gets each client a store. The parameter is the client name that an individual client of the sink will use. | | setPublishCommand | Sets the publish command for the sink.

Valid commands include:
  • "publish" (Default)
  • "sow_delete"
  • "delta_publish" (May require a custom serialization schema)
| | setFailedWriteHandlerSupplier | Sets the supplier that gets each client a `FailedWriteHandler`. The supplier must be serializable. | | setReconnectDelayStrategySupplier | Sets the supplier that gets each client a `ReconnectDelayStrategy`. Default uses exponential delay. The supplier must be serializable. | | setExceptionListenerSupplier | Sets the supplier that gets each client an exception listener. The supplier must be serializable. | | setHeartbeat | Sets the heartbeat interval in seconds for the clients. | | setRetryOnDisconnect | Set whether or not messages being sent to the server should retry if the client is disconnected. | | setExpiration | Sets the expiration for SOW/queue messages sent. | | setCorrelationId | Sets the correlation ID to use on every message sent to AMPS from this sink. Must contain only Base64 characters.

The correlation ID provided by an `AMPSSerializationSchema` will override the correlation ID provided here. If there is no correlation ID provided by the schema, then this value will be used. | | setConnectorInitializer | Sets the `ConnectorInitializer` that will run its `init(HAClient)` method before the reader connects to AMPS. This can be used to execute code such as setting up an SSLContext for the AMPS clients. | ## Optional - Other The following methods in the builder are optional for additional functionality not directly related to the AMPS client. | Method | Description | | ------ | ----------- | | setDeliveryGuarantee | Sets the sink's delivery guarantee. Default is NONE.

This should only be set to AT_LEAST_ONCE or EXACTLY_ONCE if checkpointing is enabled. EXACTLY_ONCE requires strictly increasing timestamps. AT_LEAST_ONCE should be used over EXACTLY_ONCE for message queues. | | setUseSuffix | Sets the flag for if a suffix should be appended to the client name. Default is false, but it will be set to true if parallelism is greater than 1. | --- # Checkpointing Flink's checkpointing provides both fault tolerance and delivery guarantees. For the AMPS Sink, checkpointing is used for fault tolerance and delivery guarantees. ## Delivery Guarantees In order to achieve at least once delivery guarantees, the AMPS Sink uses a Publish Store and coordinates with Flink's checkpoints to determine when to publish messages. The sink stores all messages within a given checkpoint, and when that checkpoint completes, the connector flushes all the stored messages to AMPS. An AMPS Sink that flushes its stored messages on checkpoint completion can be constructed similar to the following: ```java AMPSSink sink = AMPSSink.builder() .setUri(uri) .setTopic(topic) .setSerializationSchema(new SimpleStringSchema()) .setDeliveryGuarantee(DeliveryGuarantee.AT_LEAST_ONCE) .build(); ``` :::warning Checkpointing should be enabled if a delivery guarantee is used. Otherwise, messages will be stored in the sink indefinitely, and the JVM will eventually run out of memory. ::: ## Publish Store Alternatively, a Publish Store can be supplied to the AMPS Sink to enable at least once delivery guarantees. In this case, the sink publishes a message as soon as it receives the message from Flink, so checkpointing would not be required. However, the sink would not coordinate with Flink's checkpointing, which could result in some message replay in the event of failure. An AMPS Sink that only uses a Publish Store can be constructed similar to the following: ```java AMPSSink sink = AMPSSink.builder() .setUri(uri) .setTopic(topic) .setSerializationSchema(new SimpleStringSchema()) .setPublishStoreFunction(new SerializableFunction()) .build(); ``` :::info This will require an implementation of the Serializable Function from the AMPS Java client API with the following type parameters:
  • String argument that is the client name the sink will use
  • Store return type
    • ::: --- # Parallelism Flink provides scalibility of a job through parallelism, and the AMPS Flink connectors utilize parallelism as a simple way to improve performance. ## AMPS Sink Parallelism The AMPS Sink can use parallelism to create multiple clients to publish to AMPS in parallel. The following example uses two clients to publish to AMPS by setting parallelism to 2: ```java AMPSSink sink = AMPSSink.builder() .setUri(uri) .setTopic(topic) .setSerializationSchema(new SimpleStringSchema()) .build(); DataStream ds = ...; ds.sinkTo(sink).setParallelism(2); ``` ## Order When parallelism is greater than 1, there is no guarantee that all the messages in a data stream will be published in that exact order to AMPS. For example, if an AMPS Sink with a parallelism of 2 receives the data stream 1 -> 2 -> 3 -> 4, then Flink might assign the first client 1 and 3, and the second client might get 2 and 4. When publishing to AMPS, it is possible that some messages from one client arrive before some from the other client despite those messages appearing later in the data stream. If the exact order of a data stream must be maintained, use a parallelism of 1. This ensures that a single client publishes the messages in the order that they arrive. --- # AMPS Sink The `AMPSSink` publishes data to AMPS. It uses the Sink V2 API as described in [FLIP-191](https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction) and [FLIP-372](https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Enhance+and+synchronize+Sink+API+to+match+the+Source+API) to enable developers to move data from Flink into AMPS. The sink is responsible for serializing and publishing messages to AMPS. The sink follows the builder pattern to easily allow developers to only modify the sink based on specific needs. The following is an example of the simplest `AMPSSink` that can be constructed. In this example, the `uri` is the AMPS connection URI, and the `topic` is the topic the sink will use to publish to AMPS. The `SimpleStringSchema` is a serialization schema that serializes a `String` object from Flink into a byte array. ```java AMPSSink sink = AMPSSink.builder() .setUri(uri) .setTopic(topic) .setSerializationSchema(new SimpleStringSchema()) .build(); ``` :::info A valid `AMPSSink` requires a connection to AMPS, a topic to publish to, and a serialization schema to serialize the objects from Flink. The methods above demonstrate the simplest way to provide these to the sink through a URI string, a topic string, and a serialization schema. ::: --- # Boundedness In Flink, sources can involve bounded or unbounded data, which are implied in the BATCH and STREAMING modes. The following will make an AMPS Source a bounded operator: - Using a SOW query - Using a SOW query with topN defined - Using a bookmark subscription with topN defined - Using a ranged bookmark subscription In other words, any of the above will make an AMPS Source transition from the RUNNING state to the FINISHED state after it receives all intended messages from AMPS, which allows jobs to finish without being explicitly canceled. --- # Builder The builder pattern is used to construct new AMPS Source instances. ## Required The following methods in the builder are required to construct a valid AMPS Source instance. ### Connection to AMPS One of the following must be used for the AMPS Source to connect to AMPS. | Method | Description | | ------ | ----------- | | setUri | Sets the URI that will be used to connect to AMPS. | | setServerChooserSupplier | Sets the supplier used to get a server chooser to connect to AMPS. The supplier must be serializable. | ### Topic from AMPS One of the following must be used to determine what topic the AMPS Source should read from. | Method | Description | | ------ | ----------- | | setTopic | Sets the topic that will be used by the subscribe command. | | setAMPSSplits | Sets an `AMPSSplit` collection that will be used to create subscriptions. | ### Deserializing Data from AMPS The following method is overloaded to allow either a `DeserializationSchema` or an `AMPSDeserializationSchema` to be used by an AMPS Source. | Method | Description | | ------ | ----------- | | setDeserializationSchema | Sets the deserialization schema that will be used to deserialize messages from AMPS. | ### Building the AMPS Source The following method must be used to build the AMPS Source based on the set builder methods. | Method | Description | | ------ | ----------- | | build | Returns the constructed AMPS Source. | ## Optional - AMPS Client The following methods in the builder are optional for functionality related to the AMPS client. | Method | Description | | ------ | ----------- | | setClientName | Sets the base client name. | | setContentFilter | Sets the content filter used by all clients from this source. This can be used to provide a filter that all clients should use in addition to any filter provided by a split. | | setOptions | Sets additional options. | | setBookmarkStoreFunction | Sets the `SerializableFunction` that gets each client a bookmark store. The parameter is the client name that an individual client of the source will use.

    This should only be used if checkpointing is enabled or `discardAfterEmit` is true. | | setBookmark | Sets the starting bookmark.

    Only valid for topics with a transaction log. | | setSplits | Sets the splits for the source using a collection of strings that are content filters that will shard the topic.

    This can be used instead of `setAMPSSplits` to use content filters on the topic set by `setTopic`. | | setQueueSemantics | Sets the queue semantics if the source is subscribing to a queue.

    Valid semantics are:
    • "at-least-once"
    • "at-most-once"
    Only valid for queues. | | setAckBatchSize | Sets how many acknowledgement messages will be batched.

    Only valid for queues. | | setAckTimeout | Sets how long acknowledgement messages will be held.

    Only valid for queues. | | setTopN | Sets the amount of messages that should be received from AMPS.

    Only valid for SOW queries and topics with a transaction log. | | setSkipN | Sets the amount of messages in a SOW query that should be skipped. If "skip_n=n" is set using options, then this value will be ignored.

    Only valid for SOW queries with a set topN. | | setSubscribeCommand | Sets the subscription command for the source.

    Valid commands include:
    • "subscribe" (Default)
    • "sow"
    • "sow_and_subscribe"
    • "delta_subscribe" (May require a custom deserialization schema)
    • "sow_and_delta_subscribe" (May require a custom deserialization schema)
    | | setExceptionListenerSupplier | Sets the supplier that gets each client an exception listener. The supplier must be serializable. | | setBatchSize | Sets the batch size for SOW queries.

    Only valid for SOW queries. | | setOrderBy | Sets how SOW results should be ordered.

    Only valid for SOW queries. | | setReconnectDelayStrategySupplier | Sets the supplier that gets each client a `ReconnectDelayStrategy`. Default uses exponential delay. The supplier must be serializable. | | setHeartbeat | Sets the heartbeat interval in seconds for the clients. | | setMaxBacklog | Sets the max backlog when subscribing to a message queue. If "max_backlog=n" is set using options, then this value will be ignored.

    Only valid for queues. | | setPruneInterval | Sets the milliseconds that must pass before a `LoggedBookmarkStore` is pruned. Default is 30_000L or 30 seconds. This field only has an effect if there is a bookmark store defined, and it is a `LoggedBookmarkStore`. | | setHeaderKeys | Sets the headers that should be preserved from a message from AMPS. The headers will be preserved in an `AMPSMessage` and can only be accessed using the `.getHeader(AMPSSourceHeaderKeys)` method in an `AMPSDeserializationSchema`. | | setConnectorInitializer | Sets the `ConnectorInitializer` that will run its `init(HAClient)` method before the reader connects to AMPS. This can be used to execute code such as setting up an SSLContext for the AMPS clients. | ## Optional - Other The following methods in the builder are optional for additional functionality not directly related to the AMPS client. | Method | Description | | ------ | ----------- | | setDeliveryGuarantee | Sets the source's delivery guarantee by internally setting up a Memory Bookmark Store with a Recovery Point Adapter. Default is NONE.

    This should only be set to AT_LEAST_ONCE or EXACTLY_ONCE if checkpointing is enabled or `discardAfterEmit` is true. | | setInternalBufferSize | Sets the buffer size of the queue that each client uses to buffer messages from AMPS. Default is 1000. This should be adjusted based on the application the source is being used for. For example, applications with large message sizes may want to use a smaller buffer size to avoid excessive memory use on buffering messages. | | setConfiguration | Sets the Flink `Configuration` used to supply `SourceReaderOptions`. | | setSleepMillisAfterBlock | Sets the amount of milliseconds to sleep for after blocking for a message. Default is 0. This can be used to allow some messages to buffer in the queue before handing them to Flink. | | setDiscardAfterEmit | Sets the flag for if bookmarks should be discarded after a record is emitted rather than on checkpoint completion. Default is false. If the checkpoint interval for an application is high, this may help reduce memory usage for the source's bookmark store. | | setUseSuffix | Sets the flag for if a suffix should be appended to the client name. Default is false, but it will be set to true if parallelism is greater than 1 and there are multiple splits. | --- # Checkpointing Flink's checkpointing provides both fault tolerance and delivery guarantees. For the AMPS Source, checkpointing is used for fault tolerance, delivery guarantees, and "at-least-once" queues. ## Topics with a Transaction Log In order to work with Flink's checkpointing, the AMPS Source needs to be able to records its state and perform message replay. The source combines bookmarks and Flink's checkpointing to provide at least once delivery guarantees. If the source subscribes to a topic with a transaction log and checkpointing is enabled, then at least once delivery guarantee is achieved. Basically, sources will store the most recent bookmark, and if a failure occurs, the subscription will resume from that stored bookmark, which provides fault tolerance for the AMPS Source. An AMPS Source that works with Flink's checkpointing can be constructed similar to the following: ```java AMPSSource source = AMPSSource.builder() .setUri(uri) .setTopic(topic) .setDeserializationSchema(new SimpleStringSchema()) .setBookmark("0") // EPOCH bookmark .setDeliveryGuarantee(DeliveryGuarantee.AT_LEAST_ONCE) // Makes the source use a BookmarkStore to work with Flink's checkpointing .build(); ``` :::info The above example uses a Memory Bookmark Store that is set up internally to coordinate bookmarks with Flink's checkpoints. Any implementation of a Bookmark Store can be used. Refer to the [builder methods](./amps-source-builder) to see an overview of all options when constructing an AMPS Source. ::: :::warning An AMPS Source discards bookmarks on checkpoint completion by default. This means that checkpointing should be enabled if a Bookmark Store is used. Otherwise, bookmarks will not be discarded, and the JVM will eventually run out of memory. Alternatively, an AMPS Source can be configured to discard bookmarks when the source emits a record to Flink. ::: ## Message Queues The AMPS Source supports both "at-least-once" and "at-most-once" queues, and due do the differing queue semantics, they both acknowledge messages at different times. For an "at-most-once" queue, messages are acknowledged when an AMPS Source emits a message to Flink. This means that checkpointing is not required for an "at-most-once" queue to properly function. However, unlike "at-most-once" queues which do not require checkpointing, "at-least-once" queues need Flink's checkpointing to guarantee each message is processed at least once. Instead of discarding bookmarks, the source acknowledges messages on checkpoint completion. If a checkpoint fails to complete or a failure occurs, messages from the source are canceled and returned to the queue in AMPS. An AMPS Source that subscribes to a queue can be constructed similar to the following: ```java AMPSSource source = AMPSSource.builder() .setUri(uri) .setTopic(topic) .setDeserializationSchema(new SimpleStringSchema()) .setQueueSemantics("at-least-once") // Required .setMaxBacklog(20) // Optional .setAckBatchSize(5) // Optional .setAckTimeout(5000) // Optional .build(); ``` :::warning Checkpointing must be enabled if the source is subscribing to an "at-least-once" queue. Otherwise, the source will only receive the amount of messages specified in max backlog as it will never acknowledge the messages that it received. ::: --- # Parallelism Flink provides scalibility of a job through parallelism, and the AMPS Flink connectors utilize parallelism as a simple way to improve performance. ## AMPS Source Parallelism The AMPS Source can use splits to create multiple clients to read from AMPS in parallel. The following example shards a topic along the `id` field for incoming messages and uses two clients to read from AMPS by setting parallelism to 2: ```java AMPSSource source = AMPSSource.builder() .setUri(uri) .setTopic(topic) .setDeserializationSchema(new JsonDeserializationSchema<>(MyClass.class)) .setSplits(List.of("/id MOD 2 = 0", "/id MOD 2 = 1")) // Add the splits to tell the source to split along /id .build(); StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); // Use .setParallelism(2) to tell Flink the parallelism to use DataStream ds = env.fromSource(source, WatermarkStrategy.noWatermarks(), "AMPS Source Example").setParallelism(2); ``` :::info If the amount of splits is greater than the set parallelism, then some clients may receive multiple splits. This may reduce performance as a single client is being used to read messages from multiple subscriptions to AMPS as opposed to multiple clients being used to read messages from their own individual subscription to AMPS. ::: ## Order Messages from AMPS will always arrive in order from each split, but not necessarily in order when combined. For example, if the ids 1, 2, 3, and 4 are stored in AMPS in that specific order, the above AMPS Source may emit any of the following orders: - 1, 2, 3, 4 - 1, 3, 2, 4 - 2, 1, 3, 4 Basically, 1 will always occur before 3, and 2 will always occur before 4 since that is how the topics are sharded along the `id` field. This is because the split "/id MOD 2 = 0" processes the messages 1 and 3 (in order), and the split "/id MOD 2 = 1" processes the messages 2 and 4 (in order). The messages will be in order within the split, but there are no order guarantees across splits. --- # AMPS Source The `AMPSSource` reads data from AMPS. It is a [FLIP-27](https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface) source that enables developers to move data from AMPS into Flink. The source is responsible for receiving messages from AMPS, acknowledging messages from a queue, and deserializing messages. The source follows the builder pattern to easily allow developers to only modify the source based on specific needs. The following is an example of the simplest `AMPSSource` that can be constructed. In this example, the `uri` is the AMPS connection URI, and the `topic` is the topic the source will use to subscribe to AMPS. The `SimpleStringSchema` is a deserialization schema that deserializes the payload of a message into a `String`. ```java AMPSSource source = AMPSSource.builder() .setUri(uri) .setTopic(topic) .setDeserializationSchema(new SimpleStringSchema()) .build(); ``` :::info A valid `AMPSSource` requires a connection to AMPS, a topic to subscribe/query, and a deserialization schema to deserialize messages from AMPS. The methods above demonstrate the simplest way to provide these to the source through a URI string, a topic string, and a deserialization schema. ::: --- # Before You Start Welcome to developing applications with Apache Flink and AMPS, the Advanced Message Processing System from 60East Technologies. These guides will help you learn how to use the AMPS Flink connector to develop applications with AMPS and Flink. Before reading this guide, it is important to have a good understanding of the following topics: - *Developing Applications in Java* To be successful using this guide, you will need to possess a working knowledge of the Java language. Visit [http://java.oracle.com](http://java.oracle.com) for resources on learning Java. - *AMPS Concepts* The connector uses the AMPS Java client, so we recommend reading about the [AMPS Java Client](https://crankuptheamps.com/clients/amps-client-java). Many features included in the connector require an understanding of AMPS, so we recommend reading the [Introduction to AMPS](/docs/intro-guide/intro) guide. Detailed explanations of the AMPS server behavior are in the [AMPS Server Documentation](/docs). - *Flink Concepts* The connector is used to receive data from AMPS as well as sink data to AMPS. Before working through this guide, we recommend reading the [First Steps](https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/try-flink/local_installation/) documentation to help set up Flink. To better understand how to create applications and programs using Flink, we recommend reading the [Programming Guide](https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/overview/) for Flink's DataStream API. ## Setting up Development Instances In order to use the connector, a Flink cluster and a running AMPS server are required. You can write and compile programs that use the connector without a Flink cluster or AMPS instance, but you will get more out of this guide by running the programs against a Flink cluster and a working AMPS server. ### Setting up an AMPS Development Instance Instructions for starting an instance of AMPS are available in the [Introduction to AMPS](/docs/intro-guide/intro) guide. :::tip The AMPS server runs on x64 Linux. The [Introduction to AMPS](/docs/intro-guide/intro) and [AMPS FAQ](/faq) contain information on how to run an AMPS server on a development system that does not run Linux. ::: ### Setting up a Flink Cluster Instructions for starting a Flink cluster are available in the [First Steps](https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/try-flink/local_installation/) section from Flink. :::tip Although Java 11 is the minimum required version to run Flink, the [recommended version](https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/java_compatibility/#java-17) for running Flink 2.x is Java 17. The connectors were built with Java 17 in mind and may not be compatible with lower Java versions. ::: --- # Examples Several examples of Flink jobs that use different aspects of the AMPS Flink connector are located in the repository for the connector: https://github.com/60East/amps-integration-apache-flink :::tip `flink-connector-amps-examples/src/main/com/crankuptheamps/flink/example/helper/Constants.java` may need to have the URI constants updated with the IP address or DNS name of the host running AMPS unless you are running both the Flink cluster and the AMPS server on the same system. ::: The repository includes jobs such as: | Example Name | Demonstrates | | ------------ | ------------ | | `AMPSSinkExample.java` | The AMPS Sink publishing messages to AMPS. | | `AMPSSourceExample.java` | The AMPS Source receiving messages from AMPS. | | `AggregationExample.java` | A job that uses Flink's aggregation and watermarks to aggregate messages from AMPS. | | `BatchExample.java` | A job that uses topN to demonstrate bounded/batch usage of the AMPS Source. | | `BlogExample.java` | Jobs that use Flink's windowing to monitor a data stream from AMPS. This example is further explored in a blog. | | `CheckpointExample.java` | A job that intentionally throws exceptions to demonstrate the AMPS Source and AMPS Sink delivery guarantees as well as their fault tolerance. | | `CustomDeserializerExample.java` | A job that uses a custom deserialization schema when deserializing messages from AMPS. | | `CustomSerializerExample.java` | A job that uses a custom serialization schema when serializing messages being published to AMPS. | | `MessageQueueExample.java` | Jobs that use message queues with the connectors. Also includes an example that demonstrates fault tolerance when using the connectors with a message queue. | | `ParallelSourceExample.java` | A job that highlights how messages are not lost when using parallelism with the AMPS Sources. | | `PublishStoreExample.java` | A job that shows how publish stores can be used by the AMPS Sink to ensure all messages are delivered. | | `ReplicationExample.java` | Jobs that revolve around replication with the connectors. | | `SOWExample.java` | A job that focuses on SOW queries. | | `SSLExample.java` | A job that is a modification of SimpleExample.java that involves setting up an SSLContext for the AMPS clients used by the connectors. | | `SimpleExample.java` | A job that uses the connectors to publish and receive ad hoc messages. | | `SoakTest.java` | A long-running job that uses ad hoc messages to ensure the connectors do not encounter a failure when running for long periods of time. | | `TableExample.java` | Jobs that involve using the Flink table API along with the connectors. | | `VolumeExample.java` | A job that receives a large volume of messages from an AMPS Source and publishes those messages back to AMPS with an AMPS Sink. | --- # Your First Flink Job with the Connector This chapter provides a basic walkthrough using the connector in a Maven project. For this example, we will create a simple Maven project manually. ## Initialize the Maven Project First, create a new directory that will be used as the Maven project. The project will be used to create the JAR that we will submit to the Flink cluster. ```bash mkdir first-program ``` Now, enter the newly created directory. ```bash cd first-program ``` Create a `pom.xml` and copy the following into the file: ```xml 4.0.0 example first-program 1.0.0 first-program jar UTF-8 17 2.2.0 5.3.5.1 1.2.2-2.2 com.crankuptheamps amps-client ${amps.client.version} com.crankuptheamps.flink flink-connector-amps ${amps.flink.connector.version} org.apache.flink flink-streaming-java ${flink.version} provided org.apache.maven.plugins maven-shade-plugin 3.6.2 package shade *:* META-INF/MANIFEST.MF META-INF/DEPENDENCIES META-INF/LICENSE example.FirstProgram false org.apache.maven.plugins maven-compiler-plugin 3.11.0 ``` Create a directory for the source code. ```bash mkdir src && mkdir src/main && mkdir src/main/java && mkdir src/main/java/example ``` Create a file that will submit a job to Flink: ```bash touch src/main/java/example/FirstProgram.java ``` Copy the following into the file: ```java package example; import com.crankuptheamps.client.Client; import com.crankuptheamps.client.Message; import com.crankuptheamps.client.MessageHandler; import com.crankuptheamps.flink.source.AMPSSource; import com.crankuptheamps.flink.sink.AMPSSink; import org.apache.flink.api.common.eventtime.WatermarkStrategy; import org.apache.flink.api.common.serialization.SimpleStringSchema; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.streaming.api.functions.ProcessFunction; import org.apache.flink.util.Collector; public class FirstProgram { public static void main(String[] args) throws Exception { String uri = "tcp://localhost:9006/amps/json"; String initTopic = "initial-topic"; String modifiedTopic = "modified-topic"; int pubCount = 100; try (Client client = new Client("client");) { client.connect(uri); client.logon(); for (int i = 0; i < pubCount; i++) { client.publish(initTopic, String.format("{\"id\":%d}", i)); } client.publishFlush(); client.subscribe(new MessageHandler() { @Override public void invoke(Message message) { System.out.println(message.getData()); } }, modifiedTopic, 0); AMPSSource source = AMPSSource.builder() .setUri(uri) .setTopic(initTopic) .setBookmark("0") .setTopN(pubCount) .setDeserializationSchema(new SimpleStringSchema()) .build(); AMPSSink sink = AMPSSink.builder() .setUri(uri) .setTopic(modifiedTopic) .setSerializationSchema(new SimpleStringSchema()) .build(); StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setParallelism(1); env.fromSource(source, WatermarkStrategy.noWatermarks(), "AMPS Source") .process(new ProcessFunction() { @Override public void processElement(String value, ProcessFunction.Context ctx, Collector out) throws Exception { Thread.sleep(200); String outStr = value.substring(0, value.length() - 1); out.collect(outStr + ",\"num\":" + Math.random() + "}"); } }) .sinkTo(sink); env.execute("First Program with AMPS Flink Connectors"); Thread.sleep(1000); } } } ``` Build and package the job. This will create the JAR `target/first-program-1.0.0.jar` that should be submitted to Flink. ```bash mvn clean package ``` ## Set Up AMPS and Flink Create an AMPS configuration file in the first-program directory. ```bash touch amps-config.xml ``` Copy the following AMPS configuration into amps-config.xml: ```xml AMPS-First-Connector-Program any-tcp tcp 9006 amps any-ws tcp 9007 websocket 8085 any-ws ./amps-files/amps-flink/stats.db Basic realm="AMPS Admin" stdout error 00-0015 ./amps-files/amps-flink/journals 10MB initial-topic json ``` Open a new terminal, navigate to the first-program directory, and start an AMPS instance using the configuration file. ```bash /path/to/amps/AMPS-{amps_version}-Release-Linux/bin/ampServer amps-config.xml ``` Open a new terminal, navigate to your directory that contains Flink. ```bash cd /path/to/flink-2.2.0 ``` Start the Flink cluster. ```bash bin/start-cluster.sh ``` ## Submit the Job to Flink In the terminal in the Flink directory, submit the job to Flink. ```bash bin/flink run /path/to/first-program/target/first-program-1.0.0.jar ``` ## Examining the Code for the Job Most of the steps above were about setting up a basic Maven project that will create the JAR for Flink. Now, let's take a closer look at the job we are submitting to Flink. ```java package example; /* * These imports are for publishing the initial messages to AMPS and * printing the messages that were modified by Flink and published * back to AMPS by the connectors. They are not necessary for a * real job as they are included for demonstration purposes. */ import com.crankuptheamps.client.Client; import com.crankuptheamps.client.Message; import com.crankuptheamps.client.MessageHandler; /* * These imports are for the connectors. */ import com.crankuptheamps.flink.source.AMPSSource; import com.crankuptheamps.flink.sink.AMPSSink; /* * These imports are for creating and submitting the Flink job. */ import org.apache.flink.api.common.eventtime.WatermarkStrategy; import org.apache.flink.api.common.serialization.SimpleStringSchema; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.streaming.api.functions.ProcessFunction; import org.apache.flink.util.Collector; public class FirstProgram { public static void main(String[] args) throws Exception { /* * The connection URI the client and connectors use to connect to AMPS. */ String uri = "tcp://localhost:9006/amps/json"; /* * The initial topic with a transaction log the client publishes to * and the AMPS Source subscribes to. */ String initTopic = "initial-topic"; /* * The modified ad hoc topic the AMPS Sink publishes to and the * client subscribes to. */ String modifiedTopic = "modified-topic"; /* * The amount of messages the client will publish and the AMPS Source * will read from AMPS. */ int pubCount = 100; /* * Try with resources block to create and close the client. */ try (Client client = new Client("client");) { /* * Connect the client to AMPS. */ client.connect(uri); client.logon(); /* * Publish the initial messages to AMPS. */ for (int i = 0; i < pubCount; i++) { client.publish(initTopic, String.format("{\"id\":%d}", i)); } client.publishFlush(); /* * Subscribe to the modified ad hoc topic. To print * the messages modified by Flink. */ client.subscribe(new MessageHandler() { @Override public void invoke(Message message) { System.out.println(message.getData()); } }, modifiedTopic, 0); /* * Create an AMPS Source that will subscribe to the initial topic. * The bookmark is the EPOCH bookmark, which starts the replay * from the first message in the specified topic. The topN * tells the source to only receive pubCount messages from AMPS * before considering the job finished. The URI, topic, * deserialization schema, and build() are all required to * create a valid AMPS Source. The bookmark and topN are optional. */ AMPSSource source = AMPSSource.builder() .setUri(uri) .setTopic(initTopic) .setBookmark("0") .setTopN(pubCount) .setDeserializationSchema(new SimpleStringSchema()) .build(); /* * Creates an AMPS Sink that will publish to the modified topic. * The URI, topic, serialization schema, and build() are all * required to create a valid AMPS Sink. */ AMPSSink sink = AMPSSink.builder() .setUri(uri) .setTopic(modifiedTopic) .setSerializationSchema(new SimpleStringSchema()) .build(); /* * The Stream Execution Environment is used to create the job * and submit it to Flink. */ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); /* * Parallelism is set to 1 since this example does not use parallelism. */ env.setParallelism(1); /* * Create the job graph. First, we use the AMPS Source to receive data * from AMPS and create a Data Stream of Strings. Then, we use a Process * Function to add a new field to the Strings. Finally, we sink the * Strings back to AMPS using the AMPS Sink. In this example, all three * steps are chained together. */ env.fromSource(source, WatermarkStrategy.noWatermarks(), "AMPS Source") .process(new ProcessFunction() { @Override public void processElement(String value, ProcessFunction.Context ctx, Collector out) throws Exception { /* * Sleep for a short duration to avoid cluttering the console with messages. */ Thread.sleep(200); /* * Remove the trailing '}' to allow a new field to be added. */ String outStr = value.substring(0, value.length() - 1); /* * Collect the String plus a new field 'num'. */ out.collect(outStr + ",\"num\":" + Math.random() + "}"); } }) .sinkTo(sink); /* * Submit the job to Flink. This method is synchronous and will wait until * the job completes to resume the rest of the Java program. Since we * are using topN with a bookmark subscription, the AMPS Source is * BOUNDED and the job will be finished once pubCount messages are * received by the AMPS Source. */ env.execute("First Program with AMPS Flink Connectors"); /* * Sleep for a short duration to make sure all messages are printed to the console. */ Thread.sleep(1000); } } } ``` In this example, we are using a simple AMPS Source and AMPS Sink, and the data type we are working with is just a String. Much more customization is available for the connector such as using user-defined data types and serialization schemas, which can make processing the stream data much more convenient when compared to working with Strings. See the [examples](./examples) for several jobs utilizing different aspects of the connector. --- # Installing and Using the Connector ## Prerequisites Before proceeding with this guide, make sure that the following programs are installed and functioning properly on your development machine: - Java Development Kit version 17 - Java Runtime Environment version 17 - Apache Maven 3.6.3 or greater ## Obtaining the Connector The latest version of the connector can be installed from Maven Central Repository using the following dependency snippet: ```xml com.crankuptheamps.flink flink-connector-amps 1.2.2-2.2 ``` Additionally, the source code can be found at the following repository: https://github.com/60East/amps-integration-apache-flink --- # Welcome to the AMPS Apache Flink Connector This guide provides information you need to get started with the AMPS Flink connector. It focuses specifically on the connector. AMPS, the AMPS Java client, and Apache Flink are not covered in detail. For an overview of AMPS and instructions on setting up your development environment, refer to the [Introduction to AMPS](https://crankuptheamps.com/docs/intro-guide/intro) guide. For an overview of the AMPS Java client, refer to the [AMPS Java Client](https://crankuptheamps.com/clients/amps-client-java) guide. For an overview of Apache Flink, refer to the [Apache Flink Website](https://flink.apache.org/) and the [Apache Flink Documentation](https://nightlies.apache.org/flink/flink-docs-stable/). :::tip This guide assumes that you have a development environment for Java, access to an AMPS server, and access to Apache Flink 2.x to run any examples/samples. ::: --- # AMPS Connectors and Integrations This section provides documentation for the connectors and integrations sponsored by 60East Technologies, Inc. In this section: * [AMPS Apache Flink Connector](./flink/intro.md) * [NGINX Proxy Configuration](./nginx-proxy/intro.md) * [Apache Proxy Configuration](./apache-proxy/intro.md) --- # Configuration The sink connector is configured with Kafka Connect properties. These properties can be supplied in a JSON request to a distributed worker or in a properties file for a standalone worker. ## Required The following properties are required to construct a valid AMPS sink connector. ### Kafka Connect | Property | Description | | -------- | ----------- | | `connector.class` | Must be `com.crankuptheamps.kafka.AMPSKafkaSink`. | | `topics` or `topics.regex` | Use exactly one of these properties. `topics` is a Kafka topic or comma-separated list of topics to consume from. `topics.regex` is a Java regular expression for Kafka topics to consume from. | | `tasks.max` | Maximum number of sink tasks. A value of `1` is recommended unless the pipeline is designed for parallel consumption. | ### Connection to AMPS | Property | Description | | -------- | ----------- | | `uri` | AMPS server URI or comma-separated list of URIs, such as `tcp://localhost:9007/amps/json`. | | `clientName` | Base name for the AMPS client. Task suffixes are added automatically when multiple tasks are used. | ### Topic for AMPS | Property | Description | | -------- | ----------- | | `ampsTopic` | AMPS topic to publish to when `useTopicHeader` is `false`, or the fallback topic when `useTopicHeader` is `true`. | ## Optional ### AMPS Client The following properties are optional for functionality related to the AMPS client. | Property | Description | | -------- | ----------- | | `clientFactoryClass` | Class used to construct the AMPS client. Defaults to `com.crankuptheamps.kafka.AMPSBasicClientFunction`. | | `maxBatch` | Maximum number of Kafka records the sink processes in a batch. Defaults to `1000`. | | `useTopicHeader` | When `true`, publish each record to an AMPS topic matching the Kafka record topic. When `false`, publish to `ampsTopic`. Defaults to `true`. | | `publishFlushTimeout` | Timeout passed to AMPS `publishFlush` at the end of a batch. A value of `-1` disables the flush. A value of `0` waits indefinitely for the batch to persist. Defaults to `-1`. | | `logMsgOnWriteError` | When `true`, write the failed message to the Kafka Connect log when an AMPS failed write is detected. Defaults to `true`. | ### Publish Stores The following properties configure AMPS publish stores for the sink client. | Property | Description | | -------- | ----------- | | `pubStoreType` | Publish store type. Valid values are `memory` and `file`. | | `pubStoreInitialCap` | Publish store initial capacity in 2KB blocks. Defaults to `1000`. | | `pubStorePath` | File path for a file-backed publish store. Required when `pubStoreType` is `file`. | | `pruneTimeThreshold` | Minimum time in milliseconds between file-backed store prune operations. Defaults to 300000. | --- # Delivery Guarantees The AMPS Kafka sink publishes Kafka record values to AMPS. The main delivery guarantees are provided by the AMPS publish store configuration. ## Publish Stores The sink can create an AMPS publish store through `pubStoreType`. | Store Type | Description | | ---------- | ----------- | | `memory` | Stores outgoing publishes in the worker process. This is fast, but does not protect against process failure. | | `file` | Stores outgoing publishes on disk. This can recover after client or worker failure and depends on the performance of the storage device. | The following configuration uses a memory publish store: ```json { "name": "amps-kafka-sink", "config": { "connector.class": "com.crankuptheamps.kafka.AMPSKafkaSink", "topics": "Orders", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "header.converter": "org.apache.kafka.connect.storage.StringConverter", "clientFactoryClass": "com.crankuptheamps.kafka.AMPSBasicClientFunction", "clientName": "KafkaSink", "pubStoreType": "memory", "pubStoreInitialCap": "100", "uri": "tcp://localhost:9007/amps/json", "ampsTopic": "AMPSKafkaSinkTest", "maxBatch": "100", "useTopicHeader": "false", "publishFlushTimeout": "0" } } ``` The following properties use a file-backed publish store: ```json { "pubStoreType": "file", "pubStoreInitialCap": "1000", "pubStorePath": "./KafkaSink.PubStore", "pruneTimeThreshold": "300000" } ``` ## Failed Writes The sink registers an AMPS failed write handler. When AMPS reports a failed write, the connector logs the reason. If `logMsgOnWriteError` is `true`, the connector also logs the message that failed when the AMPS client has enough stored information to provide it. --- # Parallelism Kafka Connect provides scalability through connector tasks. The AMPS Kafka sink can run multiple tasks so Kafka Connect can assign topic partitions across multiple AMPS clients. ## AMPS Sink Parallelism The following example allows up to three sink tasks: ```json { "name": "amps-kafka-sink", "config": { "connector.class": "com.crankuptheamps.kafka.AMPSKafkaSink", "topics": "Orders", "tasks.max": "3", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "header.converter": "org.apache.kafka.connect.storage.StringConverter", "clientFactoryClass": "com.crankuptheamps.kafka.AMPSBasicClientFunction", "clientName": "KafkaSink", "uri": "tcp://localhost:9007/amps/json", "ampsTopic": "AMPSKafkaSinkTest", "maxBatch": "100", "useTopicHeader": "false" } } ``` :::info The sink appends the task index to the configured `clientName`. For example, `KafkaSink` becomes `KafkaSink_0`, `KafkaSink_1`, and `KafkaSink_2`. ::: ## Topic Regular Expressions The sink can consume from topics selected by a Java regular expression: ```json { "topics.regex": "Test.*", "useTopicHeader": "true" } ``` When `useTopicHeader` is `true`, each Kafka record is published to an AMPS topic with the same name as the Kafka topic. When using `topics.regex` or multiple Kafka topics, set `useTopicHeader` to `false` if all matching Kafka topics should publish into a single AMPS topic. ## Order When `tasks.max` is greater than `1`, Kafka Connect can process different Kafka partitions in different sink tasks. There is no global ordering guarantee across tasks. More than one sink task is not recommended if exact ordering is required. --- # AMPS Sink The `AMPSKafkaSink` reads records from Kafka through Kafka Connect and publishes them to AMPS. The sink is responsible for consuming Kafka records, selecting the AMPS topic, and publishing each record value as an AMPS message payload. The following is an example of a simple sink connector configuration. In this example, the sink consumes from the Kafka topic `SourceTest` and publishes all records to the AMPS topic `SinkTest`. ```json { "name": "amps-kafka-sink", "config": { "connector.class": "com.crankuptheamps.kafka.AMPSKafkaSink", "topics": "SourceTest", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "header.converter": "org.apache.kafka.connect.storage.StringConverter", "clientFactoryClass": "com.crankuptheamps.kafka.AMPSBasicClientFunction", "clientName": "KafkaSink", "uri": "tcp://localhost:9007/amps/json", "ampsTopic": "SinkTest", "maxBatch": "1000", "useTopicHeader": "false" } } ``` :::info A valid `AMPSKafkaSink` requires a connector class, Kafka topics or a Kafka topic regular expression, a URI that can successfully connect to AMPS, an AMPS client name, and an AMPS topic. The example above also sets `useTopicHeader` to `false` so every record is published to the configured `ampsTopic`. ::: ## Topic Selection By default, `useTopicHeader` is `true`. When this option is true, the sink publishes each Kafka record to an AMPS topic with the same name as the Kafka topic that supplied the record. Set `useTopicHeader` to `false` to publish all records to the configured `ampsTopic`. ## Message Values The sink expects each Kafka record value to be a byte array and publishes those bytes to AMPS. Use `org.apache.kafka.connect.converters.ByteArrayConverter` for `value.converter` unless another converter is intentionally paired with the record value type. --- # Configuration The source connector is configured with Kafka Connect properties. These properties can be supplied in a JSON request to a distributed worker or in a properties file for a standalone worker. ## Required The following properties are required to construct a valid AMPS source connector. ### Kafka Connect | Property | Description | | -------- | ----------- | | `connector.class` | Must be `com.crankuptheamps.kafka.AMPSKafkaSource`. | | `tasks.max` | Maximum number of source tasks. A value of `1` is recommended unless the source is explicitly partitioned with `taskFilter`. | ### Connection to AMPS | Property | Description | | -------- | ----------- | | `uri` | AMPS server URI or comma-separated list of URIs, such as `tcp://localhost:9007/amps/json`. | | `clientName` | Base name for the AMPS client. Task suffixes are added automatically when multiple tasks are used. | ### Topic from AMPS | Property | Description | | -------- | ----------- | | `ampsTopic` | AMPS topic or regular expression topic used by the subscription or query. | ## Optional ### AMPS Client The following properties are optional for functionality related to the AMPS client and AMPS command. | Property | Description | | -------- | ----------- | | `clientFactoryClass` | Class used to construct the AMPS client. Defaults to `com.crankuptheamps.kafka.AMPSBasicClientFunction`. | | `command` | AMPS command type. Defaults to `subscribe`. Common values include `subscribe` and `sow_and_subscribe`. | | `filter` | AMPS content filter expression. | | `options` | AMPS command options, such as projection, grouping, conflation, or other subscription options. | | `cmdTypeFilter` | Bit mask of AMPS command type integers to include. Messages with command types outside the mask are ignored. | | `eventHeaders` | Comma-separated list of AMPS message headers to preserve as Kafka headers. | Valid `eventHeaders` values include: | Header | Description | | ------ | ----------- | | `commandHeader` | Command type of the AMPS message. | | `topicHeader` | Topic name of the AMPS message. | | `sowKey` | SOW key of the AMPS message. Only set for SOW subscriptions. | | `ampsTimestamp` | ISO-8601 timestamp of when AMPS processed the message. | | `bookmarkHeader` | Bookmark string of the AMPS message. Only set for bookmark subscriptions. | | `correlationId` | Correlation ID of the AMPS message. | | `subId` | Subscription ID of the AMPS message. | | `length` | Length of the AMPS message body in bytes. | | `timestamp` | Timestamp from when the source received the message. | ### Bookmark Subscriptions The following properties are optional and can be configured for bookmark subscriptions. | Property | Description | | -------- | ----------- | | `bookmark` | Starting bookmark for the subscription. Accepts values such as `NOW`, `EPOCH`, `MOST_RECENT`, or any valid AMPS bookmark. | | `subscriptionId` | AMPS subscription ID. Required when `bookmark` is specified. | | `bookmarkStoreType` | Bookmark store type. Valid values are `memory` and `logged`. | | `bookmarkLog` | Bookmark log file path. Required when using a logged bookmark store. | | `pruneTimeThreshold` | Minimum time in milliseconds between logged bookmark store prune operations. Defaults to 300000. | ### Queues The following properties are optional and can be configured when subscribing to an AMPS queue. | Property | Description | | -------- | ----------- | | `isQueue` | Set to `y` when `ampsTopic` is an AMPS queue. | | `maxBacklog` | Maximum number of unacknowledged messages AMPS will provide to the queue subscription. | | `ackBatchSize` | Number of queue messages to include in an acknowledgment batch. | | `ackTimeout` | Timeout value for sending queue acknowledgments. | ### Batching and Parallelism The following properties control batching and task partitioning. | Property | Description | | -------- | ----------- | | `maxBuffers` | Maximum number of message buffers used by the source task. Defaults to `10`. | | `maxBatch` | Maximum number of AMPS messages returned to Kafka Connect in each poll batch. Defaults to `1000`. | | `taskFilter` | Filter format used to partition messages across source tasks. Required when `tasks.max` is greater than `1`. | :::info The source calls `String.format(taskFilter, tasks.max, taskIndex)` to build a task-specific AMPS filter. For example, a `taskFilter` of `/id MOD %d = %d` with `tasks.max` set to `2` produces `/id MOD 2 = 0` for one task and `/id MOD 2 = 1` for the other. ::: --- # Delivery Guarantees Kafka Connect commits source records after they have been written to Kafka. The AMPS Kafka source uses that commit callback to discard bookmark store entries or acknowledge queue messages in AMPS. ## Topics with a Transaction Log For AMPS topics with a transaction log, bookmark subscriptions can be used to provide resumable delivery. When the source is configured with a bookmark store, records are not discarded from the store until Kafka Connect commits the corresponding Kafka record. The following source uses an `EPOCH` bookmark subscription and a memory bookmark store: ```json { "name": "amps-kafka-source", "config": { "connector.class": "com.crankuptheamps.kafka.AMPSKafkaSource", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "header.converter": "org.apache.kafka.connect.storage.StringConverter", "clientFactoryClass": "com.crankuptheamps.kafka.AMPSBasicClientFunction", "clientName": "KafkaSource", "uri": "tcp://localhost:9007/amps/json", "ampsTopic": "SourceTest", "bookmark": "EPOCH", "bookmarkStoreType": "memory", "subscriptionId": "Sub-200", "maxBuffers": "10", "maxBatch": "1000" } } ``` :::info `subscriptionId` is required when `bookmark` is specified. Use a unique subscription ID for each independent subscription in the application. ::: ## Bookmark Stores The connector supports memory and logged bookmark stores. | Store Type | Description | | ---------- | ----------- | | `memory` | Stores bookmarks in the worker process. This is the highest performance option, but it does not protect against process failure. | | `logged` | Stores bookmarks on disk. This provides recovery after client or worker failure, and performance depends on the storage device used for the bookmark log. | A logged bookmark store also requires `bookmarkLog`: ```json { "bookmark": "MOST_RECENT", "bookmarkStoreType": "logged", "subscriptionId": "Sub-200", "bookmarkLog": "./bookmark.log", "pruneTimeThreshold": "300000" } ``` :::warning The source provides at-least-once delivery for bookmark subscriptions. The guarantee is managed by the source and the AMPS client bookmark store. If a failure happens after Kafka commits a record but before the bookmark store discard completes, the record can be redelivered after restart. ::: ## Message Queues The source can subscribe directly to AMPS queues. Set `isQueue` to `y` and configure the queue acknowledgment settings for the subscription. AMPS queue configuration determines whether queue delivery is at-most-once or at-least-once. The source uses the same guarantees as a regular AMPS queue consumer: the connector subscribes to the configured queue, and AMPS applies the delivery semantics configured for that queue. ```json { "name": "amps-kafka-source", "config": { "connector.class": "com.crankuptheamps.kafka.AMPSKafkaSource", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "header.converter": "org.apache.kafka.connect.storage.StringConverter", "clientFactoryClass": "com.crankuptheamps.kafka.AMPSBasicClientFunction", "clientName": "KafkaSource", "uri": "tcp://localhost:9007/amps/json", "ampsTopic": "SourceTest", "isQueue": "y", "maxBacklog": "10000", "ackBatchSize": "3000", "ackTimeout": "60000", "maxBuffers": "10", "maxBatch": "1000" } } ``` For queues, messages are acknowledged after Kafka Connect commits the corresponding Kafka record. If the connector stops before a message is committed to Kafka, AMPS can redeliver the unacknowledged message according to the queue's configured delivery semantics. ## Aggregated Subscriptions The source can use AMPS command options, including projection, grouping, and conflation. The following configuration uses `sow_and_subscribe` with an aggregated subscription: ```json { "name": "amps-kafka-source", "config": { "connector.class": "com.crankuptheamps.kafka.AMPSKafkaSource", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "header.converter": "org.apache.kafka.connect.storage.StringConverter", "clientFactoryClass": "com.crankuptheamps.kafka.AMPSBasicClientFunction", "clientName": "KafkaSource", "uri": "tcp://localhost:9007/amps/json", "command": "sow_and_subscribe", "ampsTopic": "Orders", "filter": "/symbol IN ('IBM', 'MSFT')", "options": "projection=[/symbol,avg(/price) as /avg_price, avg(/price*/qty) as /avg_total],grouping=[/symbol],conflation=1s", "subscriptionId": "Sub-100", "cmdTypeFilter": "9", "maxBuffers": "10", "maxBatch": "1000", "pruneTimeThreshold": "300000", "eventHeaders": "topicHeader,timestamp" } } ``` --- # Parallelism Kafka Connect provides scalability through connector tasks. The AMPS Kafka source can run multiple tasks when the AMPS subscription is partitioned with task-specific filters. ## AMPS Source Parallelism When `tasks.max` is greater than `1`, the source requires `taskFilter`. The connector uses the configured `taskFilter` as a format string and generates a different AMPS content filter for each task. The following example creates two source tasks and partitions messages by `/id`: ```json { "name": "amps-kafka-source", "config": { "connector.class": "com.crankuptheamps.kafka.AMPSKafkaSource", "tasks.max": "2", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "header.converter": "org.apache.kafka.connect.storage.StringConverter", "clientFactoryClass": "com.crankuptheamps.kafka.AMPSBasicClientFunction", "clientName": "KafkaSource", "uri": "tcp://localhost:9007/amps/json", "ampsTopic": "SourceTest", "taskFilter": "/id MOD %d = %d", "maxBuffers": "10", "maxBatch": "1000" } } ``` With this configuration, the connector creates one task using `/id MOD 2 = 0` and another task using `/id MOD 2 = 1`. If `filter` is also configured, the connector combines the base filter and the task filter with `AND`. :::info The source also appends the task index to the configured `clientName`. For example, `KafkaSource` becomes `KafkaSource_0` and `KafkaSource_1`. ::: ## Order Messages remain ordered within each task's AMPS subscription, but there are no order guarantees across tasks. If exact global ordering is required, use a single source task. --- # AMPS Source The `AMPSKafkaSource` reads data from AMPS and writes Kafka source records through Kafka Connect. The source is responsible for subscribing or querying AMPS, receiving messages, optionally acknowledging queue messages or discarding bookmark store entries, and passing message payloads to Kafka. The source connector is pollable. An AMPS client receives messages asynchronously. The AMPS source buffers those messages in batches, and returns those batches when Kafka Connect polls this source task. This avoids committing each message individually as it arrives. The following is an example of a simple source connector configuration. In this example, the source subscribes to the AMPS topic `SourceTest` and writes records to the Kafka topic with the same topic name received from AMPS. ```json { "name": "amps-kafka-source", "config": { "connector.class": "com.crankuptheamps.kafka.AMPSKafkaSource", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "header.converter": "org.apache.kafka.connect.storage.StringConverter", "clientFactoryClass": "com.crankuptheamps.kafka.AMPSBasicClientFunction", "clientName": "KafkaSource", "uri": "tcp://localhost:9007/amps/json", "ampsTopic": "SourceTest", "maxBuffers": "10", "maxBatch": "1000" } } ``` :::info A valid `AMPSKafkaSource` requires a connector class, a connection to AMPS, an AMPS client name, and a topic or topic expression to subscribe to. The example above also sets Kafka Connect converters so record values are handled as bytes. ::: ## Topic Names The source uses the topic name on each AMPS message as the destination Kafka topic. When `ampsTopic` is a regular expression topic, records can be written to multiple Kafka topics that match the AMPS message topics. ## Message Values The source copies the AMPS message body into the Kafka record value as a byte array. Use `org.apache.kafka.connect.converters.ByteArrayConverter` for `value.converter` unless another converter is intentionally paired with the data format being produced. --- # Before You Start Welcome to developing Kafka Connect pipelines with Apache Kafka and AMPS, the Advanced Message Processing System from 60East Technologies. These guides will help you learn how to use the AMPS Kafka connector to move messages between AMPS and Apache Kafka. Before reading this guide, it is important to have a good understanding of the following topics: - *Developing Applications in Java* To be successful using this guide, you will need to possess a working knowledge of the Java language. Visit [http://java.oracle.com](http://java.oracle.com) for resources on learning Java. - *AMPS Concepts* The connector uses the AMPS Java client, so we recommend reading about the [AMPS Java Client](https://crankuptheamps.com/clients/amps-client-java). Many features included in the connector require an understanding of AMPS, so we recommend reading the [Introduction to AMPS](/docs/intro-guide/intro) guide. Detailed explanations of the AMPS server behavior are in the [AMPS Server Documentation](/docs). - *Kafka and Kafka Connect Concepts* The connector is used by Kafka Connect to receive data from AMPS and to sink data to AMPS. Before working through this guide, we recommend reading the [Kafka Connect](https://kafka.apache.org/documentation/#connect) documentation to understand workers, connector plugins, source connectors, sink connectors, converter settings, and standalone or distributed mode. The connector examples use a Kafka worker, a Kafka broker, and an AMPS server running on the development system. ## Setting up Development Instances In order to use the connector, a Kafka Connect worker, a Kafka broker, and a running AMPS server are required. You can build the connector without a Kafka or AMPS instance, but you will get more out of this guide by running the examples against working development instances. ### Setting up an AMPS Development Instance Instructions for starting an instance of AMPS are available in the [Introduction to AMPS](/docs/intro-guide/intro) guide. :::tip The AMPS server runs on x64 Linux. The [Introduction to AMPS](/docs/intro-guide/intro) and [AMPS FAQ](/faq) contain information on how to run an AMPS server on a development system that does not run Linux. ::: ### Setting up Kafka and Kafka Connect Instructions for starting Apache Kafka are available in the [Apache Kafka Quickstart](https://kafka.apache.org/quickstart) guide. Kafka Connect is included with Apache Kafka distributions. You can run the connector in either standalone mode or distributed mode. Distributed mode is recommended for the connector examples because connectors can be loaded and managed through the Kafka Connect REST API. :::tip The connector is built for Java 8 or higher. If you are using Kafka 4.0 or higher, use Java 17 or higher for the Kafka runtime. ::: --- # Examples Several examples that use different aspects of the AMPS Kafka connector are located in the connector repository under `src/test/resources/examples/`. :::tip The examples assume Kafka Connect can find the AMPS Kafka connector plugin and that the AMPS Java client JAR is available in the plugin directory. ::: ## Included JSON Examples The repository includes these connector JSON files: | Example | Connector | Demonstrates | | ------- | --------- | ------------ | | `AMPSKafkaGenericSource.json` | Source | A basic AMPS source connector that subscribes to `SourceTest` and writes records to Kafka. | | `AMPSKafkaGenericSink.json` | Sink | A basic AMPS sink connector that consumes `SourceTest` from Kafka and publishes to `SinkTest` in AMPS. | | `aggSub/AMPSKafkaSource.json` | Source | An AMPS `sow_and_subscribe` with a content filter, projection, grouping, conflation, command type filtering, and Kafka headers. | | `aggSub/AMPSKafkaSink.json` | Sink | A sink connector that consumes `Orders` from Kafka and publishes to `AMPSKafkaSinkTest` in AMPS. | | `bookmarkSub/AMPSKafkaMemoryBookmarkSource.json` | Source | A bookmark source using the `EPOCH` bookmark and a memory bookmark store. | | `bookmarkSub/AMPSKafkaLoggedBookmarkSource.json` | Source | A bookmark source using the `MOST_RECENT` bookmark and a logged bookmark store. | | `queueSub/AMPSKafkaQueueSource.json` | Source | An AMPS queue subscription that acknowledges messages after Kafka commits them. | | `dynamicSub/AMPSKafkaDynamicSource.json` | Source | A source connector that subscribes to AMPS topics with a regular expression. | | `dynamicSub/AMPSKafkaDynamicSink.json` | Sink | A sink connector that consumes Kafka topics selected by `topics.regex` and publishes to matching AMPS topics with `useTopicHeader`. | | `failoverTest/AMPSKafkaFailTestSource.json` | Source | A source connector configured with multiple AMPS URIs and a memory bookmark store for failover testing. | | `failoverTest/AMPSKafkaFailTestSink.json` | Sink | A sink connector configured with multiple AMPS URIs and a memory publish store for failover testing. | The example directory also includes `aggSub/messages.json`, a JSON-lines data file with 15 `Orders` messages used by the aggregate subscription walkthrough and tests. ## JUnit Integration Tests The connector repository includes JUnit tests in `src/test/java/com/crankuptheamps/kafka/AMPSKafkaExampleIntegrationTest.java`. These tests use AMPS clients, Kafka, and the Kafka Connect REST API to create the example connectors from `src/test/resources/examples/`, publish messages, and verify that the expected messages arrive back in AMPS. | Test | Demonstrates | | ---- | ------------ | | `basicFlowTest` | Uses `AMPSKafkaGenericSource.json` and `AMPSKafkaGenericSink.json` to verify the basic AMPS to Kafka to AMPS path with fixed-size JSON messages. | | `dynamicSubTest` | Uses `dynamicSub/AMPSKafkaDynamicSource.json` and `dynamicSub/AMPSKafkaDynamicSink.json` to verify regular expression topic subscription and sink publication using Kafka topic headers. | | `deliveryQueueTest` | Uses `queueSub/AMPSKafkaQueueSource.json` with the generic sink to verify AMPS queue consumption and delivery back to an AMPS sink topic. | | `bookmarkTest` | Uses `bookmarkSub/AMPSKafkaMemoryBookmarkSource.json` and `bookmarkSub/AMPSKafkaLoggedBookmarkSource.json` to verify memory and logged bookmark source configurations. | | `aggregateTest` | Uses `aggSub/AMPSKafkaSource.json`, `aggSub/AMPSKafkaSink.json`, and `aggSub/messages.json` to verify aggregate subscription filtering, grouping, projection, conflation, and sink delivery. | | `failoverTest` | Uses `failoverTest/AMPSKafkaFailTestSource.json` and `failoverTest/AMPSKafkaFailTestSink.json` to verify that HA source and sink connectors continue delivering messages after the primary AMPS TCP transport is disabled. | :::info The JUnit suite requires Kafka, Kafka Connect, and two AMPS servers to be running before `mvn test` starts. See `docs/TEST.md` in the connector repository for the test setup, default endpoints, Maven command, and single-test command. ::: --- # Your First AMPS Kafka Connect Pipeline This chapter provides a basic walkthrough using the connector with Kafka Connect. For this example, the AMPS Kafka source subscribes to AMPS, writes records into Kafka, and the AMPS Kafka sink reads those records from Kafka and publishes them back to AMPS. The connector repository includes the same example under `src/test/resources/examples/aggSub/`. ## Start Kafka Start a local Kafka broker from the Kafka installation directory. For KRaft mode, generate a cluster ID, format the log directories, and start the server: ```bash KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c config/server.properties bin/kafka-server-start.sh config/server.properties ``` For ZooKeeper-based Kafka distributions, start ZooKeeper and then start the server: ```bash bin/zookeeper-server-start.sh config/zookeeper.properties bin/kafka-server-start.sh config/server.properties ``` ## Start AMPS From the connector repository directory, start AMPS with the configuration file provided by the connector example: ```bash AMPS-{version}-Release-Linux/bin/ampServer src/test/resources/examples/aggSub/amps-config.xml ``` ## Start Kafka Connect From the Kafka installation directory, start a distributed Kafka Connect worker: ```bash bin/connect-distributed.sh config/connect-distributed.properties ``` :::tip Make sure the worker `plugin.path` includes the directory where the AMPS Kafka connector plugin was installed before starting Kafka Connect. ::: ## Load the Source Connector From the connector repository directory, load the source connector: ```bash curl -X POST -H "Content-Type:application/json" http://localhost:8083/connectors --data @src/test/resources/examples/aggSub/AMPSKafkaSource.json ``` The source connector configuration uses an AMPS aggregated subscription: ```json { "name": "amps-kafka-source", "config": { "connector.class": "com.crankuptheamps.kafka.AMPSKafkaSource", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "header.converter": "org.apache.kafka.connect.storage.StringConverter", "clientFactoryClass": "com.crankuptheamps.kafka.AMPSBasicClientFunction", "clientName": "KafkaSource", "uri": "tcp://localhost:9007/amps/json", "command": "sow_and_subscribe", "ampsTopic": "Orders", "filter": "/symbol IN ('IBM', 'MSFT')", "options": "projection=[/symbol,avg(/price) as /avg_price, avg(/price*/qty) as /avg_total],grouping=[/symbol],conflation=1s", "subscriptionId": "Sub-100", "cmdTypeFilter": "9", "maxBuffers": "10", "maxBatch": "1000", "pruneTimeThreshold": "300000", "eventHeaders": "topicHeader,timestamp" } } ``` ## Load the Sink Connector From the connector repository directory, load the sink connector: ```bash curl -X POST -H "Content-Type:application/json" http://localhost:8083/connectors --data @src/test/resources/examples/aggSub/AMPSKafkaSink.json ``` The sink reads from the Kafka topic `Orders` and publishes records to the AMPS topic `AMPSKafkaSinkTest`: ```json { "name": "amps-kafka-sink", "config": { "connector.class": "com.crankuptheamps.kafka.AMPSKafkaSink", "topics": "Orders", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "header.converter": "org.apache.kafka.connect.storage.StringConverter", "clientFactoryClass": "com.crankuptheamps.kafka.AMPSBasicClientFunction", "clientName": "KafkaSink", "pubStoreType": "memory", "pubStoreInitialCap": "100", "uri": "tcp://localhost:9007/amps/json", "ampsTopic": "AMPSKafkaSinkTest", "maxBatch": "100", "useTopicHeader": "false" } } ``` ## Subscribe to the Sink Output Use the AMPS `spark` utility to subscribe to the topic where the sink publishes records: ```bash AMPS-{version}-Release-Linux/bin/spark subscribe -server localhost:9007 -topic AMPSKafkaSinkTest ``` ## Publish Example Messages From the connector repository directory, publish the example JSON messages to the `Orders` topic: ```bash AMPS-{version}-Release-Linux/bin/spark publish -server localhost:9007 -topic Orders -rate 1 -file src/test/resources/examples/aggSub/messages.json ``` The source filters for `/symbol` values of `IBM` and `MSFT`, uses a one second conflation interval, and projects aggregate fields. The sink subscriber receives the processed updates after they pass from AMPS to Kafka and back into AMPS. --- # Installing and Using the Connector ## Prerequisites Before proceeding with this guide, make sure that the following programs are installed and functioning properly on your development machine: - Java Development Kit version 8 or greater - Java Runtime Environment version 8 or greater - Apache Maven 3.3 or greater - Apache Kafka with Kafka Connect - AMPS Java client JAR :::tip Kafka 4.0 and later require Java 17 or greater. ::: ## Obtaining the Connector The connector can be installed from Maven Central Repository using the following dependency snippet: ```xml com.crankuptheamps.kafka amps-kafka-connector 1.2.0 ``` Additionally, the source code can be found at the following repository: https://github.com/60East/amps-integration-apache-kafka ## Installing the Kafka Connect Plugin Kafka Connect loads connectors from directories listed in the worker `plugin.path`. Create a plugin directory for AMPS and copy the connector JAR and the AMPS Java client JAR into it: ```text plugins/ amps/ amps-kafka-connector-1.2.0.jar amps_client.jar ``` Update the worker configuration to include the parent plugin directory. For standalone mode, update `config/connect-standalone.properties`: ```properties plugin.path=/path/to/plugins ``` For distributed mode, update `config/connect-distributed.properties`: ```properties plugin.path=/path/to/plugins ``` After changing `plugin.path`, restart the Kafka Connect worker so it discovers the connector classes. ## Connector Classes The AMPS Kafka connector provides a source connector and a sink connector: | Connector | Class | | --------- | ----- | | AMPS Source | `com.crankuptheamps.kafka.AMPSKafkaSource` | | AMPS Sink | `com.crankuptheamps.kafka.AMPSKafkaSink` | Use these class names in the Kafka Connect connector configuration. --- # Welcome to the AMPS Apache Kafka Connector This guide provides information you need to get started with the AMPS Kafka connector. It focuses specifically on the connector and Kafka Connect. AMPS, the AMPS Java client, and Apache Kafka are not covered in detail. For an overview of AMPS and instructions on setting up your development environment, refer to the [Introduction to AMPS](https://crankuptheamps.com/docs/intro-guide/intro) guide. For an overview of the AMPS Java client, refer to the [AMPS Java Client](https://crankuptheamps.com/clients/amps-client-java) guide. For an overview of Apache Kafka and Kafka Connect, refer to the [Apache Kafka Website](https://kafka.apache.org/) and the [Apache Kafka Documentation](https://kafka.apache.org/documentation/). :::tip This guide assumes that you have a development environment for Java, access to an AMPS server, and access to an Apache Kafka installation that includes Kafka Connect to run any examples. ::: --- # Updating AMPS Client Connections ## Client Connection Strings Use the URI that matches the proxy and AMPS transport configuration: - Secure TCP connections: `tcps://localhost:443/tcps/amps/json?http_preflight=true` - Secure WebSocket connections: `wss://localhost:443/wss/amps/json` - Standard non-SSL TCP connections: `tcp://localhost:80/tcp/amps/json?http_preflight=true` - Standard non-SSL WebSocket connections: `ws://localhost:80/ws/amps/json` The code examples below all use the secure URLs. For a non-SSL deployment, replace `tcps` with `tcp`, `wss` with `ws`, port `443` with `80`, and the `/tcps/` and `/wss/` paths with `/tcp/` and `/ws/`. WebSocket connections don't require the HTTP Preflight option as they use an HTTP Upgrade mechanism to upgrade to the WebSocket protocol by default. For more information on HTTP Preflight, see [HTTP Preflight](/docs/amps-user-guide/transports/http-preflight) in the AMPS User Guide. ## Example Client Connections ### JavaScript For install and usage details, see [Obtaining and Installing the AMPS Client](/clients/amps-client-javascript/installing). ```javascript showLineNumbers const { Client } = require('amps') // required when the CA certificates are self-signed process.env.NODE_TLS_REJECT_UNAUTHORIZED = 0; async function main() { // Connect and logon const client = new Client('javascript-proxy-client') // HTTP preflight is not required for standard websocket connections await client.connect('wss://localhost:443/wss/amps/json') console.log('connected!') // No difference in usage once the client is connected client.publish("test", "{\"hello\":\"world\"}") client.disconnect() } main() ``` This example creates an AMPS Client and establishes a secure WebSocket connection to AMPS through the NGINX reverse proxy. ### Python For install and usage details, see [Python Quickstart](/clients/amps-client-python/quickstart). ```python showLineNumbers import AMPS client = AMPS.Client('python-proxy-client') # Connect and logon using HTTP preflight client.connect('tcps://localhost:443/tcps/amps/json?http_preflight=true') client.logon() print('connected!') # No difference in usage once the client is connected client.publish("test", "{\"hello\":\"world\"}") client.disconnect() client.close() ``` This example creates an AMPS Client and uses HTTP Preflight to establish an upgraded TCPS connection to AMPS through the NGINX reverse proxy. ### C++ For install and usage details, see [Obtaining and Installing the AMPS Client](/clients/amps-client-cpp/installing). ```cpp showLineNumbers #include #include int main() { AMPS::Client client("cpp-proxy-client"); // Connect and logon using HTTP preflight client.connect("tcps://localhost:443/tcps/amps/json?http_preflight=true"); client.logon(); std::cout << "Connected!" << std::endl; // No difference in usage once the client is connected client.publish("test", "{\"hello\":\"world\"}"); client.disconnect(); return 0; } ``` This example creates an AMPS Client and uses HTTP Preflight to establish an upgraded TCPS connection to AMPS through the NGINX reverse proxy. ### Java For install and usage details, see [Obtaining and Installing the AMPS Client](/clients/amps-client-java/installing). :::note SSL connections in Java require a properly configured truststore so the client can verify and trust the server’s certificate during the TLS handshake. You can create a truststore by running the following: ```bash keytool -importcert -file /etc/pki/tls/certs/localhost.crt -keystore amps-truststore.p12 -storepass 123456 ``` ::: ```java showLineNumbers import com.crankuptheamps.client.Client; public class Example { public static void main(String[] args) { // required for providing the proper certs for ssl connections System.setProperty("javax.net.ssl.trustStore", "amps-truststore.p12"); System.setProperty("javax.net.ssl.trustStorePassword", "123456"); Client client = new Client("java-proxy-client"); try { // Connect and logon using HTTP preflight client.connect("tcps://localhost:443/tcps/amps/json?http_preflight=true"); client.logon(); System.out.println("Connected!"); // No difference in usage once the client is connected client.publish("test", "{\"hello\":\"world\"}"); } catch (Exception e) { System.err.println("Exception: " + e); } finally { client.close(); } } } ``` This example creates an AMPS Client and uses HTTP Preflight to establish an upgraded TCPS connection to AMPS through the NGINX reverse proxy. For additional SSL guidance, see [Providing SSL Certificates to the AMPS Java Client](/clients/amps-client-java/advanced-topics#providing-ssl-certificates-to-the-amps-java-client). ### C#/.NET For install and usage details, see [Obtaining and Installing the AMPS Client](/clients/amps-client-csharp/installing). ```cs showLineNumbers using System; using AMPS.Client; using AMPS.Client.Exceptions; try { // Connect and logon using HTTP preflight and the proxy endpoint Client client = new Client("csharp-proxy-client"); client.connect("tcps://localhost:443/tcps/amps/json?http_preflight=true"); client.logon(); Console.WriteLine("Connected!"); // No difference in usage once the client is connected client.publish("test", "{\"hello\":\"world\"}"); client.close(); } catch (AMPSException exception) { Console.WriteLine(exception); } ``` This example creates an AMPS Client and uses HTTP Preflight to establish an upgraded TCPS connection to AMPS through the NGINX reverse proxy. :::tip If the clients fail to connect after following the above steps on **Fedora/Red Hat**, try running the following command: ```bash sudo setsebool -P httpd_can_network_connect 1 ``` This command modifies a security setting in **SELinux** (Security-Enhanced Linux) to allow the **httpd** (web server) process to make network connections. It also ensures that this setting persists across system reboots. If you use the Fedora `localhost` certificate shown in this guide, connect to `localhost` rather than `127.0.0.1`, or replace the certificate with one that matches the hostname clients will use. ::: --- # Configuring AMPS for Use via a Proxy ## Update the AMPS Configuration First, we need to define the AMPS transports that the proxy will forward connection requests to. For `TLS/SSL` transports, set `Type` to `tcps` and add the `Certificate` and `PrivateKey` elements, like so: ```xml any-tcps tcps amps 9007 /etc/pki/tls/certs/localhost.crt /etc/pki/tls/private/localhost.key any-wss tcps websocket 9008 /etc/pki/tls/certs/localhost.crt /etc/pki/tls/private/localhost.key ``` For a standard non-SSL configuration, you can simply omit the `Certificate` and `PrivateKey` elements from the above example and use a `Type` of `tcp`: ```xml any-tcp tcp amps 9007 any-ws tcp websocket 9008 ``` Second, update the `Admin` configuration with the `SQLTransport` element to specify which websocket `Transport` Galvanometer should use to submit queries and subscriptions: ```xml 8085 any-wss /etc/pki/tls/certs/localhost.crt /etc/pki/tls/private/localhost.key ``` The `Certificate` and `PrivateKey` elements on `Admin` are only necessary when the Admin interface itself is served over TLS/SSL. For a non-SSL deployment, use the non-secure websocket transport name (`any-ws`) and omit the TLS configuration. Lastly, include the `ExternalInetAddr` element in the `Admin` configuration to specify the value that should be used for connections to the Admin interface. `ExternalInetAddr` is not intended to override the `InetAddr` parameter or change the network addresses that the Admin server uses, but instead to provide an externally visible address that will reach the `InetAddr`. For example, a proxy on address `proxy_host_address` that exposes an `/admin` endpoint for the Admin interface: ```xml proxy_host_address/admin ``` Additionally, you can include the `SQLTransportInetAddr` element in the `Admin` configuration to directly provide the URI that the Galvanometer will use for submitting queries and subscriptions to AMPS. For example, if the proxy exposes a secure websocket transport at `proxy_host_address/wss`, use: ```xml showLineNumbers proxy_host_address/wss ``` ## Start the AMPS Server ```bash .//ampServer config.xml ``` If you need help installing and starting AMPS, see [Getting Started With AMPS](/docs/intro-guide/getting_started). You can find additional information on [Using AMPS with a Proxy](/docs/amps-user-guide/operation/proxy) in the AMPS User Guide. --- # NGINX Proxy Configuration Guide An NGINX reverse proxy sits in front of AMPS and forwards client connections to the appropriate backend transports, simplifying network routing, and providing a single entry point for AMPS traffic. This guide covers SSL (`tcps`, `wss`) and non-SSL (`tcp`, `ws`) proxy setups, as well as the corresponding AMPS transport configuration. It also demonstrates how AMPS Clients that use `TCP/SSL` support connection through an HTTP proxy when the HTTP Preflight option is provided in the connection string. For an overview of AMPS and instructions for setting up your development environment, refer to the [Introduction to AMPS](/docs/intro-guide/intro) guide. For additional information on accessing AMPS through a proxy, see the [Using AMPS with a Proxy](/docs/amps-user-guide/operation/proxy) section. For more information on HTTP Preflight, see the [HTTP Preflight](/docs/amps-user-guide/transports/http-preflight) section in the AMPS User Guide. For an overview of NGINX reverse proxy features, see the [NGINX Reverse Proxy Admin Guide](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/). :::tip This guide assumes that you have a development environment for configuring a reverse proxy (including sudo privileges), and access to an AMPS server. ::: --- # Install and Configure NGINX ## Prerequisites Before proceeding with this guide, make sure that you have access to the following on your development machine: - Fedora/Red Hat Environment - Sudo Privileges ## Install NGINX The NGINX package can be installed via the `dnf` package manager: ```bash sudo dnf install nginx ``` Additionally, the NGINX service must be enabled and started: ```bash sudo systemctl enable --now nginx # enable: Configures NGINX to start automatically at system boot. # --now: Starts the NGINX service immediately ``` ## Configuring NGINX as a Reverse Proxy :::info This guide focuses on three kinds of proxy traffic: - Galvanometer over HTTP or HTTPS - TCP or TCPS connections using HTTP Preflight - WS or WSS connections using WebSocket upgrade For connections over SSL, use `tcps://` and `wss://` on the client side, `/tcps/` and `/wss/` on the NGINX side, and `https://` backend targets for the secure AMPS transports. For non-SSL connections, use `tcp://` and `ws://` on the client side, `/tcp/` and `/ws/` on the NGINX side, and `http://` backend targets. If you are unsure which connection type to use, see [TLS/SSL Transports](/docs/amps-user-guide/transports/configuring-transports#tlsssl-transports) in the AMPS User Guide. ::: Modify `/etc/nginx/nginx.conf` (`sudo` access is required). The first step in configuring NGINX is setting the number of worker processes. This determines how many processes will handle incoming requests. For small applications, one worker process is usually sufficient. For production environments, set this to match the number of CPU cores on your server. You can determine the number of CPU cores on your machine with `nproc`. Next, define how NGINX handles concurrent connections inside the `events` block. The `worker_connections` directive sets the maximum number of simultaneous connections a worker process can handle. If you expect high traffic, consider increasing this number, but ensure your server has enough resources to handle it. Next, add HTTP requests in the `http` block. - `include mime.types;` ensures that NGINX sets the correct Content-Type headers based on file extensions. - `default_type application/octet-stream;` is a fallback for unknown file types. - `sendfile on;` allows NGINX to send files efficiently by reading them directly from disk. - `keepalive_timeout 65;` keeps idle connections open for 65 seconds before closing them. To properly handle connection upgrades, such as through the use of HTTP Preflight, add a `map` directive inside the `http` block. This checks if the `Upgrade` header is set in an incoming request. If it is, NGINX will upgrade the connection; otherwise, it will close it. ```NGINX worker_processes 1; # Event Handling events { worker_connections 1024; } # HTTP Configuration http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; # WebSocket Upgrade Mapping map $http_upgrade $connection_upgrade { default upgrade; '' close; } } ``` Now, we define the main `server` block within the `http` block, which tells NGINX how to handle incoming requests. - `listen 443 ssl;` makes NGINX listen for all HTTPS requests on port `443`. For a non-SSL configuration, use `listen 80;` to listen for all HTTP requests. - `server_name localhost;` means the server will respond to requests sent to `localhost`. Replace `localhost` with your domain if hosting publicly. Next, we have to provide the necessary SSL certificates to NGINX by including `ssl_certificate` and `ssl_certificate_key`. For this demonstration, we will use the Fedora localhost certificates located at `/etc/pki/tls/certs/localhost.crt` and `/etc/pki/tls/private/localhost.key`. Inside the `server` block, add `location` blocks to define how different types of requests should be handled. :::note This guide assumes the AMPS server is running on the same host as the reverse proxy. If this is not the case, replace `127.0.0.1` with the address of your AMPS server. ::: First, add a `location` block that proxies `/admin/` to the backend service running on port `8085`. ```NGINX location /admin/ { proxy_pass https://127.0.0.1:8085/; } ``` Next, add a block for `/tcps/` to forward TCPS connections to port `9007` with an expected connection upgrade. ```NGINX location /tcps/ { proxy_pass https://127.0.0.1:9007/; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header Connection $connection_upgrade; proxy_set_header Upgrade $http_upgrade; } ``` Finally, add a block for `/wss/` that will forward Secure WebSocket traffic to port `9008` with an expected connection upgrade. ```NGINX location /wss/ { proxy_pass https://127.0.0.1:9008/; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header Connection $connection_upgrade; proxy_set_header Upgrade $http_upgrade; } ``` The Complete NGINX configuration for SSL connections: ```NGINX worker_processes 1; # Event Handling events { worker_connections 1024; } # HTTP Configuration http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; # WebSocket Upgrade Mapping map $http_upgrade $connection_upgrade { default upgrade; '' close; } # Server Block (Handles Incoming HTTPS Requests) server { listen 443 ssl; server_name localhost; ssl_certificate /etc/pki/tls/certs/localhost.crt; ssl_certificate_key /etc/pki/tls/private/localhost.key; location /admin/ { proxy_pass https://127.0.0.1:8085/; } location /tcps/ { proxy_pass https://127.0.0.1:9007/; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header Connection $connection_upgrade; proxy_set_header Upgrade $http_upgrade; } location /wss/ { proxy_pass https://127.0.0.1:9008/; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header Connection $connection_upgrade; proxy_set_header Upgrade $http_upgrade; } } } ``` For a standard non-SSL deployment, use the following `server` block: ```NGINX # Server Block (Handles Incoming HTTP Requests) server { listen 80; server_name localhost; location /admin/ { proxy_pass http://127.0.0.1:8085/; } location /tcp/ { proxy_pass http://127.0.0.1:9007/; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; } location /ws/ { proxy_pass http://127.0.0.1:9008/; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header Connection $connection_upgrade; proxy_set_header Upgrade $http_upgrade; } } ``` ## Validate and Apply the NGINX Configuration Validate the NGINX configuration file: ```bash sudo nginx -t ``` Restart the NGINX service to pick up the new changes: ```bash sudo systemctl restart nginx ``` Verify the NGINX service is running: ```bash sudo systemctl status nginx ``` The examples above show both SSL and non-SSL NGINX reverse proxy configurations for AMPS TCP, WebSocket, and Galvanometer connections. --- # AMPS Monitoring Guide This guide is a quick reference to the information monitored by the AMPS administrative interface. It includes information on the RESTful AMPS interface as well as the information recorded in the AMPS statistics database. For use with the AMPS administrative interface, the guide is intended to be a guide to help you understand the information presented in the interface. Likewise, when working with the statistics database, the guide is intended to provide a quick reference to the metrics in the database. The guide is organized in the same way that the administrative interface is: * [**Administrative Actions**](amps-monitoring-guide/admin) - This section describes the administrator resource in the administrative console. That resource provides the ability to change the state of the AMPS instance, and also generate diagnostic information to be written to the logs. There is no corresponding information in the statistics database. * [**Host Statistics**](amps-monitoring-guide/host-interface) - This section describes the `host` resource in the administrative console and the host tables in the statistics database (the tables prefixed with `H`). These metrics collect information about the host system. * [**AMPS Instance Statistics**](amps-monitoring-guide/instance-interface) - This section describes the `instance` resource in the administrative console and the instance tables in the statistics database (the tables prefixed with `I`). These metrics collect information about the AMPS instance that is collecting the statistics. ## Resource Mapping The guide is designed to make it easy to locate a particular resource. **RESTful Administrative Interface** To find the correct URI for a particular resource, you can start with the guide to find the path to the area, then use a browser to find the path to the specific resource you are looking for. For example, to find information about a specific drive, you might use the following steps: * Drive information is managed by the host, rather than a specific instance, so the information will be located in the host interface. * Within the host section of the guide, the information for drives is listed under the `disks` heading. Therefore, to start exploring information about drives attached to the host, you would use the `/amps/host/disks` path in a browser to locate a specific device and the information about this device. **Statistics Database** To find the correct table for a particular resource, you can start with the guide to find the path to the area, then use `sqlite3` or `amps-sqlite` to query the information for that resource. For example, to find information about a specific drive, you might use the following steps: * Drive information is managed by the host, rather than a specific instance, so the table that holds the information will have an `H` prefix. * Within the host section of the guide, the information for drives is listed under the `disks` heading. Therefore, to find information about drives, you would start with the `HDISKS` view (in `amps-sqlite`) or the `HDISKS_*` family of tables (in `sqlite3`). --- # Administrative Actions The `administrator` interface provides administrative actions for this AMPS instance. These actions are designed to be usable from a minimal web browser interface. Retrieving the URI for an action requests that AMPS take that action. In other words, clicking on the link for the action in a browser or performing a GET from a script will cause AMPS to take the appropriate action. * **Authorization**: Selecting the `authorization` resource will allow the `authentication` or `entitlement` resources to be reset. Selecting either one of these will present a `reset` link, which calls the reset function as defined by the respective authentication or entitlement resource. **Admin Path**: /amps/administrator/authorization/_\_/_\_ * **Clients**: Selecting the `clients` resource will list all connected clients by identifier. Selecting a single client will permit that client to be disconnected. **Admin Path**: /amps/administrator/clients/_\_/_\_ * **Diagnostics**: Selecting the `diagnostics` resource will provide the option to write a diagnostics `dump` message to the log. This message is logged at `info` level, and is event number `31-0010`. In this release of AMPS, the diagnostic message provides information on the current state of the queues configured for the instance. **Admin Path**: /amps/administrator/diagnostics/_\_ * **Minidump**: Selecting the `minidump` resource will create a minidump of the currently running AMPS instance. The minidump will be saved in a directory specified by the `MiniDumpDirectory`, or `/tmp` if no directory is specified. See [Instance-Level Configuration](/docs/amps-user-guide/configuring-amps/instance-configuration) for more information. **Admin Path**: /amps/administrator/minidump * **Queues**: Selecting the `queues` resource allows you to select whether this instance will respect the original ownership of messages in the queue, or allow the instance to claim ownership of messages if the original owner is unreachable. The options available from this interface are `enable_proxied_transfer` (meaning to allow this instance to claim ownership if a direct link to the owning instance is not connected) or `disable_proxied_transfer` (meaning that only the instance that owned the original message can transfer ownership). By default, proxied transfer is disabled. **Admin Path**: /amps/administrator/queues/_\_/_\_ * **Replication**: Selecting the `replication` resource will list all currently configured replications. Selecting any individual replication destination will permit the destination to be downgraded or upgraded. **Admin Path**: /amps/administrator/replication/_\_/_\_ * **SOW**: Selecting the `sow` resource will list all currently configured sow. topics. Selecting a topic will permit you to `compact` that topic. Compacting a SOW causes AMPS to release unused space in the SOW: this is normally done during startup. **Admin Path**: /amps/administrator/sow/_\_/_\_ * **Transaction Log**: Selecting the `transaction_log` resource will allow the `journals` resources to be compressed, archived, or removed. Only journal files that are full will be displayed. Selecting a journal will permit you to `compress`, `archive`, or `remove` that journal and all older journals. Compressing a journal will compress the selected journal and all older journals. Archiving a journal will move the selected journal and all older journals to the specified archive location. Removing a journal will delete the selected journal and all older journals. **Admin Path**: /amps/administrator/transaction\_log/journals/_\_/_\_ * **Transports**: Selecting the `transports` resource will list all currently configured transports. Selecting any individual transport will permit the transport to be enabled or disabled. **Admin Path**: /amps/administrator/transports/_\_/_\_ ### Entitlements Check for Admin Actions The administrative interface provides a way to check access to an administrative action without running the action. **Admin Path**: /amps/administrator/authorization/entitlement/transports/amps-admin/check This resource requires a `path` parameter with the full path of the action to check. AMPS will run the entitlement check, and then return a `true` or `false` result indicating whether the user has permission to run the action. For example, a full entitlement check on the ability to disconnect the client with object ID 1, requesting the result as a JSON document, would be: ```bash http://server:admin_port/amps/administrator/authorization/entitlement/transport/amps-admin/check.json?path=/amps/administrator/clients/1/disconnect ``` --- # Host Statistics The `host` URI contains information about the current operating system devices, such as the CPU, memory, disk and network. In addition, a host’s network hostname and system timestamp time are also exposed through the monitoring interface. --- # cpu (host statistics) The `cpu` resource allows an administrator to view the CPU devices attached to the host. Selection of the `cpu` link in the `host` resource generates a list of all CPUs attached to the host: you can produce data for each individual CPU, or use the aggregate `all` option to produce information for all CPUs. Notice that AMPS also records CPU statistics for the AMPS process itself. | Element | Description | Type | | ----------------- | ------------------------------------------------------------------------------------------------------------ | --------- | | `idle_percent` | Percent of CPU time that the system was waiting for an operation _other than_ an I/O request to complete. | snapshot | | `iowait_percent` | Percent of CPU time spent waiting for I/O requests to complete. | snapshot | | `system_percent` | Percent of CPU utilization time which occurred while executing kernel processes. | snapshot | | `user_percent` | Percent of CPU utilization time which occurred while running at the application level. | snapshot | **Statistics Database Tables**: `HCPU_STATIC`, `HCPU_DYNAMIC` **Admin Path**: /amps/host/cpu/_\_/_\_ --- # disks (host statistics) The `disks` resource lists each of the disk devices attached to the host and permits the inspection of disk usage statistics. This information is a readily consumable version of the file `/proc/diskstats`. | Element | Description | Type | | --------------------------- | ----------------------------------------------------------------- | ----------------- | | `file_system_free_percent` | Percentage of the filesystem currently free. | snapshot | | `mount_point` | The mount point for this filesystem. | fixed | | `in_progress` | Number of I/O requests waiting to be processed. | snapshot | | `read_await` | Average read time completion in milliseconds. | interval average | | `write_await` | Average write time completion in milliseconds. | interval average | | `read_bytes_per_sec` | Average bytes per second read. | interval average | | `write_bytes_per_sec` | Average bytes per second written. | interval average | **Statistics Database Tables**: `HDISKS_STATIC`, `HDISKS_DYNAMIC` **Admin Path**: /amps/host/disks/_\_/_\_ --- # memory (host statistics) The `memory` resource gives details about the system memory statistics. All statistics reported are based on the current system statistics reported by examining the file `/proc/meminfo`. | Element | Description | Type | | ------------ | ------------------------------------------------------------------------------------------------------------- | -------- | | `anonymous` | The total amount of anonymous memory allocated. | snapshot | | `available` |

    The total amount of memory available.

    This is the `MemAvailable` reported by the operating system, or the sum of free, buffers and cached if `MemAvailable` is not provided by the operating system.

    | snapshot | | `buffers` | The amount of physical memory available for file buffers. | snapshot | | `cached` | The amount of physical memory used as cache memory. | snapshot | | `free` | The amount of physical memory left unused by the system. | snapshot | | `in_use` | The amount of memory currently in use. | snapshot | | `swap_free` | The amount of swap memory which is unused. | snapshot | | `swap_total` | The total amount of physical swap memory. | fixed | | `total` | Total amount of RAM. | fixed | **Statistics Database Tables**: `HMEMORY_STATIC`, `HMEMORY_DYNAMIC` **Admin Path**: /amps/host/memory/_\_ --- # name (host statistics) The `name` resource displays the network DNS name for the host. **Admin Path**: /amps/host/name --- # network (host statistics) The `network` resource allows an administrator to examine networking interface statistics on the host. Selecting the `network` resource displays a list of the network interfaces attached to the host. Selecting one of the interfaces will list the available properties. | Element | Description | Type | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------- | | `bytes_in` | Number of bytes received by the interface. | interval snapshot | | `bytes_in_per_second` | Rate of bytes received by the interface. | interval average | | `bytes_out` | Number of bytes transmitted by the interface. | interval snapshot | | `bytes_out_per_second` | Rate of bytes sent by the interface. | interval average | | `errors` |

    Total errors both incoming and outgoing.

    This number includes packets dropped, collisions, fifo, frame, and carrier errors.

    | interval snapshot | | `packets_in` | The total number of packets received by the interface. | interval snapshot | | `packets_out` | The total number of packets sent by the interface. | interval snapshot | **Statistics Database Tables**: `HNET_STATIC`, `HNET_DYNAMIC` **Admin Path**: /amps/host/network/_\_/_\_ --- # AMPS Instance Statistics The `Instance` resource provided by the AMPS monitoring interface is the administrative overview of a running an AMPS instance. At a glance an administrator has access to a wide view of statistic and configuration information related to AMPS usage. :::info All statistics in the instance interface are server-side metrics. There is no metrics reporting from applications back to the instance: any application-side monitoring should be handled on the application side. ::: --- # api (instance statistics) Selecting the `api` resource lists information about the AMPS internal API. | Metric | Description | Type | | --------------------- | ------------------------------- | -------- | | `command_queue_depth` | The number of pending commands. | snapshot | **Statistics Database Tables**: `IGLOBALS_STATIC`, `IGLOBALS_DYNAMIC` **Admin Path**: /amps/instance/api/_\_ --- # clients (instance statistics) Selecting the `clients` resource will list all connected clients by name. Selecting a single client will show various statistics for that client. | Metric | Description | Type | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | `authenticated_id` | The ID used to authenticate this client, if any. | fixed | | `bytes_in` | Number of bytes received. | cumulative | | `bytes_in_per_sec` | Rate of bytes received. | interval average | | `bytes_out` | Number of bytes sent. | cumulative | | `bytes_out_per_sec` | Rate of bytes sent. | interval average | | `client_name` | Identifier for the client, set during logon. | fixed | | `client_name_hash` | AMPS hash for the client name. | fixed | | `client_version` | Version string provided by the client. | fixed | | `connect_time` | UTC time client connection is established. | fixed | | `connection_name` | Name of the connection. | fixed | | `correlation_id` | The CorrelationId provided with the logon command, if any. | fixed | | `denied_reads` | Number of read requests which have been denied due to an entitlement filter. | cumulative | | `denied_writes` | Number of write requests which have been denied due to an entitlement filter. | cumulative | | `messages_in` | Number of messages received from client. | cumulative | | `messages_in_per_sec` | Rate of messages received. | interval average | | `messages_out` | Number of messages sent to the client. | cumulative | | `messages_out_per_sec` | Rate of messages sent to the client. | interval average | | `query_time` | The amount of time spent for queries from this client. | cumulative | | `queue_depth_out` |

    Number of messages queued to be sent to client.

    This represents a count of the messages that AMPS cannot write to the outgoing socket due to the transmit buffer being full.

    This does not count messages already written to the transmit buffer. (The transport_tx_queue has information about the transmit buffer.)

    | snapshot | | `queue_max_latency` |

    The age of the oldest item in the queue which has not yet been sent.

    This is used as a measure of how far behind AMPS believes a subscribing client is.

    This measures the age of the oldest message that AMPS cannot write to the outgoing socket due to the transmit buffer being full.

    This does not count messages already written to the transmit buffer. (The transport_tx_queue has information about the transmit buffer.)

    The latency is measured in seconds at the resolution of the system clock.

    | snapshot | | `queued_bytes_out` |

    Number of queued bytes waiting to be sent.

    This represents a count of the number of bytes that AMPS cannot write to the outgoing socket due to the transmit buffer being full.

    This does not count messages already written to the transmit buffer (transport_tx_queue will show messages written to the transmit buffer that have not yet been sent).

    | snapshot | | `remote_address` | Address and port of the remote side of the client connection. | fixed | | `subscription_count` | Number of subscriptions currently active for the client. | snapshot | | `tcp_zero_window_advert` |

    Shows whether this client is currently advertising a zero window size.

    This is 1 if the client is advertising a windows size of zero, set to 0 if the client is advertising any other value or if the client connection does not use TCP (for example, the client uses UDS to connect to AMPS).

    | snapshot | | `transport_rx_queue` |

    Number of bytes in the transport receive buffer (typically the TCP buffer) for this client.

    This measures messages arriving from the client.

    | snapshot | | `transport_tx_queue` |

    Number of bytes in transport transmit buffer (typically the TCP buffer) for this client.

    This measures messages being sent to the client.

    | snapshot | **Statistics Database Tables**: `ICLIENTS_STATIC`, `ICLIENTS_DYNAMIC` **Admin Path**: /amps/instance/clients/_\_/_\_ --- # config.xml (instance statistics) Selecting this will display the current AMPS configuration file. To keep the path consistent, AMPS provides the file under the `config.xml` path, regardless of the actual name of the file. **Admin Path**: /amps/instance/config.xml --- # config_path (instance statistics) Filesystem location of the configuration file. **Admin Path**: /amps/instance/config\_path --- # conflated_topics (instance statistics) Selecting the `conflated_topics` resource will display a list of the conflated topics in the instance. | Metric | Description | Type | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | | `conflation_ratio` | Ratio representing the amount of conflation for this topic. (If there are no updates, this is not calculated and shows as 0.0.) | running average | | `interval` | Conflation interval. | fixed | | `message_type` | Message type for this topic. | fixed | | `topic` | Name of the conflated topic. | fixed | | `total_executions` | Total number of times the conflation algorithm has been executed. | cumulative | | `total_time` | Amount of time spent processing this topic. | cumulative | | `underlying_topic` | Name of the underlying topic that this topic conflates. | fixed | | `filter` | The optionally configured filter for this topic. | fixed | | `pattern` | The generated pattern for this topic (defined when the underlying topic is a regex topic). | fixed | | `topic_format` | The topic format for this topic (defined when the underlying topic is a regex topic). | fixed | **Statistics Database Tables**: `ICONFLATEDTOPICS_STATIC`, `ICONFLATEDTOPICS_DYNAMIC` **Additional topic information**: `ISOW_STATIC`, `ISOW_DYNAMIC` **Admin Path**: /amps/instance/conflated\_topics/_\_/_\_ --- # cpu (instance statistics) The CPU resource lists properties related to overall CPU usage of the AMPS instance. Selecting the items below give more specific information to the type of CPU utilization being consumed by the AMPS process. Notice that AMPS also collects metrics for the overall host system, available in the host resources. | Metric | Description | Type | | ---------------- | ---------------------------------------------------------------------------------------------------------- | -------- | | `system_percent` | Percent of CPU utilization time consumed while executing kernel processes on behalf of this AMPS instance. | snapshot | | `total_percent` | Total percent of CPU utilization on behalf of this AMPS instance. | snapshot | | `user_percent` | Percent of CPU utilization time consumed while processing non-I/O events on behalf of this AMPS instance. | snapshot | **Statistics Database Tables**: `ICPUS_STATIC`, `ICPUS_DYNAMIC` **Admin Path**: /amps/instance/cpu/_\_ --- # cwd (instance statistics) The current working directory from which the AMPS instance was invoked. **Admin Path**: /amps/instance/cwd --- # description (instance statistics) The contents of the `Description` element in the configuration file. **Admin Path**: /amps/instance/description --- # environment (instance statistics) The contents of the `Environment` element in the configuration file. **Admin Path**: /amps/instance/environment --- # lifetimes (instance statistics) Information about the lifetime of the AMPS instance, including historical information if `stats.db` is persisted. Each time an event related to startup or shutdown is logged, AMPS creates an entry in this resource. Each entry contains the following statistics: | Element | Description | Type | | ----------- | --------------------------------------------------------------- | -------- | | `event` | The type of event logged. For example, `started` or `shutdown`. | fixed | | `timestamp` | The timestamp of the event. | fixed | | `version` | The AMPS version string for the instance that logged the event. | fixed | **Statistics Database Tables**: `ILIFETIMES_STATIC`, `ILIFETIMES_DYNAMIC` **Admin Path**: /amps/instance/lifetimes/_\_/_\_ --- # logging (instance statistics) The `logging` resource contains information about the resources consumed during various AMPS logging processes. Selecting a logging mechanism (console, file or syslog) will first list all logs of that particular type. Drilling down into one of those logs will pull up more granular information about logging. If a logging mechanism is not defined in the configuration, then the results will be blank when the logging resource is selected. **Statistics Database Tables**: `ICONSOLE_LOGGERS_STATIC`, `ICONSOLE_LOGGERS_DYNAMIC`, `IFILE_LOGGERS_STATIC`, `IFILE_LOGGERS_DYNAMIC`, `ISYSLOG_LOGGERS_STATIC`, `ISYSLOG_LOGGERS_DYNAMIC` **Admin Path**: /amps/instance/logging/_\_/_\_/_\_ ## console Below are the options available for reporting when `console` logging is enabled: | Metric | Description | Type | | ---------------- | ---------------------------------------------------------------------------------------------- | ---------- | | `bytes_written` | Number of bytes written to the console. | cumulative | | `exclude_errors` | Errors which are excluded from logging. | fixed | | `include_errors` | Errors which are included during logging. | fixed | | `log_levels` | Log level used to control logging output. | fixed | | `target` |

    Console to which logging output is directed.

    Default: `stdout`

    | fixed | ## file Below are the options available for reporting when `file` logging is enabled: | Metric | Description | Type | | -------------------------- | ------------------------------------------------------------------------ | ---------- | | `bytes_written` | Number of bytes written to the log. | cumulative | | `exclude_errors` | Errors which are excluded from logging. | fixed | | `file_name` | File defined in the configuration file where the log file is written to. | fixed | | `file_name_mask` | Mask of the logging output file name, if available. | fixed | | `file_system_free_percent` | Percentage of free space on the file system that contains the file. | snapshot | | `include_errors` | Errors which are included during logging. | fixed | | `log_levels` | Log level used to control logging output. | fixed | | `rotation` | Boolean representation denoting if log rotation is turned on. | fixed | | `rotation_threshold` | Log size at which log rotation will occur. | fixed | ## syslog Below are the options available for reporting when `syslog` logging is enabled: | Metric | Description | Type | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | | `bytes_written` | Number of bytes written to syslog. | cumulative | | `exclude_errors` | Errors which are excluded from logging. | fixed | | `facility` | Integer enumeration of the logging facility used by syslog. | fixed | | `ident` | Syslog name of the logging instance. | fixed | | `include_errors` | Errors which are included during logging. | fixed | | `log_levels` | Log level used to control logging output. | fixed | | `logopt` |

    Bitfield of possible log options included.

    These values are configured in the configuration file in the `` tag.

    | fixed | --- # memory (instance statistics) AMPS can provide information regarding the process’s memory usage in its RSS and VMSize via the `memory` resource in the monitoring interface. | Metric | Description | Type | | ------------- | -------------------------------------------- | ----------------- | | `caches` | Information about AMPS memory caches. | (see caches) | | `paginations` | Information about paginated result sets. | (see paginations) | | `rss` | The resident set size of the AMPS process. | snapshot | | `vmsize` | The virtual memory size of the AMPS process. | snapshot | **Statistics Database Tables**: `IMEMORY_STATIC`, `IMEMORY_DYNAMIC`, `IMEMORY_CACHES_STATIC`, `IMEMORY_CACHES_DYNAMIC` **Admin Path**: /amps/instance/memory/_\_, /amps/instance/memory/caches/_\/\*\_, /amps/instance/memory/paginations/_\_ The `caches` element provides information about currently-active memory caches. | Metric | Description | Type | | ---------------- | ------------------------------------------------- | ---------- | | `allocations` | Number of memory allocations for this cache. | cumulative | | `bytes` | Number of bytes allocated to this cache. | snapshot | | `description` | Description of the cache. | fixed | | `efficiency` | Ratio of hits to requests for this cache. | snapshot | | `entries` | Number of entries in this cache. | snapshot | | `evictions` | Count of evictions from this cache. | cumulative | | `fetches` | Count of fetches from this cache. | cumulative | | `overflow_bytes` | Bytes allocated for slow consumers to this cache. | cumulative | The following caches may appear in AMPS statistics: | Cache Name | Description | | ------------------------------- | ---------------------------------------------- | | `byte buffer cache` | General purpose cache. | | `byte in buffer cache` | Cache for bytes being received from clients. | | `byte out buffer cache` | Cache for bytes being sent to clients. | | `client cache` | Cache for client objects. | | `client entitlement cache` | Cache for maintaining entitlement information. | | `client session cache` | Cache for maintaining current session state. | | `client status cache` | Cache for forming client status messages. | | `message cache` | Cache for message objects. | | `query context cache` | Cache for query contexts. | | `sow update cache` | Cache for SOW updates. | | `subscription cache` | Cache for subscription state. | | `xpath value data buffer cache` | Cache for XPath values. | **Statistics Database Table**: `IMEMORY_CACHES` The `paginations` element provides information about currently-active paginated subscriptions in AMPS. | Metric | Description | Type | | --------------------- | -------------------------------------------------------------- | ---------- | | `memory_bytes` | Number of bytes consumed to maintain this paginated set. | snapshot | | `message_type` | Message type for this paginated set. | fixed | | `subscription_count` | Number of subscriptions using this paginated set. | snapshot | | `topic` | Source topic for this paginated set. | fixed | **Statistics Database Table**: `IPAGINATIONS` --- # message_types (instance statistics) Information regarding the message types used by AMPS are maintained in the `message_types` resource. AMPS can track the following information for all message types loaded into the instance. | Metric | Description | Type | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `module` | The name of the module that implements the message type. | fixed | | `name` | The name of the message type. | fixed | | `options` | Any options provided to the module. | fixed | | `type` |

    The type configured for the message type module.

    This configuration parameter is obsolete in 4.0 and later releases.

    | (obsolete) | **Admin Path**: /amps/instance/message\_type/_\_/_\_ --- # name (instance statistics) Name of the AMPS Instance. **Admin Path**: /amps/instance/name --- # name_hash (instance statistics) The hashed value of the AMPS Instance name. **Admin Path**: /amps/instance/name\_hash --- # pid (instance statistics) The process ID of the current `ampServer` process. **Admin Path**: /amps/instance/pid --- # processors (instance statistics) Selecting the `processors` resource will list all the available message processors that the AMPS instance has invoked to handle messages. Each AMPS message processor will be listed individually, or selecting the `all` resource will list an aggregate of the available message processors. All AMPS message processors have the following attributes available: | Metric | Description | Type | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | `denied_reads` | Number of read requests which have been denied due to an entitlement filter. | cumulative | | `denied_writes` | Number of write requests which have been denied. | cumulative | | `description` | Descriptor of the processor. | fixed | | `last_active` |

    Number of milliseconds since a processor was last active.

    For each statistics snapshot, this indicates the longest period of time between the time that the statistics were collected and the time an instance of a processor of this type marked itself as active.

    This counter is expected to have variation in a healthy instance. A steady increase in this counter over a number of samples could indicate that the processor is not able to become active (for example, due to CPU saturation).

    | snapshot | | `matches_found` | Number of messages found. | cumulative | | `matches_found_per_sec` | Rate of messages found. | interval average | | `matches_found_bytes` | Number of bytes matched. | cumulative | | `matches_found_bytes_per_sec` | Rate of bytes matched for this processor. | interval average | | `messages_received` | Number of messages received. | cumulative | | `messages_received_per_sec` | Rate of messages received. | interval average | | `messages_received_bytes` | Number of bytes received. | cumulative | | `messages_received_bytes_per_sec` | Rate of bytes received for this processor. | interval average | | `throttle_count` |

    Number of times the processor had to wait to add a message to the processing pipeline due to the instance reaching capacity limits on the number of in-progress messages.

    This metric can indicate resource constraints on AMPS.

    | cumulative | AMPS also includes information for the following _processing types_, presented as an entry for a message processor with the given name: | Processing Type | Description | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `bookmark` | Messages from transaction log replays (bookmark subscriptions). | | `detached` | Messages related to subscriptions that hold messages before delivering them (such as conflated subscriptions, aggregated subscriptions, and subscriptions that use pagination). | | `external` | Messages to and from regular publish/subscribe subscriptions (that is, not SOW queries, message queue subscriptions, or transaction log replays). | | `internal` | Messages internally generated by AMPS. | | `queue` | Messages to and from message queues. | | `replication` | Messages to and from replication destinations. | | `sow` | Messages from queries of a SOW topic. | **Statistics Database Tables**: `IPROCESSORS_STATIC`, `IPROCESSORS_DYNAMIC` **Admin Path**: /amps/instance/processors/_\_/_\_ --- # queries (instance statistics) The `queries` resource lists all available information regarding queries of SOW topics. ## queued\_queries A count of all queries which have not yet completed processing at the time the last statistics snapshot was recorded. **Admin Path**: /amps/instance/queries/_\_ **Statistics Database Tables**: `IGLOBALS_DYNAMIC`, `IGLOBALS_STATIC` --- # queues (instance statistics) The `queues` resource lists available information regarding the queues defined for this instance. | Metric | Description | Type | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `age_of_oldest_lease` | The age of the oldest current lease, in seconds. | snapshot | | `backlog` | The number of leased messages awaiting acknowledgment. | snapshot | | `deferred_ack_count` | The number of acknowledgments received for messages that have not yet become active in the queue. | snapshot | | `expired_leases` |

    The number of leases that have expired for this queue.

    This counter resets when the instance is restarted.

    | cumulative | | `inactive_message_count` | The number of messages for this queue that are present in the transaction log, but have not yet become active because they are beyond current active limit of the queue (as set by `TargetQueueDepth`, when that option is present). | snapshot | | `max_backlog` | The configured `MaxBacklog` for the queue. | fixed | | `target_queue_depth` | The configured `TargetQueueDepth` for the queue. | fixed | | `message_type` | The message type for the queue. | fixed | | `queue_depth` |

    Total number of unacknowledged messages currently active in the queue.

    For queues that do not set a `MaxQueueDepth`, this is the total set of messages in the queue.

    For queues that set a `MaxQueueDepth`, this represents only the messages that are within the specified depth.

    Messages that are present, but not yet active are shown in the `inactive_message_count`.

    | snapshot | | `seconds_behind` |

    Age of the oldest unacknowledged message in the queue.

    This counter resets when the instance is restarted.

    This statistic is measured in seconds, at the resolution of the system clock.

    | snapshot | | `owned` | Number of messages currently owned by this instance of the queue. | snapshot | | `proxied_transfer` | State of the proxied transfer setting. | fixed | | `topic` | Name of the queue topic. | fixed | | `transferred_in` | The number of messages originally published to another instance that have been transferred to this instance for delivery from this queue. | cumulative | | `transferred_out` | The number of messages originally published to this instance that have been transferred to another instance for delivery from the replicated instance of this queue. | cumulative | **Statistics Database Tables**: `IQUEUES_STATIC`, `IQUEUES_DYNAMIC` **Admin Path**: /amps/instance/queues/_\_/_\_ The `queues` resource also contains the following resource that produces information on the current live state of the queue. This information is produced directly from the internal state of AMPS, and is not recorded in the statistics database. | Metric | Description | Type | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `details` |

    Detailed information about the state of the queue.

    This information is produced in JSON format, and includes detailed internal metrics for the queue as well as the current depth of the queue and information about the individual messages at the head of the queue (up to the first 1000 messages).

    This information is produced on demand from the current state, and is not produced from the statistics database.

    | live state | The details element returns a document in JSON format that contains the following information: | Metric | Description | Type | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | | `internal_depth` |

    The current amount of space the queue has reserved for metadata entries.

    If the queue has grown and then messages have been acknowledged, this can be larger than the current number of messages in the queue.

    | live state | | `insert_count` |

    The number of messages inserted into the queue.

    This is the metric that will be recorded in the SOW metrics for the queue topic.

    | live state | | `delete_count` |

    The number of messages removed from the queue due to being acknowledged or having expired.

    This is the metric that will be recorded in the SOW metrics for the queue topic.

    | live state | | `depth` |

    Current number of unacknowledged messages in the queue.

    This is the metric that will be recorded in the SOW metrics for the queue topic.

    | live state | | `last_queued_txid` | The local transaction ID of the last transaction processed for the queue. | live state | | `last_acked_txid` | The local transaction ID of the last acknowledged point in the queue. | live state | | `priority_count` | The number of distinct priority values for the queue. (This will be 0 if the queue does not have a priority expression configured.) | live state | | `seconds_behind` | The point in the transaction log of the oldest message in the queue, in seconds, as measured by the time between the time the message was added to the local transaction log and the current time. | live state | | `age_of_oldest_lease` | The amount of time, in seconds, that the oldest current lease has been held by a client. | live state | | `backlog` | Number of messages currently leased from the queue. | live state | | `expired_leases` | Number of leases that have expired from this queue. | cumulative live state | | `locally_owned` | Number of messages owned by this instance. | live state | | `transferred_in` | Number of messages that have had ownership transferred to this instance. | cumulative live state | | `cursors` | Details of delivery cursors for this queue. | live state | | `messages` | Details for messages currently in this queue. | live state | The `cursors` element of the queue details contains the following information: | Metric | Description | Type | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `cursor_id` | The internal ID of the cursor. | live state | | `state` | Current state of the cursor. | live state | | `last_processed_txid` | Last local transaction ID processed by this cursor. | live state | | `last_delivered_txid` | Last local transaction ID delivered to a subscription from this cursor. | live state | | `last_result` | Last result recorded by the cursor when evaluating a message for delivery to a subscriber. | live state | | `processing_count` | Count of delivery evaluations by this cursor. | live state | | `cursor_subscriptions` | Details for the subscriptions serviced by this cursor, including the client name, filter in use by the client, current and maximum backlog for the subscription, and so on. | live state | The `messages` element of the queue details contains the following information: | Metric | Description | Type | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `txid` | The local transaction ID of the message. | live state | | `bookmark` | The bookmark of the message. | live state | | `journal` | The path to the journal file that contains the message. | live state | | `deliverable` |

    Flag indicating whether the message is currently deliverable (1 is true, 0 is false).

    A message may not be deliverable if this instance does not currently own the message, or if it is already leased to a subscriber.

    | live state | | `age` | The age of the message, as measured by the time between the time the message was added to the local transaction log and the current time. | live state | | `locally_owned` | Flag indicating whether the message is currently owned by this instance (1 is true, 0 is false). | live state | | `priority` | Priority value of this message. | live state | | `leased_to` |

    If the message is currently leased, the client name of the connection the message is leased to.

    This field is not present if the message is not currently leased.

    | live state | | `leased_age` |

    If the message is currently leased, the amount of time, in seconds, the message has been leased.

    This field is not present if the message is not currently leased.

    | live state | **Admin Path**: /amps/instance/queues/_\_/details --- # replication (instance statistics) Selecting the `replication` resource will display a list of available downstream replication instances used by this instance of AMPS. Selecting an individual replication instance will display the following statistics: | Metric | Description | Type | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | `authenticated_id` | The ID used to authenticate the connection to this instance of AMPS. | fixed | | `bytes_out` | Number of bytes sent to this destination. | cumulative | | `bytes_out_per_sec` | Rate of bytes sent. | interval average | | `client_name` | The client name used for this destination. | fixed | | `client_type` | Specifies whether client is a replication source or destination. | fixed | | `connect_time` | Time connected to this destination. | snapshot | | `destination_admin_addr` | The admin address of the destination. | fixed | | `destination_group_name` | The group name of the destination. | fixed | | `destination_name` | The name of the destination. | fixed | | `disconnect_count` | Number of times replication destination has been disconnected. | cumulative | | `disconnect_time` | Timestamp of the last time the replication destination disconnected. | snapshot | | `is_connected` | Boolean telling whether replication destination is currently connected. | snapshot | | `messages_out` | Number of messages sent to this destination. | cumulative | | `messages_out_per_sec` | Rate of messages sent to this destination. | interval average | | `name` | Name of replication configuration. | fixed | | `pass_through` | Boolean stating whether messages received via replication can be forwarded on this connection. | fixed | | `replication_type` | One of either `sync` or `async`. | snapshot | | `seconds_behind` |

    The current point in the transaction log that has been acknowledged by this destination.

    This is calculated as the difference in seconds between the time that the last message acknowledged by the destination was written to the transaction log and the time that the most recent transaction was processed.

    That is, if the last message that the destination has acknowledged was written to the local transaction log at `12:00:01.100` (one second and 100 ms after 12:00) and the current time is `12:00:03.212`, the seconds behind shown in the current statistics would be approximately `2.112`. Acknowledgments are transmitted at a specific interval (1s by default) from the destination instance to the source instance.

    AMPS rounds any value below `1` to `0`.

    | snapshot | **Statistics Database Tables**: `IREPLICATIONS_STATIC`, `IREPLICATIONS_DYNAMIC` **Admin Path**: /amps/instance/replication/_\_/_\_ The `replication` resource also provides options for managing replication instances. The following management functions are available: | Element | Description | | ----------- | ---------------------------------------------------------------------- | | `downgrade` | Change the replication type of this connection from `sync` to `async`. | | `reconnect` | Close and reopen the connection to the remote instance. | --- # sow (instance statistics) Clicking the `sow` link will list all available topics in the SOW for this AMPS instance. Selecting a single topic will list the following available statistics about the topic: | Element | Description | Type | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------- | | `delete_count` | Number of deletes processed by the SOW. | cumulative | | `deletes_per_sec` | Number of deletes per second processed by the SOW. | interval average | | `device` | Device the topic is stored on, if applicable. | fixed | | `historical_granularity` | The granularity at which the SOW maintains history for this topic (if set). | fixed | | `historical_window` | The window for which the SOW maintains history for this topic (if set). | fixed | | `insert_count` | Count of the number of new records inserted into this topic. | cumulative | | `inserts_per_sec` | Rate of inserts into this topic. | interval average | | `memory_bytes` |

    The number of bytes of memory used for this topic.

    This metric includes:

    • messages stored in the topic
    • metadata for the messages in the topic
    • indices for the topic (both memo indices and hash indices)

    Notice that, particularly when the topic is heavily indexed, this can be larger than the message size. Indices are not persisted, so this can also be larger than the space used for persisting the topic data.

    See the section on [Estimating AMPS Instance Memory Usage](../../amps-user-guide/operation/capacity-planning.md#estimating-amps-instance-memory-usage) for details.

    | snapshot | | `mmaps` | The number of memory maps used for this topic in the SOW. | snapshot | | `message_type` | Message type for this topic. | fixed | | `path` | File system location of the SOW topics file store. | fixed | | `queries_per_sec` | Rate of queries for this SOW topic. | interval average | | `query_count` | Number of queries processed for this topic. | cumulative | | `record_size` |

    Record size for the topic in the SOW.

    For SOW files created with current versions of AMPS, this will always return the same value.

    | fixed | | `resident_percent` |

    Percentage of the storage of this topic that is currently resident in memory.

    Messages that are not currently in memory will need to be retrieved from storage before they are delivered. When part of a topic is not resident, either the instance is under memory pressure _or_ the messages (if any) in that part of the topic have not been recently updated or delivered.

    | snapshot | | `slab_count` | The total number of slabs allocated for this SOW topic. | snapshot | | `slab_size` | The slab size for this SOW topic. | fixed | | `stored_bytes` | Number of bytes stored for this topic. | snapshot | | `topic` | Name of this SOW topic. | fixed | | `update_count` | Number of updates to existing records processed by this topic. | cumulative | | `updates_per_sec` | Number of updates to existing records per second. | interval average | | `valid_keys` |

    Number of distinct messages in the SOW - defined by the SOW topic key.

    For topics that maintain a history, this shows the total number of messages that the SOW maintains, which may be larger than the number of messages that would be returned by a query at the current time, or the number of messages that would be returned by a query at a historical point in time.

    | snapshot | **Statistics Database Tables**: `ISOW_STATIC`, `ISOW_DYNAMIC` **Admin Path**: /amps/instance/sow/_\_/_\_ Sample `amps-sqlite3` query: ```sql SELECT iso8601_local(timestamp), topic, deletes_per_sec, inserts_per_sec, updates_per_sec, valid_keys FROM ISOW GROUP BY topic ORDER BY topic, timestamp ``` This query shows the overall activity and number of records available for each SOW topic in the instance. Notice that the activity fields are averaged over the sample interval, while the number of valid keys is a snapshot at each sample. --- # statistics (instance statistics) The `statistics` resource contains information regarding how AMPS monitors its own statistics. | Element | Description | Type | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | | `disk_per_sample` | Amount of storage the stats database has grown since the last sample interval. | snapshot | | `file_name` |

    Location where statistics are stored.

    Default: `:memory:` which stores the statistics database in system memory.

    | fixed | | `file_size` | Size on disk of the statistics database. | snapshot | | `interval` | Time in milliseconds between statistics database updates. | fixed | | `memory_used` | Size in bytes of the system memory consumption of the statistics database. | snapshot | | `queries` | Number of queries processed from the statistics database. | cumulative | | `time_per_sample` | Time taken to process each statistics database query. | snapshot | | `total_commit_time` | Total amount of time spent committing statistics information to the database. | cumulative | | `total_samples` | Number of statistics database updates that have taken place since the AMPS server started. | cumulative | | `total_time` | Total amount of time spent publishing statistics, including the commit time, since the AMPS server started. | cumulative | **Statistics Database Tables**: `ISTATISTICS_STATIC`, `ISTATISTICS_DYNAMIC` **Admin Path**: /amps/instance/statistics/_\_ --- # subscriptions (instance statistics) Each client that submits a `subscribe` command message is tracked by AMPS, and their relevant metrics are captured in the monitoring instance database. Selecting the `subscriptions` resource lists the available subscribers. Selecting a subscriber will list the available statistics below: | Metric | Description | Type | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | | `backlog` |

    The current number of messages leased on this subscription.

    Applies to subscriptions to a queue.

    | snapshot | | `bookmark` | The bookmark the client provided when the subscription was entered, if any. | fixed | | `client_id` | The ID of the subscribing client. | fixed | | `cursor_id` | For bookmark subscriptions, the ID of the cursor replaying messages in the transaction log. | snapshot | | `entitlement_filter` |

    The filter applied to this subscription by the entitlement module, if any.

    (Since a regular expression subscription can have multiple entitlement filters, one for each matching topic, this is blank for subscriptions that use a regular expression for the topic name.)

    | fixed | | `filter` | The filter requested on the subscription, if any. | fixed | | `message_type` |

    Message type for the subscription message.

    Message type for a subscription is established when the client connects to a Transport. It will be the `MessageType` of the Transport or the message type supplied in the URI in the case the Transport doesn’t specify a `MessageType`.

    | fixed | | `options` | The options string for the subscription. | fixed | | `pagination_id` | For subscriptions that use pagination, the identifier of the paginated set for this subscription. | fixed | | `seconds_behind` |

    For bookmark subscriptions, the age of the last message enqueued for the client.

    This indicates the point in the transaction log at which replay is currently happening, and does not necessarily correspond to the rate at which the client is receiving messages or the amount of time required for the client to complete replay.

    | snapshot | | `sub_id` | The subscription ID for this subscription. | fixed | | `topic` | Subscription topic. | fixed | **Statistics Database Tables**: `ISUBSCRIPTIONS_STATIC`, `ISUBSCRIPTIONS_DYNAMIC` **Admin Path**: /amps/instance/subscriptions/_\_/_\_ --- # timestamp (instance statistics) The timestamps of the historical admin statistics intervals as recorded by AMPS. The interval between these timestamps is determined by the `Interval` configured in the [Admin Server and Statistics](/docs/amps-user-guide/monitoring/monitor-configuration) block in the configuration. These values can be used to ensure valid results are returned from a [Time Range Selection](/docs/amps-user-guide/monitoring/time-range-selection). All times used for the report generation and presentation are ISO- 8601 formatted. ISO-8601 formatting is of the following form: `YYYYMMDDThhmmss`, where `YYYY` is the year, `MM` is the month, `DD` is the year, `T` is a separator between the date and time, `hh` is the hours, `mm` is the minutes and `ss` is the seconds. Decimals are permitted after the `ss` units. All times used for the report generation and presentation are stored and returned in UTC time. **Admin Path**: /amps/instance/timestamp **Statistics Database Tables**: `IGLOBALS_DYNAMIC`, `IGLOBALS_STATIC` --- # transaction_log (instance statistics) Clicking the `transaction_log` link will display the statistics that AMPS gathers for the transaction log, if one is configured. | Metric | Description | Type | | --------------- | ----------------------------------------------------------------- | -------- | | `journals` | A list of all journal file names. | snapshot | | `max_timestamp` | The largest timestamp in the transaction log. | snapshot | | `min_timestamp` | The smallest timestamp in the transaction log. | snapshot | | `write_latency` | Statistics covering the latency of writes to the transaction log. | snapshot | | `write_size` | Statistics covering the size of writes to the transaction log. | snapshot | **Statistics Database Tables**: `ITRANSACTION_LOG_DYNAMIC`, `ITRANSACTION_LOG_STATIC`, `ITRANSACTION_WRITE_LATENCY_DYNAMIC`, `ITRANSACTION_LOG_WRITE_LATENCY_STATIC`, `ITRANSACTION_WRITE_SIZE_DYNAMIC`, `ITRANSACTION_LOG_WRITE_SIZE_STATIC` **Admin Path**: /amps/instance/transaction\_log/_\_ Selecting the `journals` resource will list all journal file names. Selecting a single journal will show the following details: | Metric | Description | Type | | --------------- | ----------------------------------------- | -------- | | `file_name` | The file name of the selected journal. | fixed | | `min_timestamp` | The smallest timestamp of the journal. | fixed | | `max_timestamp` | The largest timestamp of the journal. | fixed | | `is_archived` | Whether or not the journal is archived. | snapshot | | `is_compressed` | Whether or not the journal is compressed. | snapshot | **Admin Path**: /amps/instance/transaction\_log/journals/_\_/_\_ The `write_latency` and `write_size` metrics contain the following details: | Metric | Description | Type | | ----------- | ----------------------------------------------------------------------- | -------- | | `histogram` | An ASCII histogram of the monitored statistic. | snapshot | | `minimum` | The lowest observed sample of the statistic, in microseconds or bytes. | snapshot | | `maximum` | The largest observed sample of the statistic, in microseconds or bytes. | snapshot | --- # transports (instance statistics) Clicking the `transports` link will give a list of the transports defined in the configuration file for the AMPS instance. Clicking a view will display the detailed resources for views. | Metric | Description | Type | | -------------- | ------------------------------------------- | -------- | | `is_enabled` | Indicates whether the transport is enabled. | snapshot | | `message_type` | The message type for this transport. | fixed | | `name` | The name of this transport. | fixed | | `options` | The options provided for this transport. | fixed | | `type` | The type of transport. | fixed | **Statistics Database Tables**: `ITRANSPORTS_STATIC`, `ITRANSPORTS_DYNAMIC` **Admin Path**: /amps/instance/transports/_\_/_\_ --- # tuning (instance statistics) Clicking the `tuning` link will give a list of the tuning parameters for the instance. Clicking a parameter will give the current value for the instance. | Metric | Description | Type | | ---------- | ---------------------------------------------- | -------- | | `NUMA` | Indicates whether AMPS NUMA tuning is enabled. | fixed | **Admin Path**: /amps/instance/tuning/_\_ --- # uptime (instance statistics) The length of time that the AMPS instance has been running, which conforms to a `hh:mm:ss.uuuuuu` format. This format is explained in the table below: | Element | Description | Type | | ------------ | --------------- | -------- | | `hh` | Hours | snapshot | | `mm` | Minutes | snapshot | | `ss` | Seconds | snapshot | | `uuuuuu` | Microseconds | snapshot | **Admin Path**: /amps/instance/uptime --- # user_id (instance statistics) The username for the owner for the `ampServer` process. **Admin Path**: /amps/instance/user\_id --- # version (instance statistics) Version string of the current running instance of AMPS. **Admin Path**: /amps/instance/version --- # views (instance statistics) The `views` resource contains information about the views in the AMPS instance. Clicking a view will display the detailed statistics for views. AMPS also collects SOW statistics for views. These are available from the `sow` resource, with the name of the `view` as the topic name. | Element | Description | Type | | ------------------ | --------------------------------------------------------------------------------------------------------------- | -------- | | `conflation` | The inline conflation mode of the view. | fixed | | `conflation_ratio` | The ratio of incoming to conflated updates. (If there are no updates, this is not calculated and shows as 0.0.) | snapshot | | `grouping` | List of one or more fields, which are used to determine message aggregation. | fixed | | `message_type` | The message type of messages produced by this view. | fixed | | `projection` | The formula defined in the AMPS config for the computed transformation of one or more fields onto a new field. | fixed | | `queue_depth` | The number of updates to the view that are pending, but have not yet been applied. | snapshot | | `topic` | The name of the new AMPS topic created by this view. | fixed | | `underlying_topic` | The source topic used to compute the projected view. | fixed | **Admin Path**: /amps/instance/views/_\_/_\_ **Statistics Database Tables**: `IVIEWS_STATIC`, `IVIEWS_DYNAMIC` **Additional topic information**: `ISOW_STATIC`, `ISOW_DYNAMIC` Sample `amps-sqlite3` query: ```sql SELECT iso8601_local(v.timestamp), v.topic, v.queue_depth, v.conflation_ratio, s.updates_per_sec, s.inserts_per_sec, s.deletes_per_sec FROM IVIEWS AS v JOIN ISOW AS s ON v.oid = s.oid AND v.timestamp=s.timestamp WHERE (v.queue_depth + s.updates_per_sec + s.inserts_per_sec + s.deletes_per_sec) > 0 ORDER BY v.oid, v.timestamp asc ``` This query shows information about changes to the views in the instance. The query joins information from the ISOW record for the topics using matching `oid` and `timestamp` fields to show the inserts, updates, and deletes to the View (per second) for each sample. The query includes the pending updates (queue depth) at each sample. The `WHERE` clause only includes samples where there is activity to the view, to avoid showing samples where the contents of the queue are not changing. --- # Statistics Types AMPS collects statistics in several different ways. For each individual statistic, understanding how the number is collected and calculated -- that is, which _type_ of statistic you are working with -- is important for accurately understanding the statistic. The table below lists the statistics types in AMPS: | Type | Description | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Cumulative | Statistics are cumulative since the instance was started (for example, the number of bytes AMPS has sent over a given network interface since the instance started). | | Fixed | Information that is fixed for the lifetime of the instance (for example, the process ID of the AMPS server). | | Snapshot | The value of a metric at a specific point in time (for example, the number of active subscriptions for a client at a particular moment). | | Interval Average | Average computed for this statistics interval (for example, the number of bytes sent per second on a given network interface for the last statistics interval). | | Running Average | Average computed since the instance was started (for example, the conflation ratio for a conflated topic). | --- # Table Reference This section lists the statistics database tables that are related to each performance metric. The table below lists the base name for the tables that contain each type of metric. In the AMPS statistics database, static properties (such as the client name of a connection) are stored in a `STATIC` table, while statistics that are captured at each interval are stored in a `DYNAMIC` table. The `amps-sqlite3` script automatically handles the join from `STATIC` tables to `DYNAMIC` tables to allow you to query using the base table name directly. For queries that use other tools, include a join between the corresponding `STATIC` and `DYNAMIC` table `static_id` fields. For example, to query information for clients, join `ICLIENTS_STATIC` and `ICLIENTS_DYNAMIC` on `ICLIENTS_STATIC.static_id = ICLIENTS_DYNAMIC.static_id`. **Host Metrics** | Metric Category | Base Table Names | | ----------------------------------------------------- | -------------------------------------- | | [cpu](host-interface/cpu) (host level) | `HCPUS` (instance info in `ICPUS`) | | [disk capacity and activity](host-interface/disks) | `HDISKS` | | [memory](host-interface/memory) (host level) | `HMEMORY` (instance info in `IMEMORY`) | | [network activity](host-interface/network) | `HNET` | **Instance Metrics** | Metric Category | Base Table Names | | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | [api](instance-interface/api) (embedded client) | `IGLOBALS` | | [clients](instance-interface/clients) | `ICLIENTS` | | [conflated topics](instance-interface/conflated\_topics) | `ICONFLATEDTOPICS` (info also in `ISOW`) | | [cpu](instance-interface/cpu) (instance) | `ICPUS` (host level info in `HCPU`) | | [logging](instance-interface/logging) | `ICONSOLE_LOGGERS`, `IFILE_LOGGERS`, `ISYSLOG_LOGGERS` | | [memory](instance-interface/memory) (instance) | `IMEMORY`, `IMEMORY_CACHES`, `IPAGINATIONS` | | [message processors](instance-interface/processors) | `IPROCESSORS` | | [queries](instance-interface/queries) | `IGLOBALS` | | [queues](instance-interface/queues) | `IQUEUES` (info also in `ISOW`) | | [replication](instance-interface/replication) | `IREPLICATIONS` | | [sow](instance-interface/sow) (including information on views, conflated topics, queues) | `ISOW` | | [statistics](instance-interface/statistics) | `ISTATISTICS` | | [subscriptions](instance-interface/subscriptions) | `ISUBSCRIPTIONS` | | [timestamp](instance-interface/timestamp) | `IGLOBALS` | | [transaction log](instance-interface/transaction\_log) | `ITRANSACTION_LOG`, `ITRANSACTION_WRITE_LATENCY`, `ITRANSACTION_WRITE_SIZE` | | [transports](instance-interface/transports) | `ITRANSPORTS` | | [views](instance-interface/views) | `IVIEWS` | --- # AMPS Developer Guides Welcome to developing applications with AMPS, the Advanced Message Processing System from 60East Technologies! These guides will help you learn how to develop applications using AMPS. ## Before You Start Before getting started with this guide, it is important to have a good understanding of the following topics: * *Developing Applications in your Language of Choice* - To be successful using this guide, and developing applications with AMPS, you will need to have a working knowledge of the language you are developing in. * *AMPS Concepts* - This guide focuses on using the AMPS client libraries and how those libraries work with the AMPS server. Before working through this guide, we recommend reading the [Introduction to AMPS](intro-guide/intro) guide. Detailed explanations of the AMPS server behavior are in the [AMPS Server Documentation](/). You will also need a system on which you can compile and run code, and a system where you can host the AMPS server. ## Setting Up a Development Instance You will need an installed and running AMPS server to use the product as well. You can write and compile programs that use AMPS without a running server, but you will get the most out of this guide by running the programs against a working server. Instructions for starting an instance of AMPS are available in the [Introduction to AMPS](intro-guide/intro) guide. :::tip The AMPS server runs on x64 Linux. The [Introduction to AMPS](intro-guide/intro) and [AMPS FAQ](/faq) contain information on how to run an AMPS server on a development system that does not run Linux. ::: --- # Performance Tips and Best Practices This chapter presents tips and techniques for writing high-performance applications with AMPS. This section presents principles and approaches that describe how to use the features of AMPS and the AMPS client libraries to achieve high performance and reliability. Specific techniques (for example, the details on how to write a message handler) are described in other parts of the AMPS documentation and referenced here. Other techniques require information specific to the application (for example, determining the minimum set of information required in a message), and are best done as part of your application design. All of the recommendations in this section are general guidelines. There are few, if any, universal rules for performance: at times, a design decision that is absolutely necessary to meet the requirements for an application might reduce performance somewhat. For example, your application might involve sending large binary data that cannot be incrementally updated. That application will use more bandwidth per message than an application that sends 100-byte messages with fields that can be incrementally updated. However, since the application depends on being able to deliver the binary payloads, this difference in bandwidth consumption is a part of the requirements for the application, not a design decision that can be optimized. ## Measure Performance and Set Goals The most important tools for creating high performance applications that use AMPS are clear goals and accurate measurement. Without accurate measurement, it's impossible to know whether a particular change has improved performance or not. Without clear goals, it's difficult to know whether a given result is sufficient, or whether you need to continue improving performance. 60East recommends that your measurements include baseline metrics for the part of your message processing that does not involve AMPS. As an example, imagine your task is to reduce the amount of time that elapses between when an order is sent and when the processed response is received from 100ms in total to 85ms in total. To achieve this reduction, you might first measure the processing that your application performs on the order. If that processing consumes 65ms, the most effective optimization may be to improve the order processing. On the other hand, if processing an order consumes 15ms, then optimizing message delivery or network utilization may be the most effective way to meet your goals. When measuring performance, simulate your production environment as closely as possible. For example, AMPS is highly parallelized, so sending a pattern of subscriptions and publishes from a single test client that would normally come from 20 clients will produce a very different performance profile. Likewise, AMPS can typically perform at rates that fill the available bandwidth. Performance measured on a 1GbE connection may be very different than performance measured over a 10GbE connection. Consider the characteristics of your data, and the number of messages you expect to store and process. A 1GB data set consisting of 1 million records will perform differently than a 1GB data set consisting of 10 million records, or a 1GB data set consisting of 100 records. When collecting information about performance, 60East recommends enabling persistence for the Statistics Database (`stats.db`), so you can easily collect historical data on both AMPS and the operating system. For example, a dip in performance correlated with high CPU and memory usage at the same time each day may be correlated with other activity on the system (such as cron jobs or close of business processing). In a situation like that, where the performance reduction is based on factors external to the AMPS application, the overall system metrics captured in `stats.db` can help you re-create the external state and understand the state of the system as a whole. AMPS collects the statistics in memory by default, and persisting that data into a database does not typically have a measurable effect on performance itself, but makes measuring and tuning performance much easier. For performance testing, 60East recommends using dedicated hardware for AMPS to eliminate the effects of other processes. If dedicated hardware is not available and other processes are consuming resources, 60East recommends disabling AMPS NUMA tuning to ensure that AMPS threads do not unnecessarily compete with other processes during performance tuning. ## Use HAClient and Heartbeating Where Appropriate Not every application that uses AMPS requires high availability and the ability to automatically fail over if connectivity is lost or an instance of AMPS is offline. For applications that do need automatic reconnection, 60East strongly recommends using the `HAClient` and setting heartbeating for the client to effectively detect disconnection. When using the `HAClient` and heartbeating, there are two important guidelines to follow: - Do not replace the disconnect handler on the `HAClient`. The disconnect handler is responsible for reconnection, resubscription, and so on. If you need to detect disconnection, use a connection state listener. - Set the interval for heartbeating to approximately one-half the time that the application can tolerate interruption in message flow. Notice that it's not possible for the `HAClient` to tell the difference between an interruption in message flow caused by a server going offline and interruptions caused by an increase in latency due to network saturation or so on, so the interval should be somewhat larger than the highest expected latency between AMPS and the application. Last, but not least, if the application uses asynchronous message handling, the interval should also be set to a value larger than the maximum amount of time expected for the message handler to process a single message. ## Simplify Message Format and Contents AMPS supports a wide range of message types, and is capable of filtering and processing large and complex messages. For many applications, the simplicity of being able to use messages that contain the full information is the most important consideration. For other applications, however, achieving the minimum possible latency and the maximum possible network utilization is important enough to warrant choosing a simplified message format. To simplify message contents, carefully consider the information that downstream processors require. If a downstream process will not use information in the message, there is no need to send the information. For example, consider an application that provides orders from a UI. In such an application, the object that represents the order often contains information relevant to the local state of the application that is not relevant to a downstream system. Rather than simply serializing the full object, your application may perform better if you serialize only the fields that a downstream system will take action on. To simplify message format, choose the simplest format that can convey the information that your application needs. The general principle is that the simpler the message format is, the more quickly AMPS and client libraries can parse messages of that type. Likewise, the more complicated the structure of each message is, the more work is required to parse the message. For the highest levels of performance, 60East recommends keeping the message structure simple and preferring message formats such as NVFIX, BFlat, or flattened JSON (structured as key/value pairs) as compared with more complicated formats such as XML or BSON. ## Measure Serialization and Deserialization When creating baseline performance numbers, measure serialization and deserialization performance independent of the AMPS server or client libraries. This can help you to: - Understand the baseline performance of creating and processing message data under ideal conditions (that is, where there is no application processing, networking, routing, etc. involved). - Easily compare the application-side performance of different message formats or different message layouts within a single format. When testing this performance, it is helpful to use data similar to the data that the application will actually process during a business day, at the volumes the application would typically process. This will help you understand the performance of serialization and deserialization for this specific application. For example, a library for working with a given message format might be less efficient when processing messages with a large number of string fields in a deeply-nested structure, but your application might exchange only numeric data in a relatively flat structure. Likewise, the library for a given format could be efficient for processing a small number of fields, but have lower performance for a message type with hundreds of fields. As with all performance testing, the more closely the test environment matches the actual data and volumes of a production environment, the more helpful those measurements will be for understanding system performance. ## Use Content Filtering Where Possible AMPS content filtering helps your application perform better by ensuring that your application only receives the messages that it needs. Wherever possible, we recommend using content filtering to precisely specify which messages your application needs. In particular, if at any point your application is receiving a message, parsing the message, and then determining whether to act on the message or not, 60East recommends using content filters to ensure that your application only receives messages that it needs to act on. ## Use Asynchronous Message Processing The synchronous message processing interface is straightforward, and presents a convenient interface for getting started with AMPS. However, the `MessageStream` used by the synchronous interface makes a full copy of each message and provides it from the background reader thread to the thread that consumes the message. This memory overhead and synchronization between the reader thread and consumer thread happens regardless of whether the application needs all of the header fields in the message or even processes the message. The `MessageStream` also does not take into account the speed at which your program is consuming messages, and will read messages into memory as fast as the network and processor allow. If your application cannot consume messages at wire speed, this can lead to increasing memory consumption as the application falls further behind the `MessageStream`. Most applications see improved performance by using a `MessageHandler`. With this approach, the `MessageHandler` does minimal work. If more extensive processing is needed, the `MessageHandler` dispatches the work to another thread: but it does this only when the work is necessary, and it only saves the part of the message needed to accomplish the work. ## Use Hash Indexes Where Possible for SOW Queries When querying a SOW, hash indexes on SOW topics are supported for exact matching on string data as described in the *AMPS User Guide*. A hash index can perform many times faster than a parallel query. If the query pattern for your application can take advantage of hash indexes, 60East recommends creating those hash indexes on your SOW topics. More recent versions of AMPS can use hash indexes for a wider variety of filters. When planning your queries, review the SOW queries section of the *AMPS User Guide* for the version you are using for guidelines on the optimizations available in that version. ## Use a Failed Write Handler and Exception Listener In many cases, particularly during the early stages of development, performance problems can point to defects in the application. Even after the application is tuned, monitoring for failure is important to keep applications running smoothly. 60East recommends always installing a failed write handler if your application is publishing messages. This will help you to quickly identify cases where AMPS is rejecting publishes due to entitlement failures, message type mismatches, or other similar problems. 60East recommends always installing an exception listener if your application is using asynchronous message processing. This will help you to identify and correct any problems with your message handler. An exception listener should typically log the message received and return. If recovery is needed, the listener should set a flag for another thread to process rather than attempting to recover on the thread that calls the exception listener. ## Reduce Bandwidth Requirements In many applications that use AMPS, network bandwidth is the single most important factor in overall performance. Your application can use bandwidth most efficiently by reducing message size. For example, rather than serializing an entire object, you might serialize only the fields that the remote process needs to act on, as mentioned above. Likewise, rather than sending one message that contains a collected set of information that processors will need to extract, consider sending a message in the units that processors will work with. This can reduce bandwidth to processors substantially. For example, rather than sending a single message with all of the activity for a single customer over a given period of time (such as a trading day), consider breaking out the record into the individual transactions for the customer. ### Tune Batch Size for SOW Queries As described in the section on [SOW Batch Size](../amps-user-guide/sow-queries/batching-query-results), tuning the batch size for SOW queries can improve overall performance by improving network utilization. In addition, because the AMPS header is only parsed once per batch, a larger batch size can dramatically improve processing performance for smaller messages. The AMPS clients default to a batch size of `10`. This provides generally good performance for most transactional messages (such as order records or inventory records). For large messages, particularly messages greater than a megabyte in size, a batch size of `1` may reduce memory pressure in the client and improve performance. With smaller messages (for example, message sizes of a few hundred bytes), 60East recommends measuring performance with larger batch sizes such as `50` or `100`. For large messages, reducing the batch size may improve overall performance by requiring less memory consumption on the AMPS server. ### Conflate Fast-Changing Information If your data source publishes information faster than your clients need to consume it, consider using a conflated topic. For example, in a system that presents a user interface and displays fast-moving data, it is common for the data to change at a rate faster than the user interface can format and render the data. In this case, a conflated topic can both reduce bandwidth and simplify processing in the user interface. ### Minimize Bandwidth for Updates If your application uses a SOW and processes frequent updates, consider using delta publish and delta subscribe to reduce the size of the messages transmitted. These features are designed to minimize bandwidth while still providing full-fidelity data streams. ### Conflate Queue Acknowledgments The AMPS clients include the ability to conflate acknowledgments back to AMPS as queue messages are processed. Using these features, with an appropriate `max_backlog`, can reduce the amount of network traffic required for acknowledgments. ### Use a Transaction Log When Monitoring Publish Failures When a topic is not covered by a transaction log, AMPS returns acknowledgment messages for every publish that requests one. This ensures that each message is acknowledged, even when AMPS has no persistent record of the messages in the topic. However, acknowledging each message requires more network traffic for each publish message. When a topic is covered by a transaction log, AMPS conflates persisted acknowledgments. Conflation is possible in this case because AMPS has a full record of the messages and does not have to store additional state to conflate the acknowledgments. With conflated acknowledgments, AMPS will send a success acknowledgment periodically that covers all messages up to that point. If a message fails, AMPS immediately sends the conflated success acknowledgment for all previous messages and the failure acknowledgment for the failed message. ### Combine Conflation and Deltas In many cases, using an approach that combines delta publishes to a SOW with delta subscriptions to a conflated topic can dramatically reduce bandwidth to the application with no loss of information. ## Limit Unnecessary Copies One of the most effective ways to increase performance is to limit the amount of data copied within your application. For example, if your message handler submits work to a set of processors that only use the `Data` and `Bookmark` from a `Message`, create a data structure that holds only those fields and copy that information into instances of that data structure rather than copying the entire `Message`. While this approach requires a few extra lines of code, the performance benefits can be substantial. When publishing messages to AMPS, avoid unnecessary copies of the data. For example, if you have the data in a byte array, use the `publish` methods that use a byte array rather than converting the data to a string unnecessarily. Likewise, if you have the data in the form of a string, avoid converting it to a byte array where possible. ## Manage Publish Stores When using a publish store, the Client holds messages until they are acknowledged as persisted by AMPS, as determined by the replication configuration for the AMPS instance. In the event that an instance with `sync` replication goes offline, the publish store for the Client will grow, since the messages are not being fully persisted. To avoid this problem, 60East recommends that an instance that uses `sync` replication always configure Actions to automatically downgrade the replication link if the remote instance goes offline for a period of time, and upgrade the link when the remote instance comes back online. Further, 60East recommends that, where possible, a publisher is provisioned with enough storage to hold its complete publish stream for the amount of time that a destination may be offline or unavailable without downgrading from `sync` replication to `async` replication. For example, if the server considers a downstream system to be unreachable if it has not acknowledged a replicated message in 60 seconds, and the server checks this threshold every 10 seconds, then a publisher should plan that, at any time, the publisher may need to retain approximately 70 seconds worth of published messages. This is calculated as the 60 seconds threshold that the server has established for a destination to run behind, plus the 10 second interval at which the server checks whether the destination is within the threshold. Also notice that, with a configuration like this, a downstream replication destination could run as much as 59 seconds behind indefinitely. A publisher should be provisioned to be able to run effectively in a "worst case" (or nearly "worst case") scenario for an extended period of time. See the *High Availability and Replication* chapter in the *AMPS User Guide* for more information on replication, sync and async acknowledgment modes, and the Actions used to manage replication. ## Use the Server Logs to Help Troubleshoot When troubleshooting problems with an application that uses AMPS, the server-side logs often provide the most helpful information. For example, `trace` level logging shows the data that is flowing through AMPS. Log messages at `info` level show events as incoming connections, commands from clients, and so on. When questions arise about how the server and application interact, the server logs often contain the information. 60East recommends that an AMPS instance used for development and testing log at `trace` level, and that a server used for production log at `info` level, with the ability to log at `trace` level when necessary for investigating any problems that may arise. When a command does not have the expected result, or an application reports an error, the fastest way to understand the problem is often to review the `trace` level logging for the instance. See the *AMPS User Guide* for details on configuring logging and common patterns for searching for information in AMPS logs. ## Work with 60East as Necessary 60East offers performance advice adapted for your specific usage through your support agreement. Once you've set your performance goals, worked through the general best practices and applied the practices that make sense for your application, 60East can help with detailed performance tuning, including recommendations that are specific to your use case and performance needs. --- # AMPS User Guide Welcome to the Advanced Message Processing System (AMPS) from 60East Technologies. AMPS is a feature-rich message processing system that delivers previously unattainable low-latency and high-throughput performance to users. AMPS provides both publish-and-subscribe messaging and high-performance message queuing. AMPS also provides current value caching / message database functionality, analysis and aggregation. --- # Bookmark Subscriptions and Completed Acknowledgments When AMPS is processing a bookmark subscription, a `completed` acknowledgment indicates that the subscription has completed replay from the transaction log. This means that the subscription has reached the point in the transaction log at which the subscribe command was received, and messages delivered on the subscription after the `completed` acknowledgment are from new publishes. --- # Bookmark Subscriptions and Persisted Acknowledgments A bookmark subscription typically also requests the `persisted` acknowledgment (the AMPS clients do this automatically). In this case, AMPS periodically returns an acknowledgment that includes the last bookmark written to the local transaction log _**and**_ acknowledged by all replication destinations that are currently using `sync` acknowledgment. This information helps with application recovery, particularly in cases where publishers are intermittent or low-volume, and the last message for a subscription is significantly older than the last message evaluated for delivery to the subscription. This also helps to manage failover in situations where multiple instances in a given replication fabric are accepting publishes by providing a stable failover point across instances. :::tip The bookmark provided in the persisted ack is the last persisted point, which does not need to be a message that matches the subscription. This provides the benefit of persisted acknowledgements even in cases where the subscription has a narrow filter, or subscribes to a low-velocity topic interleaved with higher-velocity topics. ::: --- # Acknowledgment Conflation and Publish Acknowledgments For some commands, AMPS will *conflate* acknowledgments and return acknowledgments for multiple commands at one time. When AMPS conflates acknowledgments, AMPS provides an identifier other than the command identifier that describes which commands the acknowledgment applies to. For example, when a transaction log is configured (with at least one topic recorded), AMPS conflates `persisted` acknowledgments in response to `publish` commands and `sow_delete` commands. These conflated acknowledgments contain the last client sequence number that the acknowledgment applies to rather than the command identifiers or sequence numbers for all messages being acknowledged. For example, if an application publishes messages with sequence numbers `1`, `2`, `3`, `4`, and `5`, and message `3` fails due to entitlement restrictions, AMPS will return an `ack` indicating success for message `2`, an `ack` indicating failure for message `3`, and an `ack` indicating success for message `5`. By default, AMPS produces conflated acknowledgments for a given connection approximately once a second. Starting with version 5.2.3.0, AMPS allows configuration of the conflation interval. To set a different conflation interval: - For an individual client, provide an `ack_conflation` option in the options string for the logon for that client. This sets the interval at which AMPS will provide acknowledgments to that client. - For replication acknowledgments, specify the `AckConflationInterval` in the replication `Destination`. This sets the interval at which the downstream AMPS server will provide acknowledgments to this AMPS server. Notice that when a publisher is publishing to a replicated topic, the `ack_conflation` interval sets the interval at which AMPS acknowledges messages that have already been acknowledged by synchronous replication destinations, while the `AckConflationInterval` specifies how often those downstream destinations produce an acknowledgment. In a situation where it is important to reduce the default latency of an acknowledgment, **both** the server option and the client side option typically need to be set. The AMPS client libraries use `persisted` acknowledgments to manage reliable publishing when a publish store is configured. The `persisted` acknowledgments allow the library to remove messages that have been safely persisted by AMPS (or that have produced an error). See the Developer Guides for details. To see more information about the different commands and their supported acknowledgment types, please refer to the *AMPS Command Reference*, provided with 4.0 and greater versions of the AMPS clients and available from the [60East documentation](/docs). --- # Receiving Acknowledgments Acknowledgments for a specific command or subscription are delivered as a part of the message stream for that command. In an AMPS client, this means that whatever method an application is using to receive responses from AMPS will include acknowledgment messages. Acknowledgments from AMPS have a command type of `ack`. This command type is reserved for responses from AMPS to a client. The `CommandId` of the acknowledgment message contains the `CommandId` provided by the client on the command that AMPS is responding to. The client uses this `CommandId` to route the acknowledgment message to the correct message processor in the application. Notice that when acknowledgments are _conflated_ -- that is, the acknowledgment refers to a number of publish commands -- AMPS does not include a `CommandId` in the acknowledgment message. Further details are provided in the section on [Acknowledgment Conflation](publish-acks). --- # Requesting Acknowledgments Acknowledgments from the AMPS server are always optional. The AMPS clients will request acknowledgments as necessary for processing and error reporting. In addition, an application can request acknowledgments as necessary to meet the requirements of an application. The _AMPS Command Reference_ contains information on the acknowledgment types available for each command and the meaning of those acknowledgments. :::info **Developer Tip**: The AMPS client libraries automatically request and automatically process the acknowledgments needed for the library itself to work as expected. An application can also request acknowledgments explicitly. When an application explicitly requests an acknowledgment that is _not_ conflated, the AMPS clients deliver that acknowledgment to the message handler provided on the command or message stream returned by the command. If the acknowledgment is conflated (see the section on [Acknowledgment Conflation and Publish Acknowledgments](publish-acks) following), the application must either provide a last chance message handler _or_ provide a global command type message handler to process acknowledgments. Because the acknowledgment will typically apply to more than one command, AMPS does not include the individual command identifiers, and the clients cannot route the message to the individual command handlers. ::: --- # Command Acknowledgment AMPS command processing is designed to be asynchronous. The design of the server makes it possible for an application to send a command to AMPS, and receive the results of that command at a later time. Acknowledgment of commands is always optional: the server makes no requirement that an application request acknowledgment. The AMPS client libraries automatically request the acknowledgments required to maintain the guarantees the client API provides. The status and results of a command are returned to a client in the form of an acknowledgment, or `ack`, message. AMPS can return status updates at various checkpoints throughout the command processing sequence. For many applications, it may not be necessary for the application to request message acknowledgments explicitly. The AMPS clients request a set of acknowledgments by default that balance performance with error detection. AMPS supports a variety of `ack` types, and allows you to request multiple `ack` types on each command. For example, the received `ack` type requests that AMPS acknowledge when the command is received, while the completed `ack` type requests that AMPS acknowledge when it has completed the command (or the portion of the command that runs immediately). Each AMPS command supports a different set of types, and the precise meaning of the `ack` returned depends on the command that AMPS is acknowledging. AMPS commands are inherently _asynchronous_, and AMPS does not provide acknowledgment messages by default. A client must both explicitly request an acknowledgment and then receive and process that acknowledgment to know the results of a command. It is normal for time to elapse between the request and the acknowledgment, therefore AMPS acknowledgments provide a way to correlate the acknowledgment with the command that produced it. This is typically done with an identifier that the client assigns to a command, which is then returned in the acknowledgment for the command. AMPS supports the acknowledgment types listed in the following table: | Acknowledgment Type | Description | | ----------------------- | ------------------------------------------------------------------ | | `completed` | The command (or a portion of the command) has completed. | | `persisted` | The results of the command have been persisted to durable storage. | | `processed` | AMPS has processed the command. | | `received` | AMPS has received the command. | | `stats` | AMPS returns statistics associated with the command. | Acknowledgments for different commands may not arrive in the order that commands were submitted to AMPS. For example, a `publish` command to a topic that uses synchronous replication will not return a `persisted` acknowledgment until the synchronous replication destinations have persisted the message. If the client issues a `subscribe` command in the meantime, the `processed` acknowledgment for the `subscribe` command -- indicating that AMPS has processed the subscription request -- may well return before the `persisted` acknowledgment. Not all commands support all acknowledgment types, and the meaning of each acknowledgment may differ depending on the command submitted. The acknowledgments for different commands set different fields on an acknowledgment message. If an acknowledgment type is not specified for a given command, AMPS does not make specific guarantees as to when (or if) that acknowledgment is returned for that command. See the [AMPS Command Reference](../amps-command-reference/) for details. --- # Archive Journals Once a Week The listing below asks AMPS to archive transaction log journal files older than 1 week, every Saturday at 12:30 AM. This configuration moves journal files from the `JournalDirectory` to the `JournalArchiveDirectory`, while maintaining the files as an active part of the transaction log (ensuring that they are still available for bookmark replay, replication, and so on). ```xml showLineNumbers amps-action-on-schedule Saturday at 00:30 Saturday Night Fever amps-action-do-archive-journal 7d ``` --- # Archive Journals On RESTful Command The listing below directs AMPS to archive journals when a query is made to the `/amps/administrator/actions/archive_journals` resource. The request must include an `AGE` query parameter. When the request is submitted, AMPS will mark journals older than the specified age for archival. ```xml showLineNumbers amps-action-on-admin archive_journals AGE Archive Journals amps-action-do-archive-journal {{AGE}} ``` The `On` configuration for this action specifies that the action will run when a request is made to the `/amps/administrator/actions/archive_journals` resource, and that the request must contain an `AGE` query parameter. When the request runs, the `AGE` parameter is added to the context for the action. The `Do` configuration for this action runs the `amps-action-do-archive-journal` module, and fills in the value of the `AGE` parameter provided by the HTTP request. If the `AGE` parameter is missing, the `amps-action-on-admin` module will refuse the request. If the `AGE` parameter is not a valid interval (for example, if someone provided the string `"LastThursday"` instead of a valid AMPS interval like `48h`), the `amps-action-do-archive-journal` module will log an error and refuse to run the request. --- # Record Expired Queue Messages to a Dead Letter Topic The listing below detects when a message expires from a queue, and publishes those messages to a dead letter topic. ```xml showLineNumbers amps-action-on-sow-expire-message interesting-queue json amps-action-on-sow-expire-message another-interesting-queue json amps-action-do-publish-message dead-letter json {"topic":"{{AMPS_TOPIC}}","message":{{AMPS_DATA}} } ``` --- # Extract Values from a Published Message The listing below extracts values from an XML message published to the local instance and stores them into the action context as `VALUE` and `QTY` for use in later action steps. ```xml showLineNumbers amps-action-on-publish-message message-sow xml local amps-action-do-extract-values xml {{AMPS_DATA}} VALUE = /info/value QTY = /info/quantity ``` --- # Increment a Counter and Echo a Message The listing below increments a counter and echoes the counter's value when AMPS receives the `USR1` signal. ```xml showLineNumbers amps-action-on-signal SIGUSR1 amps-action-do-increment-counter MY_COUNTER CURRENT_COUNTER_VALUE amps-action-do-echo-message AMPS has gotten {{CURRENT_COUNTER_VALUE}} SIGUSR1 signals. ``` --- # Examples of Action Configuration This section includes several examples of fully-configured actions, using the elements as outlined in the previous sections. --- # Copy Messages that Exceed a Timeout to a Different Topic The listing below, in effect, copies messages from the `Orders` topic to the `Orders_Stale` topic when the status has been `PENDING` for more than 5 seconds. This example handles timeouts for a message in a `SOW/Topic`, `SOW/View`, or `SOW/ConflatedTopic`. See the [Dead Letter Queue](dead-letter-queue.md) example for handling queue expiration. ```xml showLineNumbers amps-action-on-message-condition-timeout nvfix Orders /status = 'PENDING' 5s amps-action-do-publish-message nvfix Orders_Stale {{AMPS_DATA}} ``` --- # Copy Messages to a Different Topic The listing below copies messages from the `Orders` topic to the `DuplicateOrders` topic. Whenever a message is published to the `Orders` topic, this action will republish the message data to the `DuplicateOrders` topic. The limitations for the `amps-action-on-publish-message` apply to this action. This action will only receive messages when the instance is active. This means that the action is *not* active during recovery, and will *not* publish duplicate messages when the topic the action is monitoring is recovered. This also means that adding this action to a configuration will not replay messages that were published to AMPS before the action was added. :::warning The `amps-action-on-publish-message` action is treated by the AMPS engine as a subscription from an internal AMPS client. When working with queues, use this action with the *underlying* topic of the queue rather than the queue topic itself. Because this action creates a subscription, using this action with the queue topic will cause the action to lease messages from the queue even though the action does *not* acknowledge messages. This means that, when used with the queue topic itself, the action will interfer with other subscribers and depending on the queue configuration, may only receive one message during the lifetime of the instance. ::: ```xml showLineNumbers amps-action-on-publish-message nvfix Orders amps-action-do-publish-message nvfix DuplicateOrders {{AMPS_DATA}} ``` --- # Reset Entitlements for a Disconnected Client The listing below resets entitlement cache for a user when a client with that authentication ID is disconnected. This is a practical approach when an application expects that each user will have a single connection to AMPS. ```xml showLineNumbers amps-action-on-disconnect-client amps-action-do-reset-entitlement {{AMPS_AUTHENTICATION_ID}} ``` --- # Shut Down AMPS When a Filesystem Is Full The listing below directs AMPS to perform a graceful shutdown when the filesystem becomes full, with a check run every 3 seconds. ```xml showLineNumbers amps-action-on-schedule 3s amps-action-if-file-system-usage ./ 99% amps-action-do-shutdown ``` --- # Deactivate and Reactivate Security on Signals The listing below disables authentication and entitlement when AMPS receives the `USR1` signal. When AMPS receives the `USR2` signal, AMPS re-enables authentication and entitlement. This configuration is, in effect, the configuration that AMPS installs by default for these signals. ```xml showLineNumbers amps-action-on-signal SIGUSR1 amps-action-do-disable-authentication amps-action-do-disable-entitlement amps-action-on-signal SIGUSR2 amps-action-do-enable-authentication amps-action-do-enable-entitlement ``` --- # Choosing What an Action Does ## Do Element This section outlines the default modules used to specify AMPS behavior when an action is executed, using the `Do` element. --- # Choosing When an Action Runs ## On Element This section outlines the options available for configuring when AMPS runs a given action, using the `On` element. --- # Compress Files AMPS provides the following module for compressing files. Use this action to compress error log files that are no longer needed. AMPS loads this module by default. :::info This action compresses files that match an arbitrary pattern. The original file is removed once the file is compressed. If the pattern is not specified carefully, this action can compress files that contain important data, are required for AMPS, or are required by the operating system. This action cannot be used to safely compress journal files (also known as transaction log files). Use the actions described in [Manage Transaction Log Journal Files](do-manage-journal) for these files. ::: | Module Name | Does | | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `amps-action-do-compress-files` |

    Compresses files that match the specified pattern that are older than the specified age.

    This action accepts an arbitrary pattern, and compresses files that match that pattern. While AMPS attempts to protect against deleting journal files, using a pattern that compresses files that are critical for AMPS, for the application, or for the operating system may result in loss of data.

    The module does not recurse into directories. It skips open files. The module does not compress AMPS journals (that is, files that end with a `.journal` extension), and reports an error if a file with that extension matches the specified `Pattern`.

    The commands to compress files are executed with the current permissions of the AMPS process.

    | This module requires an `Age` parameter that specifies the age of the files to remove, as determined by the update to the file. This module also requires a `Pattern` parameter that specifies a pattern for locating files to remove. | Parameter | Description | | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Age`

    (required)

    |

    Specifies the age of files to process.

    The module removes any file older than the specified `Age` that matches the specified `Pattern`.

    For example, when the `Age` is `5d`, only files that have not been modified within 5 days and that match the pattern will be processed by the module.

    There is no default for this parameter.

    | |

    `Pattern`

    (required)

    |

    Specifies the pattern for files to remove.

    The module removes any files that match the specified `Pattern` that have not been modified more recently than the specified `Age`.

    This parameter is interpreted as a Unix shell globbing pattern. It is not interpreted as a regular expression.

    As with other parameters that use the file system, when the pattern specified is a relative path, the parameter is interpreted relative to the current working directory of the AMPS process. When the pattern specified is an absolute path, AMPS uses the absolute path.

    There is no default for this parameter.

    | | `Keep` |

    Specifies the number of files that meet the `Age` and `Pattern` criteria to leave uncompressed.

    When this parameter is specified, AMPS will compress files matching the criteria, starting with the oldest files, and stop when the number of remaining files is the number specified in this parameter.

    There is no default for this parameter. When both `Keep` and `Count` are specified, AMPS will not compress any files if the number of files meeting the criteria is less than the number specified in the `Keep` parameter.

    | | `Count` |

    Specifies the maximum number of files that meet the `Age` and `Pattern` criteria to compress.

    AMPS will compress files matching the criteria, starting with the oldest files, and stop when the number of files specified in this parameter have been compressed.

    There is no default for this parameter. When both `Keep` and `Count` are specified, AMPS will not compress any files if the number of files meeting the criteria is less than the number specified in the `Keep` parameter.

    | This module does not add any variables to the AMPS context. --- # Create Minidump AMPS provides a module for creating minidumps. The `amps-action-do-minidump` module provides a way for developers and/or administrators to easily create minidumps for diagnostic purposes. Running this module does *not* cause AMPS to exit. |Module Name |Does | |-------------------------------------------------------------|-------------------| |`amps-action-do-minidump`|Creates a minidump. | This module does not require any parameters. This module does not add any variables to the AMPS context. --- # Debug Action Configuration AMPS provides modules for debugging your AMPS action configuration. | Module Name | Does | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `amps-action-do-nothing` |

    Takes no action.

    Does not modify the state of AMPS in any way. The module simply logs that it was called.

    | | `amps-action-do-echo-message` |

    Echoes the specified message to the log. The message appears in the log as message `29-0103`, at `info` level.

    The logging configuration must allow this message to be recorded for the output of this action to appear in the log.

    | The `amps-action-do-nothing` module requires no parameters. The `amps-action-do-echo-message` module requires the following parameter: | Parameter | Description | | --------- | ------------------------------------------------------------------------------ | | `Message` | The message to echo. The default for this parameter is simply an empty string. | These modules do not add any variables to the AMPS context. --- # Remove Files AMPS provides the following module for removing files. Use this action to remove error log files that are no longer needed. AMPS loads this module by default. :::info This action removes files that match an arbitrary pattern. If the pattern is not specified carefully, this action can remove files that contain important data, are required for AMPS, or are required by the operating system. ::: This action cannot be used to safely remove journal files (also known as transaction log files). Use the actions described in [Manage Transaction Log Journal Files](do-manage-journal) for these files. | Module Name | Does | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `amps-action-do-remove-files` |

    Removes files that match the specified pattern that are older than the specified age. This action accepts an arbitrary pattern, and removes files that match that pattern. While AMPS attempts to protect against deleting journal files, using a pattern that removes files that are critical for AMPS, for the application, or for the operating system may result in loss of data.

    The module does not recurse into directories. It skips open files. The module does not remove AMPS journals (that is, files that end with a `.journal` extension), and reports an error if a file with that extension matches the specified `Pattern`.

    The commands to remove files are executed with the current permissions of the AMPS process.

    | This module requires an `Age` parameter that specifies the age of the files to remove, as determined by the update to the file. This module also requires a `Pattern` parameter that specifies a pattern for locating files to remove. | Parameter | Description | | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Age`

    (required)

    |

    Specifies the age of files to process. The module removes any file older than the specified `Age` that matches the specified `Pattern`.

    For example, when the `Age` is `5d`, only files that have not been modified within 5 days and that match the pattern will be processed by the module.

    There is no default for this parameter.

    | |

    `Pattern`

    (required)

    |

    Specifies the pattern for files to remove. The module removes any files that match the specified `Pattern` that have not been modified more recently than the specified `Age`.

    This parameter is interpreted as a Unix shell globbing pattern. It is not interpreted as a regular expression.

    As with other parameters that use the file system, when the pattern specified is a relative path, the parameter is interpreted relative to the current working directory of the AMPS process. When the pattern specified is an absolute path, AMPS uses the absolute path.

    There is no default for this parameter.

    | | `Keep` |

    Specifies the number of files that meet the `Age` and `Pattern` criteria to retain. When this parameter is specified, AMPS will remove files matching the criteria, starting with the oldest files, and stop when the number of remaining files is the number specified in this parameter.

    There is no default for this parameter. When both `Keep` and `Count` are specified, AMPS will not remove any files if the number of files meeting the criteria is less than the number specified in the `Keep` parameter.

    | | `Count` |

    Specifies the maximum number of files that meet the `Age` and `Pattern` criteria to remove. AMPS will remove files matching the criteria, starting with the oldest files, and stop when the number of files specified in this parameter have been removed.

    There is no default for this parameter. When both `Keep` and `Count` are specified, AMPS will not remove any files if the number of files meeting the criteria is less than the number specified in the `Keep` parameter.

    | This module does not add any variables to the AMPS context. --- # Enable or Disable Transports AMPS provides modules that can enable and disable specific transports. The `amps-action-do-enable-transport` module enables a transport. The `amps-action-do-disable-transport` module disables a transport. |Module Name |Does | |----------------------------------------------------------------------|------------------------------| |`amps-action-do-enable-transport` |Enables a specific transport. | |`amps-action-do-disable-transport`|Disables a specific transport. | Both modules require the name of the transport to disable or enable. |Parameter |Description | |----------------------|-------------------------------------------------------------------------------------------------------------------------| |`Transport`|

    The name of the transport to enable or disable.

    If no name is provided, the module affects all transports.

    | Both modules do not add any variables to the AMPS context. --- # Execute System Command The `amps-action-do-execute-system` module allows AMPS to execute system commands. The parameter for this module is simply the command. The command executes in the current working directory of the AMPS process, with the credentials and environment of the AMPS process. The thread used to execute the command is monitored by AMPS thread monitoring. This means that the command must complete within a short period of time (on the order of a second or two at the longest) or AMPS may consider the thread to have become deadlocked. | Parameter | Description | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `Command` |

    The command to execute.

    When the action runs, this command is executed as a shell command on the system where AMPS is running.

    | This module does not add any variables to the AMPS context. :::info This module executes system commands with the credentials of the AMPS process. It is possible to damage the system, interrupt the AMPS service, or cause data loss by executing commands with this module. 60East recommends against using any data extracted from an AMPS message in the command executed. ::: --- # Extract Values from a Message The `amps-action-do-extract-values` module extracts message values from a message and stores the values in a variable. To extract values from a message, this module requires the `MessageType` and `Value` parameters. In addition to that, this module also accepts an optional parameter listed below: | Parameter | Description | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

    `MessageType`

    (required)

    |

    The message type for the message to parse.


    There is no default for this parameter.

    | |

    `Value`

    (required)

    |

    An assignment statement that specifies the variable to store the extracted value in and the XPath identifier for the value to extract.

    This action can contain any number of `Value` elements, each providing an assignment statement.

    The format of the assignment is as follows:

    ` variable=amps-expression`

    For example, the following assignment statement stores the value of the `/previousRegionCode` within the message to the variable `PREVIOUS_REGION`. After this action runs, the content of the variable can be referenced in subsequent action steps as `{{PREVIOUS_REGION}}`.

    ` PREVIOUS_REGION=/previousRegionCode`

    Likewise, the following assignment statement creates a string from the values of the `/firstName` and `/lastName` fields within the message, and stores that to the variable `COMBINED_NAME`. After this action runs, the content of the variable can be referenced in subsequent action steps as `{{COMBINED_NAME}}`.

    ` COMBINED_NAME=CONCAT(/firstName, ' ', /lastName)`

    There is no default for this option. If no `Value` options are provided, AMPS does not save any values from the parsed message.

    | | `Data` |

    Contains the data to parse. Typically it is a message received from a publish event or retrieved from a SOW query.

    The action expands context variables when this action is run, which can be useful for processing variables set by other action steps.

    There is no default value for this parameter. If it is omitted, AMPS will not parse data when the action is run.

    | The module `amps-action-do-extract-values` adds the variables specified by the `Value` options to the current context. --- # Increment Counter The `amps-action-do-increment-counter` module allows AMPS to increment a counter by a value. Counters persist across action runs, and are saved in the instance memory until the instance is restarted. If a counter with the specified name does not currently exist in the instance when the action runs, AMPS creates the counter with a value of 0 and then immediately increments it with the specified value. If the counter is already present, AMPS will simply increment the counter. To see an example of `amps-action-do-increment-counter`, refer to the Action Configuration Examples section at the end of this chapter. This module requires a `Key` that tells AMPS which counter to increment and a `Value` that tells AMPS where to store the incremented value. | Parameter | Description | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | |

    `Key`

    (required)

    |

    The name of the counter that AMPS will increment.

    There is no default value for this parameter.

    | |

    `Value`

    (required)

    | The variable in which to store the current value of the counter. | This module adds a variable that contains the counter, as specified in the `Value` parameter, to the current context. --- # Manage Transaction Log Journal Files AMPS provides the following modules for managing journal files and loads these modules by default: | Module Name | Does | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `amps-action-do-archive-journal` |

    Archives journal files that are older than a specified age to the `JournalArchiveDirectory` specified for the transaction log.

    This action marks files to be archived and then returns.

    | | `amps-action-do-compress-journal` |

    Compresses journal files that are older than a specified age.

    This action marks files to be compressed and then returns.

    | | `amps-action-do-remove-journal` |

    Deletes journal files that are older than a specified age.

    This action marks files to be deleted and then returns.

    If a journal file is currently in use, it will be removed when it is no longer in use.

    | AMPS will only remove journal files that are no longer needed by the instance. AMPS ensures that all replays from a journal file are complete, all queue messages in that journal file have been delivered (and acknowledged, if required), and all messages from a journal file have been successfully replicated before removing the file. Journal files that have been compressed or archived (or both) are still part of the transaction log. AMPS will compress and archive journal files that have undelivered queue messages, or that have not yet been fully replicated, and so on. Each of these modules requires an `Age` parameter that specifies the age of the journal files to process. | Parameter | Description | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Age`

    (required)

    |

    Specifies the age of files to process. The module processes any file older than the specified `Age`.

    For example, when the `Age` is `5d`, only files that have not been written to for longer than 5 days will be processed by the module.

    AMPS does not remove the current journal file or files that are being used for replay, files that are being used for replication or files that contain unacknowledged and unexpired messages in a queue; even if the file is older than the `Age` parameter.

    AMPS does not allow gaps in the journal files, so it will only remove a given file if all previous files have been removed.

    AMPS can compress and archive journal files that are still in use since compressed or archived journal files are part of the transaction log and AMPS will replay messages from these files.

    There is no default for this parameter.

    | These modules do not add any variables to the AMPS context. --- # Manage Queue Transfers AMPS provides actions to enable and disable the "proxied transfer" setting for message queues. As described in the [Failover and Queue Message Ownership](/docs/amps-user-guide/replication/queue_replication#disaster-recovery-and-queue-message-ownership) section of the _Replicating Messages Between Instances_ chapter of the [AMPS User Guide](/docs/amps-user-guide), this setting allows an instance to manage ownership of a queue message that is owned by an instance that is offline (or otherwise unreachable from this instance). 60East recommends that these actions only be configured in situations where the administrative interface is unavailable. Unlike most other settings for which actions are provided, the "proxied transfer" setting is intended for disaster recovery rather than normal maintenance. The `amps-action-do-enable-proxied-transfer` action can be used to enable proxied transfer for a queue. The `amps-action-do-disable-proxied-transfer` action can be used to disable proxied transfer for a queue. Enabling proxied transfer will allow this instance to take ownership of messages owned by an unreachable instance, allowing those messages to be delivered. However, it also introduces the risk of duplicate delivery and duplicate processing in cases where the instance that currently owns the message is still online (but the replication network connection has failed), or in cases where the instance that currently owns the message becomes available again and begins servicing requests. See the [Failover and Queue Message Ownership](/docs/amps-user-guide/replication/queue_replication#disaster-recovery-and-queue-message-ownership) section for more details. Although an action is provided for flexibility, 60East does not recommend that the `enable_proxied_transfer` setting is enabled automatically, due to the possibility of duplicate delivery. Instead, this action should be configured to run in response to action by an administrator, after the administrator determines that one of the instances that contains the queue is, in fact, offline and that the possibility of duplicate processing is an acceptable risk. :::info These actions are intended to assist in maintaining message delivery from a distributed queue during an outage of one of the instances that services the queue. When this setting is enabled, the possibility of duplicate delivery exists. ::: These actions require the following parameters: | Parameter | Description | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | |

    `Topic`

    (required)

    |

    The name of the queue topic for which to enable or disable proxied transfer.

    There is no default value for this parameter.

    | |

    `MessageType`

    (required)

    |

    The message type of the topic for which to enable or disable proxied transfer.

    There is no default value for this parameter.

    | This module does not add any variables to the AMPS context. --- # Manage Security AMPS provides modules for managing the security features of an instance. Authentication and entitlement can be enabled or disabled, which is useful for debugging or auditing purposes. You can also reset security and authentication, which clears the AMPS internal caches and gives security and authentication modules the opportunity to reinitialize themselves, for example, by re-parsing an entitlements file. AMPS loads the following modules by default: |Module Name |Does | |---------------------------------------------------------------------------|--------------------------------------------------------------------------------| |`amps-action-do-disable-authentication`|Disables authentication for the instance. | |`amps-action-do-disable-entitlement` |Disables entitlement for the instance. | |`amps-action-do-enable-authentication` |Enables authentication for the instance. | |`amps-action-do-enable-entitlement` |Enables entitlement for the instance. | |`amps-action-do-reset-authentication` |Resets authentication by clearing AMPS caches and reinitializing authentication.| |`amps-action-do-reset-entitlement` |Resets entitlement by clearing AMPS caches and reinitializing entitlement. | These modules require no parameters. The `amps-action-do-reset-authentication` module and the `amps-action-do-reset-entitlement` module accept an optional `Transport` parameter which specifies the transport to reset. |Parameter |Description | |-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------| |`Transport` |

    The name of the transport for which to reset authentication or entitlements.

    If no name is provided, these modules affect all transports.

    | |`AuthenticationId`|

    The authentication ID of the client for which to reset entitlements.

    If no ID is provided, these modules affect all clients.

    | These modules do not add any variables to the AMPS context. --- # Truncate Statistics AMPS provides the following modules for managing the statistics database. As a maintenance strategy, 60East recommends truncating statistics on a regular basis. This frees space in the database file, which will be reused as new statistics are generated. It is generally not necessary to vacuum statistics unless you have changed your retention policy so that less data is retained between truncation operations. With regular truncation, the statistics database file will usually stabilize at the correct size to hold the amount of data your application generates between truncation operations. AMPS loads these modules by default. | Module Name | Usage | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `amps-action-do-truncate-statistics` |

    Removes statistics that are older than a specified age.

    This frees space in the statistics file, but does not reduce the size of the file.

    | | `amps-action-do-vacuum-statistics` |

    Deprecated

    This action has been deprecated in current versions of AMPS and will no longer vacuum statistics. 60East recommends offline maintenance of the statistics database instead.

    See the section on [AMPS Statistics](../../../amps-user-guide/amps-statistics), in the AMPS User Guide, for details on shrinking the size of a statistics database.

    | The `amps-action-do-truncate-statistics` module requires an `Age` parameter that specifies the age of the statistics to process. | Parameter | Description | | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Age`

    (required)

    |

    Specifies the age of the statistics to remove. The module processes any file older than the specified `Age`.

    For example, when the `Age` is `5d`, the module removes statistics that are older than 5 days.

    There is no default for this parameter.

    | These modules do not add any variables to the AMPS context. --- # Publish Message The `amps-action-do-publish-message` module publishes a message into a specified topic. Publishes from this action are treated as publishes from an AMPS client inside the AMPS engine. This means that: * There are no user credentials associated with the publish, so entitlements are not applied. * There is no special handling for the publish. The publish is recorded in the transaction log exactly as if it arrived from outside of the instance and is processed within the instance as if the publish had arrived from an external publisher. :::danger This action is treated by the AMPS engine as a publish from an internal AMPS client. When an `amps-action-do-publish-message` runs in response to the `amps-action-on-publish-message` event or the `amps-action-on-deliver-message` event, use caution, the message published from this action could cause the event to trigger again. This warning includes cases where the action publishes to a topic directly monitored by the action and cases where the action monitors a view and publishes to an underlying topic of the view. The warning also applies to configurations in which two or more actions "cross publish" to topics that are monitored by the other action. An example of the last case is an action that monitors `TopicOne` and publishes to `TopicTwo`, while another action monitors `TopicTwo` and publishes to `TopicOne`. The result of a configuration like the ones described above is called a _publish loop_. AMPS does not support unterminated publish loops or loops that produce a large number of cycles before terminating. ::: To publish a message, this module requires a `Topic`, the `MessageType` to publish on and also the `Data` that the message will contain. In addition to that, this module also accepts optional parameters listed below: | Parameter | Description | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

    `Topic`

    (required)

    | The topic of the message being published. | |

    `MessageType`

    (required)

    |

    The message type for the topic.

    There is no default for this parameter.

    | |

    `Data`

    (required)

    | The data that the message will contain. | | `Delta` |

    Whether to use a delta publish.

    When this option is present, and the value is `true`, the action will use a delta publish.

    | | `UpdateOnly` |

    Specifies whether a delta publish is allowed to insert a record, or only update a record.

    When a delta publish is specified (that is, `Delta` is `true` ), and this option is set to `true`, AMPS will only accept the publish if there is a record present to be updated.

    When no value is specified, this option is `false`.

    | This module does not add any variables to the AMPS context. --- # Query SOW Topic AMPS provides a module for querying a SOW topic. The `amps-action-do-query-sow` queries the SOW topic, and stores the first message returned by the SOW query into a user-defined variable. This module requires the `Topic`, `MessageType` and `Filter` parameters to identify the query to run. This module requires the `CaptureData` parameter in order to be able to store the result of the query. | Parameter | Description | | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Topic`

    (required)

    |

    The name of the topic to query.

    This topic must be a SOW topic, a view, a queue, or a conflated topic.

    There is no default for this parameter.

    This parameter supports regular expressions.

    | |

    `MessageType`

    (required)

    |

    The message type of the topic to query.

    There is no default for this parameter

    | |

    `Filter`

    (required)

    |

    Set the filter to apply.

    If a `Filter` is present, only messages matching that filter will be returned by the query.

    | |

    `CaptureData`

    (required)

    | Sets the name of the variable within which AMPS will store the first message returned. | | `DefaultData` | If no records are found, AMPS stores the `DefaultData` in the variable specified by `CaptureData`. | | `OrderBy` |

    An `OrderBy` expression to use to order the results returned by the query.

    For example, to order in descending order of the `/date` field in the messages, you would provide an `OrderBy` option of `/date DESC`.

    | | `Options` |

    The options for the query.

    This action accepts any valid option for a `sow` command except the `top_n` option (since this action is already limited to a `top_n` value of `1`).

    | Once you query a message from the SOW topic, you can use the captured data in other actions. The example below uses `amps-action-do-query-sow` to query the SOW on a schedule in order to echo messages to the log for diagnostic purposes: ```xml showLineNumbers amps-action-on-schedule Saturday at 23:59 Diagnostic_Schedule amps-action-do-query-sow xml SOW_TOPIC /Trans/Order/@Oname = 'PURCHASE' AMPS_DATA amps-action-do-extract-values xml {{AMPS_DATA}} SAVED_VARIABLE=/Value amps-action-do-echo-message {{SAVED_VARIABLE}} was in the message ``` --- # Raise a Custom Event The AMPS action system provides a way to create custom events and run an action when the custom event is raised. A custom event is raised by the `amps-action-do-execute-event` action. Events are received by the `amps-action-on-execute-event` module. This action requires **one** of the following parameters: | Parameter | Description | | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | |

    `Event`

    (required)

    |

    The name of the event to raise.

    There is no default value for this parameter.

    | |

    `EventVariable`

    (required)

    |

    The variable from which to retrieve the name of the event to raise.

    There is no default value for this parameter.

    | This module does not add any variables to the AMPS context. However, the variables currently in the AMPS context will be received by any action that runs in response to this event. --- # Manage Replication Acknowledgment AMPS provides modules for downgrading replication destinations that fall behind and upgrading them again when they catch up. | Module Name | Does | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `amps-action-do-downgrade-replication` | Downgrades replication connections from synchronous to asynchronous if the age of the last acknowledged message is older than a specified time period. | | `amps-action-do-upgrade-replication` |

    Upgrades previously-downgraded replication connections from asynchronous to synchronous if the age of the last acknowledged message is more recent than a specified time period.

    This action has no effect on replication destinations that are specified as `async` in the configuration file.

    | The modules determine when to downgrade and upgrade based on the age of the oldest message that a destination has not yet acknowledged. When using these modules, it is important that the thresholds for the modules are not set too close together, otherwise; AMPS may repeatedly upgrade and downgrade the connection when the destination is consistently acknowledging messages at a rate close to the threshold values. To avoid this, 60East recommends that the `Age` set for the upgrade module is 1/2 of the age used for the downgrade module. The `amps-action-do-downgrade-replication` module accepts the following options: | Parameter | Description | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

    `Age`

    (required)

    |

    Specifies the maximum message age at which AMPS downgrades a replication destination to `async`.

    When this action runs, AMPS downgrades any destination for which the oldest unacknowledged message is older than the specified `Age`.

    For example, when the `Age` is `5m`, AMPS will downgrade any destination where a message older than 5 minutes has not been acknowledged.

    There is no default for this parameter.

    | | `GracePeriod` |

    The approximate time to wait after start up before beginning to check whether to downgrade links.

    The `GracePeriod` allows time for other AMPS instances to start up and for connections to be established between AMPS instances.

    | The `amps-action-do-upgrade-replication` module only applies to destinations configured as `sync` that have been previously downgraded. The module accepts the following options: | Parameter | Description | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Age`

    (required)

    |

    Specifies the maximum message age at which a previously-downgraded destination will be upgraded to `sync` mode.

    When this action runs, AMPS upgrades any destination that has been previously downgraded where the oldest unacknowledged message is more recent than the time value specified in the `Age` parameter.

    For example, if a destination has been downgraded to `async` mode and the `Age` is `2m`, AMPS will upgrade the destination when the oldest unacknowledged message to that destination is less than 2 minutes old.

    There is no default for this parameter.

    | | `GracePeriod` |

    The approximate time to wait after start up before beginning to check whether to upgrade links.

    The `GracePeriod` allows time for other AMPS instances to start up, and for connections to be established between AMPS instances.

    | These modules do not add any variables to the AMPS context. --- # Rotate Error/Event Log AMPS provides the following module for rotating log files and loads this module by default: |Module Name |Does | |----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`amps-action-do-rotate-logs`|

    Rotates logs that are older than a specified age, for log types that support log rotation. Rotating a log involves closing the log and opening the next log in sequence.

    AMPS will use the name specifier provided in the AMPS configuration for the new log file. This may overwrite the current log file if the specifier results in the same name as the current log file.

    | This module does not require options. This module does not add any variables to the AMPS context. --- # Shut Down AMPS The `amps-action-do-shutdown` module shuts down AMPS. This module is registered as the default action for several Linux signals, as described in the section on starting actions [on a Linux signal](../on-elements/on-signal). Causing this module to run is the recommended way to shut down AMPS. This is most often done by sending `SIGINT` to the AMPS process, which is handled by `amps-action-on-signal` to run this module. | Module Name | Does | | ------------------------- | ---------------- | | `amps-action-do-shutdown` | Shuts down AMPS. | This module does not require any parameters. This module does not add any variables to the AMPS context. --- # Compact SOW Topic AMPS also provides a module for reducing the unused space in a SOW file. The `amps-action-do-compact-sow` module rearranges the messages in the SOW into a smaller amount of space, where possible. Since AMPS uses memory-mapped files to store messages for the SOW topic, this can also potentially reduce the memory footprint of the topic. This module can compact a specific SOW file or the SOW files for every topic in the instance. While messages are being added or updated within a topic in the SOW, AMPS reuses free space when possible. It is not necessary to compact the SOW file during most normal operations. This action is most useful after an activity peak that leaves a large amount of unneeded space in the file, or in installations where space is at a premium. Depending on the file size, the number of topics to be compacted, and the amount of free space, the reorganization that this operation performs may require a noticeable amount of I/O bandwidth. 60East recommends that this action run during a maintenance window or in response to a critical lack of disk space. This operation will pause updates to the topic while it is being compacted, reducing overall throughput even if the compaction process runs quickly. When a `Topic` and `MessageType` are provided, this module compacts the SOW file for that topic. Otherwise, the module compacts the file for all topics in the SOW. | Parameter | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `Topic` |

    The name of the SOW topic from which to compact.

    This option must be specified if the `MessageType` is provided.

    There is no default for this parameter.

    | | `MessageType` |

    The message type of the SOW topic or topics to compact.

    This option must be specified if the `Topic` is provided.

    There is no default for this parameter.

    | This module does not add any variables to the AMPS context. --- # Delete SOW Messages AMPS also provides modules for deleting SOW contents. The `amps-action-do-delete-sow` module deletes messages from the specified SOW topic. This module requires the `Topic`, `MessageType` and `Filter` parameters in order to delete the desired message. | Parameter | Description | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

    `Topic`

    (required)

    |

    The name of the SOW topic from which to delete messages.

    This parameter supports regular expressions.

    There is no default for this parameter.

    | |

    `MessageType`

    (required)

    |

    The message type of the SOW topic or topics to delete from.

    There is no default for this parameter.

    | |

    `Filter`

    (required)

    |

    Set the filter to apply.

    Only messages matching that filter will be deleted.

    | This module does not add any variables to the AMPS context. --- # Translate Data Within an Action The `amps-action-do-translate-data` action allows you to translate the value from variables in the current context. One common use for this action is to translate a large number of status values into a smaller number of states before publishing that information in a message. For example, an order processing system may track a large number of finely-grained status codes, while the reporting view for customers may want to map those status codes to a smaller set of codes such as "pending", "shipped" and "delivered". This action allows you to easily translate those codes within AMPS. When used to assemble a message, this action provides equivalent results to a set of nested conditional statements in a view projection. However, if you are using actions to parse, assemble and publish messages, this action gives you the ability to change values. | Parameter | Description | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

    `Data`

    (required)

    | The data to translate. Most often, this is the value of a variable in the current context. | |

    `Value`

    (required)

    | The variable to store the translated value in. | | `Case` |

    A translation statement. The translation statement takes the form of `original_value=translated_value`.

    This action allows you to provide any number of `Case` statements.

    The action matches the `Data` provided to the `original_value` in each `Case` statement. When it finds a matching value, the action stores the translated value in the variable identified by the `Value` statement.

    For example, the following translation statement translates a value of `credit_check_in_progress` to a value of `pending`.

    ```credit_check_in_progress=pending```

    There is no default for this option.

    | | `Default` |

    The default translation. AMPS sets the value of the variable to the contents of this element if no `Case` statement matches the `Data` provided.

    This element is optional. If no `Default` is specified, AMPS uses the value of the original `Data` as the default translation.

    | --- # Based on an Expression AMPS provides an `If` module for stopping execution of the action unless a specific condition is met. | Module Name | Does | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `amps-action-if-condition` |

    Checks whether the specified expression is true or false.

    If the expression evaluates to true, subsequent steps in the action run. If not, the action stops at this step.

    | | Parameter | Description | | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Condition`

    (required)

    |

    Specifies the condition to evaluate. The condition must be a valid AMPS filter.

    The condition will substitute variables from the current context before being evaluated.

    There is no default for this parameter.

    | This module does not add any variables to the AMPS context. For example, the following action publishes a message to a topic if a client with a name that matches a specific expression logs on. ```xml showLineNumbers amps-action-on-connect-client amps-action-if-condition '{{AMPS_CLIENT_NAME}}' LIKE 'important' amps-action-do-publish-message important-logon-notification json {"name":"{{AMPS_CLIENT_NAME}}"} ``` :::info The example above is provided to illustrate use of `amps-action-if-condition`. In practice, this could also be done using an `amps-action-on-publish-message` action to monitor the `/AMPS/ClientStatus` topic with a filter. ::: --- # Based on File System Capacity AMPS provides the following `If` module for taking action based on the file system capacity. AMPS loads this module by default: | Module Name | Does | | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `amps-action-if-file-system-usage` | Checks whether the specified path on the filesystem meets the specified usage level. If so, allows execution to continue. If not, stops the action. | | Parameter | Description | | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Path`

    (required)

    |

    Specifies the filesystem path to monitor.

    The AMPS process must have sufficient permissions to check the disk usage for this path at the time the check runs.

    There is no default for this parameter.

    | |

    `GreaterThan`

    (required)

    |

    The threshold to check, specified as a percentage.

    If the provided path has more space used than specified in this parameter, subsequent `Do` and `If` blocks will run. Otherwise, the action will complete with this step.

    | This module does not add any variables to the AMPS context. For example, the following action will log a message in the AMPS log every minute when the file system becomes more than 90% full, and perform a full shutdown of AMPS if the file system is more than 98% full. ```xml showLineNumbers amps-action-on-schedule 1m amps-action-if-file-system-usage 90% /mnt/fastdrive/amps amps-action-do-echo-message ALERT: You're getting low on space! amps-action-if-file-system-usage 98% /mnt/fastdrive/amps amps-action-do-echo-message CRITICAL: Shutting down AMPS amps-action-do-shutdown ``` --- # Conditionally Stopping an Action ## If Element AMPS includes the ability to run actions only if certain conditions are true. For some actions (such as the replication management actions), the condition is included as a part of the action. In other cases, AMPS provides `If` actions. An `If` action is evaluated each time the execution of an action reaches the `If` action, that is, when all of the previous `Do` steps have been called. When the condition specified in an `If` action is `true`, AMPS proceeds to the next `Do` action. If the condition in an `If` action is `False`, AMPS does not run any further `Do` elements in the action. --- # On Client Connect or Disconnect AMPS provides modules for running actions on the connection or disconnection of an AMPS client. The `amps-action-on-disconnect-client` runs actions once an AMPS client instance disconnects. The `amps-action-on-connect-client` runs actions once an instance of an AMPS client successfully connects. These modules require no parameters. These modules add the following variables to the AMPS context: |Variable |Description | |-----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`AMPS_CLIENT_NAME` |

    The name of the AMPS client.

    (For the `amps-action-on-connect-client` action, this will be the connection name since this event happens before the client has provided a client name in the logon command.)

    | |`AMPS_CONNECTION_NAME` |The name of the AMPS connection. | |`AMPS_AUTHENTICATION_ID`|The authentication ID of the AMPS connection (for `amps-action-on-disconnect-client` action only).| --- # On Client Offline Message Buffering AMPS provides modules to run actions when an AMPS client is marked as a slow client and also for when the AMPS client catches up to no longer be subject to slow client offlining. Slow client offlining is a feature in AMPS that reduces the memory resources consumed by slow clients. More on this feature can be found in [Slow Client Management and Capacity Limits](../../../amps-user-guide/ha/slow-client-management-and-capacity-limits). The `amps-action-on-offline-start` module runs actions as the first step when AMPS's result set reaches its disk limit and has to disconnect the client. The `amps-action-on-offline-stop` module runs actions as AMPS is no longer subject to slow client offlining. In both cases, actions run in the order that the actions appear in the configuration file. Both modules do not require any parameters. Both modules add the following variables to the AMPS context: | Variable | Description | | ---------------------- | -------------------------------- | | `AMPS_CLIENT_NAME` | The name of the AMPS client. | | `AMPS_CONNECTION_NAME` | The name of the AMPS connection. | --- # On SOW Message Delete AMPS provides a module to run an action when a message is deleted from a topic in the SOW. The `amps-action-on-sow-delete-message` module monitors a topic for deletions from the SOW. The action runs once for each message that is deleted in the matching topic. This action requires the following parameters: | Parameter | Description | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

    `Topic`

    (required)

    |

    The name of the topic to monitor for messages.

    This parameter does not support regular expressions. The topic name must be either a SOW topic, a view, a conflated topic or a queue.

    There is no default for this parameter.

    | |

    `MessageType`

    (required)

    |

    The message type of the topic to monitor for messages.

    There is no default for this parameter.

    | The module adds the following variables to the AMPS context: | Variable | Description | | --------------------- | -------------------------------------------------------------- | | `AMPS_TOPIC` | The topic of the message that was deleted. | | `AMPS_DATA` | The current data of the message. | | `AMPS_DATA_LENGTH` | The length of the current data of the message, in bytes. | | `AMPS_CORRELATION_ID` | The correlation ID set on the publish command, if one was set. | --- # On Custom Event The AMPS action system provides a way to create custom events and run an action when the custom event is raised. This can be useful in cases where an identical set of `Do` steps should be run in response to several different events. The `amps-action-on-execute` event runs when a specified custom event is raised by the `amps-action-do-execute-event` action. The AMPS context for this event is the same as the AMPS context from which the `amps-action-do-execute-event` runs. This action requires the following parameters: | Parameter | Description | | --------------------------- | ---------------------------- | | `Event` | The event to respond to. When an `amps-action-do-execute-event` that specifies the same `Event` runs, this action will run.
    There is no default for this parameter. | This module does not add any variables to the AMPS context. --- # On SOW Message Expiration AMPS provides a module to run an action when a message expires from a topic in the SOW. The `amps-action-on-sow-expire-message` module monitors a topic for expirations. The action runs once for each message that expires in the matching topic. Notice, in particular, that this includes monitoring messages that expire from the queue, which are presented as SOW expirations to this module. This action requires the following parameters. In addition to that, this module also accepts an optional parameter listed below: | Parameter | Description | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

    `Topic`

    (required)

    |

    The name of the topic to monitor for messages.

    This parameter does not support regular expressions. The topic name must be either a SOW topic, a view, a conflated topic or a queue.

    There is no default for this parameter.

    | |

    `MessageType`

    (required)

    |

    The message type of the topic to monitor for messages.

    There is no default for this parameter.

    | | `Reason` |

    An optional comma-delimited string indicating the expiration reasons to monitor for.

    For example, setting this to `time_limit,forced_expire` will cause this action to be invoked only when the expiration reason includes one of `time_limit` or `forced_expire`.

    For more information on the allowed values and definitions, see the documentation for `AMPS_REASON` below.

    | The module adds the following variables to the AMPS context: | Variable | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AMPS_TOPIC` | The topic of the message that expired. | | `AMPS_DATA` | The current data of the message. | | `AMPS_DATA_LENGTH` | The length of the current data of the message, in bytes. | | `AMPS_REASON` |

    A comma-delimited string indicating one or more reason(s) the message was expired.

    This string contains one or more of the following values:

    • `time_limit` - the expiration time for this message or topic has passed.
    • `forced_expire` - this queue message was acknowledged with an `expire` option resulting in the message's immediate removal.
    • `max_cancels` - this queue message was acknowledged with a `cancel` option and the number of cancels exceeded the queue's configured `MaxCancels` limit.
    • `max_deliveries` - this queue message was not successfully acknowledged by a subscriber and the number of deliveries has exceeded the queue's `MaxDeliveries` limit.
    | | `AMPS_CORRELATION_ID` | The correlation ID set on the publish command if one was set. | --- # On a REST Request AMPS includes a module that allows you to configure an action to run when a specified resource underneath the `/amps/administrator/actions` path in the admin interface is requested. The `amps-action-on-admin` module allows you to add a custom action to the admin interface and provides the ability to pass parameters into the `Do` step of those actions using query parameters in the HTTP request. The module requires the `Path` parameter, which specifies the path under `/amps/administrator/actions` where this action should appear: | Parameter | Description | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

    `Path`

    (required)

    |

    Specifies the path under `/amps/administrator/actions` where this action should appear. The action will run when a request for the specified path is received by the AMPS admin interface.

    For entitlement purposes, the AMPS admin interface treats actions configured using this module identically to resources provided by AMPS. That is, the user requesting that the action run must have access to the appropriate admin resource for the action to run.

    For example, to add an action that runs journal maintenance at the path `/amps/administrator/actions/journal_cleanup`, you would use the following `Path` element:

    ` journal_cleanup`

    The `Path` element should contain only a resource name. No `/` character should appear in the `Path` element.

    | Any query parameters provided as part of the request are added to the context before the `Do` steps for the action are run. This module adds the names and values of the query parameters without adjusting the case of the items. For some actions, it's important that a specific context value is present before the action runs. You can configure the module to require that the request provide one or more query parameters using the `RequiredParameter` option to the module. When one or more `RequiredParameter` is specified, the admin console will refuse any request for the resource that does not include all of the required parameters. The action also provides support for a `Name` element and a `Description` element, which can be used to help tools that monitor or manage AMPS (such as the Galvanometer monitoring tool included with AMPS) provide information on the action. | Optional Element | Description | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `RequiredParameter` | Specifies that a request to run this action must include the specified query parameter. If the request does not include this item as part of the query parameters, the request will return an error rather than executing the action. | | `Name` | Specifies the name to use for display purposes for this resource. | | `Description` | Specifies a description to use for display purposes for this resource. | For an example of configuring an action using this module, see [Archive Journals on RESTful Command](../../action-examples/archive-on-demand). :::warning The `amps-action-on-admin` module should be used for ad hoc maintenance (such as responding to an unexpected increase in traffic by archiving journals ahead of schedule) or tasks that are triggered unpredictably by an external system. For tasks that need to run on a regular basis or frequently (for example, every few seconds), `amps-action-on-schedule` is more efficient. ::: --- # On Incoming Replication Connections AMPS includes a set of modules that allow you to configure an action to run based on events for incoming replication connections. The `amps-action-on-connect-incoming-replication` module runs when an incoming replication connection is accepted. The `amps-action-on-disconnect-incoming-replication` module runs when an incoming replication connection is disconnected. These modules require no parameters. These modules add the following variables to the AMPS context: |Variable |Description | |--------------------------------------------|-------------------------------------------------------------------------------------------| |`AMPS_REPLICATION_PEER_NAME` |The instance name of the AMPS instance on the opposite end of the connection, if available.| |`AMPS_REPLICATION_CLIENT_NAME` |The name of the AMPS client used for this connection.| |`AMPS_REPLICATION_REMOTE_ADDRESS`|The remote address of the opposite end of the connection.| |`AMPS_REPLICATION_GROUP_NAME` |The group name of the AMPS instance on the opposite end of the connection, if available.| --- # On Client Logon AMPS provides the `amps-action-on-logon-client` module for running actions when a user logs into an AMPS client. This module does not require any parameters. This module adds the following variables to the AMPS context: |Variable |Description | |-----------------------------------|---------------------------------------------| |`AMPS_CLIENT_NAME` |The name of the AMPS client. | |`AMPS_CONNECTION_NAME` |The name of the AMPS connection. | |`AMPS_AUTHENTICATION_ID`|The authentication ID of the AMPS connection.| --- # On Message Affinity AMPS includes a module to help with building a message affinitization strategy. With message affinitization, each record in a SOW Topic can be assigned to a single affinitized client. The `amps-action-on-message-affinity` module handles monitoring the SOW topic and running an action when a key is affinitized or de-affinitized. A client connection indicates that it wants to participate in affinitization by subscribing to a topic to be used for the assignment metadata, referred to as the "control topic". Notice that although this action runs when message affinitization is updated, the `On` step does not, by itself, specify what happens when affinitization is updated. It is up to the Action configuration to take appropriate steps when affinitization is updated. :::info This module keeps track of affinitizing messages to processors and runs an action when affinity is assigned or changed. However, it is up to the configuration of the action to manage alerting processors, and it is up to the processors themselves to respond to the alert and adjust their subscriptions accordingly. ::: This action requires the following parameters. In addition to that, this module also accepts an optional parameter listed below: | Parameter | Description | | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |

    `MessageType`

    (required)

    | The message type of the monitored topic and the control topic. | |

    `ControlTopic`

    (required)

    |

    The name of the control topic.

    This topic is monitored for subscriptions. Any client that subscribes to the `ControlTopic` is considered to be eligible to have messages affinitized to that client.

    | |

    `DataTopic`

    (required)

    |

    The name of the topic that contains the messages to be affinitized. Each distinct record in the topic (that is, each distinct SOW Key) will be affinitized.

    By default, all data within the topic will be affinitized. Optionally, you can restrict affinitization to only certain records (for example, orders in a certain state) by setting the `DataFilter` option.

    | | `DataFilter` | When present, restricts affinitization to only those messages in the `DataTopic` that also match the `DataFilter`. | This module adds the following variables to the AMPS context: | Variable | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AMPS_DATA` | The data of the message being affinitized. | | `AMPS_CLIENT_NAME` | The name of the client the message is being affinitized to or removed from. | | `AMPS_AFFINITY_ACTION` |

    The type of event.

    If the message is being affinitized to this client, the type will be `assign`. If the message is being de-affinitized, the type will be `unassign`.

    | | `AMPS_AFFINITY_REASON` |

    The reason for the event.

    For example, if the message no longer matches the `DataFilter` or has been deleted, the reason would be `oof`. If the update is happening because the client that was assigned to the message is no longer active, the reason would be `unsubscribe`.

    | ## Considerations and Limitations The `amps-action-on-message-affinity` module has the following considerations for use: * Affinitization is tracked independently on _each_ instance of AMPS. In a replicated mesh of AMPS instances, each instance independently monitors the `DataTopic` and `ControlTopic`. AMPS does not make any effort to coordinate affinitization across replicated instances. * The affinitization module relies on the SOW key (or grouping, for affinitization over views) to determine message identity. Therefore, affinitization can only be used for a `Topic`, `RegexTopic`, `View`, or `ConflatedTopic`. Since each message in a `Queue`, `LocalQueue`, or `GroupLocalQueue` is considered to be a distinct message, monitoring one of these topic types for affinitization would not produce useful results (however, if the underlying topic of a queue is maintained as a `Topic` in the SOW, using that `Topic` for affinitization could be useful). ## Affinitization Configuration Example The following example shows one way to use the `amps-action-on-message-affinity` module to assign specific symbols to a processor. In this example, a processor would subscribe to the `symbol_processor_assignments` topic to receive the symbols that have been affinitized to that processor. It would then maintain a subscription to the `orders` topic with a filter that limits the subscription to just the symbols that have been provided to the subscriber from the `symbol_processor_assignments` topic. When a processor receives an event with a symbol and the event `"assign"`, the processor adds the symbol in that event to the list of symbols in the filter for the subscription to the orders topic (typically, using the `replace` option to adjust the existing subscription in place). When a processor receives an event with a symbol and the event `"unassign"`, the processor removes that symbol from the filter for the subscription to the orders topic. ```xml showLineNumbers amps-action-on-message-affinity json symbols symbol_processor_assignments amps-action-do-extract-values {{AMPS_DATA}} json SYMBOL=/symbol amps-action-do-publish-message json symbol_processor_assignments {"client_name":"{{AMPS_CLIENT_NAME}}", "symbol":"{{SYMBOL}}", "event":"{{AMPS_AFFINITY_ACTION}}", "reason":"{{AMPS_AFFINITY_REASON}}"} amps-action-if-condition "{{AMPS_AFFINITY_ACTION}}" == "unassign" amps-action-do-delete-sow symbol_processor_assignments json /symbol = "{{SYMBOL}}" ``` The underlying topics can be defined as needed. For example, the following configuration automatically tracks the set of symbols in an orders topic to be affinitized using the action above: ```xml showLineNumbers orders json /id ${AMPS_DATA}/sow/%n.sow symbols json orders /symbol /symbol symbol_processor_assignments json /symbol transient ``` This action attempts to affinitize new messages so as to keep the overall number of messages roughly balanced across the current processors. If a processor unsubscribes from the `ControlTopic`, messages currently affinitized to that processor will be distributed to other processors. However, messages are not rebalanced among running processors. Once a given SOW Key is affinitized to a processor, it remains affinitized to that processor until the processor unsubscribes, or the record is deleted from the topic. --- # On Message Delivered to Subscriber AMPS provides modules to run actions when AMPS delivers a message to subscribers. The basic flow of AMPS messaging is to first receive a published message, find the subscriber(s) to which this message will be sent, then deliver the message to the subscriber(s). The `amps-action-on-deliver-message` runs actions when AMPS delivers a `publish` message to subscribers. This module requires the `Topic` and the `MessageType` of the message that has been delivered. | Parameter | Description | | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Topic`

    (required)

    |

    The name of the topic to monitor for message delivery. This parameter supports regular expressions.

    There is no default for this parameter.

    | |

    `MessageType`

    (required)

    |

    The message type of the topic to monitor for message delivery.

    There is no default for this parameter.

    | This module adds the following variables to the AMPS context: | Variable | Description | | --------------------- | ----------------------------------------------------------------------------------------------------------- | | `AMPS_TOPIC` | The topic of the message. | | `AMPS_DATA` | The data the message contains. | | `AMPS_DATA_LENGTH` | The length of the data the message contains. | | `AMPS_BOOKMARK` | The bookmark associated with this message. This is an empty string if the message does not have a bookmark. | | `AMPS_CLIENT_NAME` | The name of the client to which this message was delivered. | | `AMPS_CORRELATION_ID` | The correlation ID set on the publish command, if one was set. | --- # On Message Published to AMPS AMPS provides modules to run actions when a message is published to AMPS. The basic flow of AMPS messaging is to first receive a published message, find the subscriber(s) to which this message will be sent, then deliver that message to the subscriber(s). The `amps-action-on-publish-message` runs actions as soon as a message is published to AMPS. This action will only receive messages when the instance is active. This means that the action is *not* active during recovery, and will *not* publish duplicate messages when the topic the action is monitoring is recovered. This also means that adding this action to a configuration will not replay messages that were published to AMPS before the action was added. :::warning This action is treated by the AMPS engine as a subscription from an internal AMPS client. When working with queues, use this action with the _underlying topic_ of the queue rather than the queue topic itself. Because this action creates a subscription, using this action with the queue topic will cause the action to lease messages from the queue even though the action does _not_ acknowledge messages. This means that, when used with the queue topic itself, the action will interfere with other subscribers and depending on the queue configuration, may only receive one message during the lifetime of the instance. ::: This module requires the `Topic` and the `MessageType` of the message that was published. In addition to that, this module also accepts the optional parameters listed below: | Parameter | Description | | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Topic`

    (required)

    |

    The name of the topic to monitor for publishes.

    This parameter supports regular expressions.

    There is no default for this parameter.

    | |

    `MessageType`

    (required)

    |

    The message type of the topic to monitor for publishes.

    There is no default for this parameter.

    | | `MessageSource` |

    The source to monitor for publishes. The source of the message defaults to `all`, which monitors both publishes directly to this AMPS instance and messages received via replication.

    This parameter also accepts `local` for when the message source is published directly to this AMPS instance and `replicated` for messages received via replication.

    | | `Filter` |

    Sets the filter to apply.

    Only messages that match this filter will cause the action to run.

    | | `Options` |

    Sets the options to apply when listening for messages.

    This action supports any option that would be supported by the subscribe command when not using a bookmark.

    | This module adds the following variables to the AMPS context: | Variable | Description | | ------------------ | ------------------------------------------------------------ | | `AMPS_TOPIC` | The topic of the message. | | `AMPS_DATA` | The data the message contains. | | `AMPS_DATA_LENGTH` | The length of the data that the message contains. | | `AMPS_BOOKMARK` | The bookmark associated with this message. | | `AMPS_TIMESTAMP` | The time at which the message was processed by AMPS. | | `AMPS_CLIENT_NAME` | The name of the client from which the message was published. | --- # On Minidump Creation AMPS provides the `amps-action-on-minidump` module for running actions when AMPS generates a minidump. This module does not require parameters. This module adds the following variable to the AMPS context: |Variable |Description | |-------------------------------|------------------------------------------| |`AMPS_MINIDUMP_PATH`|The path to where the minidump is created.| --- # On Message State Change AMPS provides two modules, `amps-action-on-alert` and `amps-action-on-message-condition-timeout`, to run an action when a message in a SOW topic meets a specific condition for longer than a specified period of time. These modules provide a way to track specific messages in a SOW topic and indicate whether the message has timed out (that is, been in the monitored state longer than expected) or whether the message has been processed as expected and is no longer being tracked within the specified timeout. The `amps-action-on-alert` action monitors a SOW topic for messages that match a filter and triggers an action when either: * The message has remained matched on the filter for at least the specified duration (indicating a timeout), _or_ * An out-of-focus message is received for the message (indicating that the message is no longer tracked) The `amps-action-on-message-condition-timeout` action monitors a SOW topic for messages that match a filter and triggers an action when: * The message has remained matched on the filter for at least the specified duration (indicating a timeout) Use `amps-action-on-message-condition-timeout` if only timeouts are important, otherwise use `amps-action-on-alert` if both timeouts and message tracking are important. For example, a set of actions might be configured to publish a message to an `Alerts` topic if an order is unprocessed for more than a specified timeout. When the order is processed, the actions might publish a message to a `Statistics` topic for tracking purposes. Rather than creating two independent actions, which could lead to overlap if the order is processed at the same time that the alert is being generated, use this action to guarantee that only one of the events will be generated. The module tracks each message that matches the filter individually, and will run once for each message tracked. This module uses the Out-of-Focus notification (OOF) mechanism. When a message matches the specified topic and filter, the module begins tracking that message. If no OOF notification is received for that message within the specified timeout, the action runs for that message and indicates that the timeout has been exceeded. If an OOF notification is received before the timeout expires, then the action runs for that message, indicating that the message is no longer tracked. This module uses the custom event system as a way of indicating whether a given message was removed from tracking, or whether the message timed out. When configuring the action, you specify the event to raise for each of these states. This module stores the exact event raised in a context variable, which you can then use with `amps-action-do-event` to raise the event. :::info While the AMPS server is running, this action will trigger exactly once for each message after it reaches the timeout period. When AMPS restarts, if a message that had previously triggered this action still exists in the SOW topic (and still matches the filter provided, if any), the action will run for that message immediately after the module initializes on restart. ::: This action requires the following parameters. In addition to that, this module also accepts the optional parameters listed below: | Parameter | Description | | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Topic`

    (required)

    |

    The name of the topic to monitor for messages.

    This parameter does not support regular expressions. The topic name must be either a SOW topic, a view or a conflated topic.

    Queues are not supported.

    There is no default for this parameter.

    | |

    `MessageType`

    (required)

    |

    The message type of the topic to monitor for messages.

    There is no default for this parameter.

    | |

    `OOFEvent`

    (required)

    |

    The value to store when an out-of-focus notification is received for a tracked message.

    There is no default for this parameter.

    | |

    `TimeoutEvent`

    (required)

    |

    The value to store when a message is in the tracked state for longer than the timeout period.

    There is no default for this parameter.

    | |

    `EventVariable`

    (required)

    |

    The name of the context variable in which to store the reason that the action is running (either the string in `OOFEvent` or the string in `TimeoutEvent`).

    There is no default for this parameter.

    | | `Duration` | The amount of time to wait for an OOF notification for the message before running the action. | | `Filter` |

    Sets the filter to apply.

    Only messages that match this filter will be monitored by this action. If no filter is provided, every message of the specified message type in topics that match the `Topic` value will be monitored.

    | The module adds the following variables to the AMPS context: | Variable | Description | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AMPS_TOPIC` | The topic of the message that triggered the alert. | | `AMPS_DATA` | The current data of the message. | | `AMPS_DATA_LENGTH` | The length of the current data of the message, in bytes. | | `AMPS_BOOKMARK` | The bookmark of the message. Empty if there is no bookmark for the message. | | `AMPS_TIMESTAMP` | The timestamp at which the module began tracking the message. | | `AMPS_CLIENT_NAME` | The client name of the current value of the message. | | `AMPS_SOW_KEY` | The current SowKey for the message. | | The variable specified in `EventVariable` | The reason the action is running. This will be either the value of `OOFEvent` or the value of `TimeoutEvent`, depending on what caused the action to run. | --- # On Message Condition Timeout AMPS provides a module to run an action when a message in a SOW topic meets a specific condition for longer than a specified period of time. For example, an action might be configured to publish a message to an `Alerts` topic if an order is unprocessed for more than a specified timeout. AMPS also provides a module, `amps-action-on-alert`, that runs when _either_ a message has been in a specific condition for a longer than expected period of time, or the module receives an out-of-focus notification indicating that the message no longer meets the tracking condition. Use that module if both conditions are important. Use this module if it is only important to track timeouts. The `amps-action-on-message-condition-timeout` monitors a SOW topic for messages that match a filter and triggers an action for each message that remains matched on that filter for at least the specified duration. This module uses the Out-of-Focus notification (OOF) mechanism. When a message matches the specified topic and filter, the module begins tracking that message. If no OOF notification is received for that message within the specified timeout, the action runs for that message. The module tracks each message that matches the filter individually, and will run once for each message that exceeds the timeout. :::info While the AMPS server is running, this action will trigger exactly once for each message after it reaches the timeout period. When AMPS restarts, if a message that had previously triggered this action still exists in the SOW topic (and still matches the filter provided, if any), the action will run for that message immediately after the module initializes on restart. ::: This action requires the following parameters. In addition to that, this module also accepts the optional parameters listed below: | Parameter | Description | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Topic`

    (required)

    |

    The name of the topic to monitor for messages.

    This parameter does not support regular expressions. The topic name must be either a SOW topic, a view or a conflated topic.

    Queues are not supported.

    There is no default for this parameter.

    | |

    `MessageType`

    (required)

    |

    The message type of the topic to monitor for messages.

    There is no default for this parameter.

    | | `Duration` | The amount of time to wait for an OOF notification for the message before running the action. | | `Filter` |

    Sets the filter to apply.

    Only messages that match this filter will be monitored by this action. If no filter is provided, every message of the specified message type in topics that match the `Topic` value will be monitored.

    | The module adds the following variables to the AMPS context: | Variable | Description | | ------------------ | --------------------------------------------------------------------------- | | `AMPS_TOPIC` | The topic of the message that triggered the alert. | | `AMPS_DATA` | The current data of the message. | | `AMPS_DATA_LENGTH` | The length of the current data of the message, in bytes. | | `AMPS_BOOKMARK` | The bookmark of the message. Empty if there is no bookmark for the message. | | `AMPS_TIMESTAMP` | The timestamp at which the module began tracking the message. | | `AMPS_CLIENT_NAME` | The client name of the current value of the message. | | `AMPS_SOW_KEY` | The current SowKey for the message. | --- # On OOF Message When a record that previously matched a subscription has been updated so that the record no longer matches its subscription, AMPS sends an out-of-focus (OOF) message to let subscribers know that their record no longer matches the subscription. With `amps-action-on-oof-message`, you can enter a subscription within AMPS and run actions when an OOF message for that subscription is produced. :::warning This action is treated by the AMPS engine as a subscription from an internal AMPS client. _Do not_ use this action with queue topics. Since this action creates a subscription, using this action with the queue topic will cause the action to lease messages from the queue even though the action does _not_ acknowledge messages. This means that, when used with the queue topic itself, the action will interfere with other subscribers and, depending on the queue configuration, may only receive one message during the lifetime of the instance. Furthermore, because each publish to a queue topic is treated as a distinct message, a subscription to a queue topic will never produce `oof` messages. ::: This module requires the `Topic` and the `MessageType` of the OOF message. In addition to that, this module also accepts the optional parameters listed below: | Parameter | Description | | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Topic`

    (required)

    |

    The topic to monitor for OOF messages.

    The topic specified must be a SOW topic, view or conflated topic. This parameter supports regular expressions.

    There is no default for this parameter.

    This module should not be used with queue topics.

    | |

    `MessageType`

    (required)

    |

    The message type of the topic to monitor for OOF messages.

    This parameter supports regular expressions.

    There is no default for this parameter.

    | | `Filter` |

    Set the filter to apply.

    This filter forms the internal subscription for which OOF messages will be generated.

    | | `Type` |

    The type of OOF message to take action on:

    • `match` - take action on OOF messages generated because message no longer matches filter.
    • `delete` - take action on OOF messages generated because message has been removed from the SOW.
    • `expire` - take action on OOF messages generated because the message expired from the SOW.
    • `all` - take action on all of the above types.

    Default: `all`

    | This module adds the following variables to the AMPS context: | Variable | Description | | --------------------------- | -------------------------------------------------------------------- | | `AMPS_TOPIC` | The topic of the OOF message. | | `AMPS_DATA` | The data of the OOF message. | | `AMPS_DATA_LENGTH` | The length of the data of the OOF message. | | `AMPS_PREVIOUS_DATA` | The data previously contained from the updated record. | | `AMPS_PREVIOUS_DATA_LENGTH` | The length of the data previously contained from the updated record. | --- # On Outgoing Replication Connections AMPS includes a set of modules that allow you to configure an action to run based on events for outgoing replication connections. The `amps-action-on-connect-replication` module runs when an outgoing `Destination` is connected. The `amps-action-on-disconnect-replication` module runs when an outgoing `Destination` is disconnected. The `amps-action-on-replication-resync-complete` module runs when a `Destination` has been brought up to date with the transaction log of the local instance. The `amps-action-on-upgrade-replication` module runs when an outgoing replication connection that has been previously downgraded to acknowledge messages `async` is being upgraded to acknowledge messages `sync`. The `amps-action-on-downgrade-replication` module runs when an outgoing replication connection that is configured to acknowledge messages `sync` is being downgraded to acknowledge messages `async`. These modules require no parameters. These modules add the following variables to the AMPS context: |Variable |Description | |--------------------------------------------|-------------------------------------------------------------------------------------------| |`AMPS_REPLICATION_PEER_NAME` |The instance name of the AMPS instance on the opposite end of the connection, if available.| |`AMPS_REPLICATION_CLIENT_NAME` |The name of the AMPS client used for this connection.| |`AMPS_REPLICATION_REMOTE_ADDRESS`|The remote address of the opposite end of the connection.| |`AMPS_REPLICATION_GROUP_NAME` |The group name of the AMPS instance on the opposite end of the connection, if available.| |`AMPS_REPLICATION_TRANSPORT_NAME`|The name of the transport making the outgoing connection.| --- # On a Schedule AMPS provides the `amps-action-on-schedule` module for running actions on a specified schedule. The options provided to the module define the schedule on which AMPS will run the actions in the `Do` element. | Parameter | Description | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Every`

    (required)

    |

    Specifies a recurring action that runs whenever the time matches the provided specification. Specifications can take three forms:

    1. Timer action - a specification that is simply a duration, such as `4h` or `1d`, creates a timer action. AMPS starts the timer when the instance starts. When the timer expires, AMPS runs the action and resets the timer.

    2. Daily action - a specification that is a time of day, such as `00:32` or `17:47`, creates a daily action. AMPS runs the action every day at the specified time. AMPS uses a 24 hour notation for daily actions.

    3. Weekly action - a specification that includes a day of the week and a time, such as `Saturday at 11:00` or `Wednesday at 03:32`, creates a weekly action. AMPS runs the action each week on the day specified, at the time specified. AMPS uses a 24 hour notation for weekly actions.

    AMPS accepts both local time and UTC for time specifications. To use UTC, append a `Z` to the time specifier. For example, the time specification `11:32` is 11:32 AM local time. The time specification `11:32Z` is 11:32 AM UTC.

    | | `Name` |

    The name of the schedule. This name appears in log messages related to this schedule.

    Default: `unknown`

    | This module does not add any variables to the AMPS context. --- # On a Linux Signal AMPS provides the `amps-action-on-signal` module for running actions when AMPS receives a specified signal. The module requires the `Signal` parameter: | Parameter | Description | | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `Signal`

    (required)

    |

    Specifies the signal to respond to. This module supports the standard Linux signals. Configuring an action uses the standard name of the signal.


    For example, to configure an action to `SIGUSR1`, the value for the `Signal` element is `SIGUSR1`. To configure an action for `SIGHUP`, the value for the `Signal` element is `SIGHUP` and so on.

    AMPS reserves `SIGQUIT` for producing minidumps and does not allow this module to override `SIGQUIT`. AMPS registers actions for several signals by default, as described in the [following table](on-signal#default-signal-actions).

    | This module does not add any variables to the AMPS context. :::warning Actions can be used to override the default signal behavior for AMPS ::: ### Default Signal Actions By default, AMPS registers the following actions for signals: | On Event | Action | | --------- | --------------------------------------- | | `SIGUSR1` | `amps-action-do-disable-authentication` | | `SIGUSR1` | `amps-action-do-disable-entitlement` | | `SIGUSR2` | `amps-action-do-enable-authentication` | | `SIGUSR2` | `amps-action-do-enable-entitlement` | | `SIGINT` | `amps-action-do-shutdown` | | `SIGTERM` | `amps-action-do-shutdown` | | `SIGHUP` | `amps-action-do-shutdown` | The actions in the table above can be overridden by creating an explicit action in the configuration file. AMPS reserves the `SIGQUIT` signal and does not allow the configuration file to override the action taken in response to `SIGQUIT`. | On Event | Action | | --------- | ------------------------- | | `SIGQUIT` | `amps-action-do-minidump` | --- # On AMPS Startup or Shutdown AMPS includes modules to run actions when AMPS starts up or shuts down. The `amps-action-on-startup` module runs actions as the last step in the startup sequence. The `amps-action-on-shutdown` module runs actions as the first step in the AMPS shutdown sequence. In both cases, actions run in the order that the actions appear in the configuration file. These modules do not require any parameters. These modules do not add any variables to the AMPS context. --- # On Subscribe or Unsubscribe AMPS provides modules for running actions when a client subscribes or unsubscribes. The `amps-action-on-subscribe` runs actions when an AMPS client enters a subscription command. The `amps-action-on-unsubscribe` runs actions when an AMPS client unsubscribes (either by sending an explicit unsubscribe command or by disconnecting). These modules require the following parameters: | Parameter | Description | | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | |

    `Topic`

    (required)

    | Specifies the topic or topic regular expression pattern to monitor for subscribe commands. | |

    `MessageType`

    (required)

    | The message type of the topic to monitor for subscribe or unsubscribe. There is no default for this parameter. | These modules add the following variables to the AMPS context: | Variable | Description | | ------------------ | ------------------------------------------------------------------------ | | `AMPS_TOPIC` | The name of the topic specified by the subscribe or unsubscribe command. | | `AMPS_CLIENT_NAME` | The name of the client submitting the command. | | `AMPS_OPTIONS` | The options on the subscription. | | `AMPS_FILTER` | The filter for the subscription. | --- # Configuring AMPS for Automation with Actions AMPS provides the ability to run scheduled tasks or respond to events, such as Linux signals, using the Actions interface. To create an action, you add an `Actions` section to the AMPS configuration file. Each `Action` contains one (or more) `On` statement which specifies when the action occurs, and one (or more) `Do` statement which specifies what the AMPS server does for the action. Within an action, AMPS performs each `Do` statement in the order in which they appear in the file. AMPS actions may require the use of parameters. AMPS allows you to use variables in the parameters of an action. You can access these variables using the following syntax: `{{VARIABLE_NAME}}` AMPS defines a set of default variables when running an action. The event, or a previous action, can add variables in the context of the action. Those variables can be expanded in subsequent parameters. If a variable is used that isn't defined at the point where it is used, AMPS will expand that variable to an empty string literal. The context can also be updated as the module is running, so any variables that are available at any given point in the file depend on what action was previously executed. By default, AMPS loads the following variables when it initializes an AMPS action: | Variable | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `AMPS_INSTANCE_NAME` | The name of the AMPS instance. | | `AMPS_BYTE_XX` |

    Insert byte XX, where XX is a 2-digit uppercase hex number (00-FF). AMPS expands this variable to the corresponding byte value.

    These variables are useful for creating field separators or producing characters that are not permitted within XML.

    | | `AMPS_DATETIME` | The current date and time in ISO-8601 format. | | `AMPS_UNIX_TIMESTAMP` | The current date and time as a UNIX timestamp. | An example to echo a message when AMPS starts up is shown below. Note the AMPS\_INSTANCE\_NAME is one of the variables that AMPS pushes to the context when an action is loaded. ```xml showLineNumbers amps-action-on-startup amps-action-do-echo-message instance={{AMPS_INSTANCE_NAME}} ``` AMPS actions are implemented as AMPS modules. To run each statement, AMPS simply calls the module that implements that `Do` statement. The module is free to take any necessary actions. If a `Do` statement returns a failure, AMPS does not run subsequent `Do` statements in that action. This is intended to help make maintenance processes reliable. For example, if a `Do` statement that is intended to copy a log file to a storage directory fails because the device that holds the storage directory is not mounted, further steps in the action -- which might do things like remove the log file from the original directory -- should not be run. Likewise, if AMPS exits unexpectedly during a given `Do` statement, subsequent statements will not be run. AMPS provides the following modules by default: ### On: Choosing When an Action Runs The following table lists the `On` actions that are provided in AMPS by default. Details for each action are provided in the section describing that action. | Condition | Modules | | -------------------------------------------------------------------------------------------------------| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [On a Schedule](/docs/amps-user-guide/actions/on-elements/on-schedule) | `amps-action-on-schedule` | | [On AMPS Startup or Shutdown](/docs/amps-user-guide/actions/on-elements/on-startup-shutdown) |

    `amps-action-on-startup`
    `amps-action-on-shutdown`

    | | [On a Linux Signal](/docs/amps-user-guide/actions/on-elements/on-signal) | `amps-action-on-signal` | | [On a REST Request](/docs/amps-user-guide/actions/on-elements/on-http) | `amps-action-on-admin` | | [On Minidump Creation](/docs/amps-user-guide/actions/on-elements/on-minidump) | `amps-action-on-minidump` | | [On Client Connect or Disconnect](/docs/amps-user-guide/actions/on-elements/on-client-connect) |

    `amps-action-on-connect-client`
    `amps-action-on-disconnect-client`

    | | [On Client Logon](/docs/amps-user-guide/actions/on-elements/on-logon) | `amps-action-on-logon-client` | | [On Client Offline Message Buffering](/docs/amps-user-guide/actions/on-elements/on-client-offline) |

    `amps-action-on-offline-start`
    `amps-action-on-offline-stop`

    | | [On Subscribe or Unsubscribe](/docs/amps-user-guide/actions/on-elements/on-sub-unsub) |

    `amps-action-on-subscribe`
    `amps-action-on-unsubscribe`

    | | [On Incoming Replication Connections](/docs/amps-user-guide/actions/on-elements/on-incoming-replication) |

    `amps-action-on-connect-incoming-replication`
    `amps-action-on-disconnect-incoming-replication`

    | | [On Outgoing Replication Connections](/docs/amps-user-guide/actions/on-elements/on-outgoing-replication) |

    `amps-action-on-connect-replication`
    `amps-action-on-disconnect-replication`

    `amps-action-on-replication-resync-complete`

    `amps-action-on-upgrade-replication`
    `amps-action-on-downgrade-replication`

    | | [On Message Published to AMPS](/docs/amps-user-guide/actions/on-elements/on-message-published) | `amps-action-on-publish-message` | | [On Message Delivered to Subscriber](/docs/amps-user-guide/actions/on-elements/on-message-delivered) | `amps-action-on-deliver-message` | | [On Message Affinity](/docs/amps-user-guide/actions/on-elements/on-message-affinity) | `amps-action-on-message-affinity` | | [On SOW Message Expiration](/docs/amps-user-guide/actions/on-elements/on-expire) | `amps-action-on-sow-expire-message` | | [On SOW Message Delete](/docs/amps-user-guide/actions/on-elements/on-delete) | `amps-action-on-sow-delete-message` | | [On Out-of-Focus Message](/docs/amps-user-guide/actions/on-elements/on-oof) | `amps-action-on-oof-message` | | [On Message Condition Timeout](/docs/amps-user-guide/actions/on-elements/on-msg-timeout) |

    `amps-action-on-message-condition-timeout`

    | | [On Message State Change](/docs/amps-user-guide/actions/on-elements/on-msg-state) | `amps-action-on-alert` | | [On Custom Event](/docs/amps-user-guide/actions/on-elements/on-event) | `amps-action-on-execute-event` | ### Do: Choosing What an Action Does The following table lists the `Do` actions that are provided in AMPS by default. Details for each action are provided in the section describing that action. | Action | Module | | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [Rotate Error/Event Log](/docs/amps-user-guide/actions/do-elements/do-rotate-log-files) | `amps-action-do-rotate-logs` | | [Compress Files](/docs/amps-user-guide/actions/do-elements/do-compress-files) | `amps-action-do-compress-files` | | [Truncate Statistics](/docs/amps-user-guide/actions/do-elements/do-manage-stats) | `amps-action-do-truncate-statistics` | | [Manage Transaction Log Journal Files](/docs/amps-user-guide/actions/do-elements/do-manage-journal) |

    `amps-action-do-archive-journal`
    `amps-action-do-compress-journal`
    `amps-action-do-remove-journal`

    | | [Remove Error/Event Log Files](/docs/amps-user-guide/actions/do-elements/do-delete-files) | `amps-action-do-remove-files` | | [Delete SOW Messages](/docs/amps-user-guide/actions/do-elements/do-sow-delete) | `amps-action-do-delete-sow` | | [Compact SOW Topics](/docs/amps-user-guide/actions/do-elements/do-sow-compact) | `amps-action-do-compact-sow` | | [Query a SOW Topic](/docs/amps-user-guide/actions/do-elements/do-query-sow) | `amps-action-do-query-sow` | | [Manage Security](/docs/amps-user-guide/actions/do-elements/do-manage-security) |

    `amps-action-do-disable-authentication`
    `amps-action-do-enable-authentication`

    `amps-action-do-reset-authentication`
    `amps-action-do-reset-entitlement`


    `amps-action-do-disable-entitlement`
    `amps-action-do-enable-entitlement`

    | | [Enable or Disable Transports](/docs/amps-user-guide/actions/do-elements/do-enable-disable-transport) |

    `amps-action-do-enable-transport`
    `amps-action-do-disable-transport`

    | | [Publish Message](/docs/amps-user-guide/actions/do-elements/do-publish-message) | `amps-action-do-publish-message` | | [Manage Replication Acknowledgment ](/docs/amps-user-guide/actions/do-elements/do-replication-upgrade-downgrade) |

    `amps-action-do-downgrade-replication`
    `amps-action-do-upgrade-replication`

    | | [Extract Values from a Message](/docs/amps-user-guide/actions/do-elements/do-extract-values) | `amps-action-do-extract-values` | | [Translate Data Within an Action](/docs/amps-user-guide/actions/do-elements/do-translate-data) | `amps-action-do-translate-data` | | [Increment Counter](/docs/amps-user-guide/actions/do-elements/do-increment-counter) | `amps-action-do-increment-counter` | | [Raise a Custom Event](/docs/amps-user-guide/actions/do-elements/do-raise-custom-event) | `amps-action-do-execute-event` | | [Execute System Command](/docs/amps-user-guide/actions/do-elements/do-execute-system) | `amps-action-do-execute-system` | | [Manage Queue Transfers](/docs/amps-user-guide/actions/do-elements/do-manage-queue-transfer) |

    `amps-action-do-enable-proxied-transfer`
    `amps-action-do-disable-proxied-transfer`

    | | [Create a Minidump](/docs/amps-user-guide/actions/do-elements/do-create-minidump) | `amps-action-do-minidump` | | [Shutdown AMPS](/docs/amps-user-guide/actions/do-elements/do-shutdown) | `amps-action-do-shutdown` | | [Debug Action Configurations](/docs/amps-user-guide/actions/do-elements/do-debug-actions) |

    `amps-action-do-nothing`

    `amps-action-do-echo-message`

    | ### Conditionally Stop an Action The following table lists the `If` actions that are provided in AMPS by default. Details for each action are provided in the section describing that action. | Condition | Module | | ---------------------------------------------------------------------------------------------- | ---------------------------------- | | [Stop Based on File System Capacity](/docs/amps-user-guide/actions/if-elements/if-file-system) | `amps-action-if-file-system-usage` | | [Stop Based on Evaluating an Expression](/docs/amps-user-guide/actions/if-elements/if-condition) | `amps-action-if-condition` | ### Examples of Action Configuration The following table lists fully-configured action examples, demonstrating the elements outlined above. | Scenario | | ---------------------------------------------------------------------------------------------------------------------- | | [Archive Journals Once a Week](/docs/amps-user-guide/action-examples/archive-each-week) | | [Archive Journals On RESTful Command](/docs/amps-user-guide/action-examples/archive-on-demand) | | [Record Expired Queue Messages to a Dead Letter Topic](/docs/amps-user-guide/action-examples/dead-letter-queue) | | [Copy Messages that Exceed a Timeout to a Different Topic](/docs/amps-user-guide/action-examples/publish-on-timeout) | | [Deactivate and Reactivate Security on Signals](/docs/amps-user-guide/action-examples/toggle-security-on-signal) | | [Reset Entitlements for a Disconnected Client](/docs/amps-user-guide/action-examples/reset-entitlement-on-disconnect) | | [Extract Values from a Published Message](/docs/amps-user-guide/action-examples/extract-values-from-message) | | [Shut Down AMPS When a Filesystem Is Full](/docs/amps-user-guide/action-examples/shutdown-on-full-filesystem) | | [Increment a Counter and Echo a Message](/docs/amps-user-guide/action-examples/increment-and-echo) | --- # AMPS Data Types Each value in AMPS is assigned a data type when the message type module parses the value. AMPS operators and functions attempt to convert values into compatible types, based on the type of operation. For example, the `*` operator (multiplication) will attempt to convert all values to numeric values, while the `CONCAT` function (string concatenation) will attempt to convert all values to strings. In effect, a value in AMPS can be transparently treated as any type to which it can be meaningfully converted. Internally, AMPS uses the data types in the table below. As mentioned above, the message type module is responsible for assigning the type of a value from an incoming message as part of the parsing process. For some types, such as JSON, XML, FIX and NVFIX, the parser infers the type of the value from the field. For other types, such as MessagePack, BFLAT, Google Protocol Buffers or BSON, the message itself contains information about the type of the field. As mentioned above, the AMPS expression language does not limit the value to the type assigned by the message type module. Instead, a value in AMPS can be used in any context. For example, given the following JSON document: ```json {"a":1,"b":"47"} ``` The values of `/a` and `/b` can be used as either string values or numeric values. AMPS will automatically convert these values as necessary, and AMPS considers the string or numeric representation to be equally correct and valid. The following table lists the data types in the AMPS expression language: | Type | Description | Untyped Message Examples | | --------------------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | NULL | Unknown, untyped value (SQL-92 semantics) |

    [no field provided]

    NVFIX: `a=`

    JSON: `{"a":null}`

    XML: ``

    | | Boolean | True (`1`) or false (`0`) | JSON: `{"e":true}` | | Integer | Signed 64-bit integer or unsigned 64-bit integer for values > LONG\_MAX |

    NVFIX: `b=24`

    JSON: `{"b":24}`

    XML: `24`

    | | Floating Point Number | 64-bit floating point number |

    NVFIX: `c=24.0`

    JSON: `{"c":24.0}`

    XML: `24.0`

    | | String |

    Arbitrary sequence of bytes of a specific length

    An empty string is considered to be `NULL`

    |

    NVFIX: `d=Grilled cheese sandwich`

    JSON: `{"d":"Grilled cheese sandwich"}`

    XML: `Grilled cheese sandwich`

    | ## Numeric Types and Literals in AMPS Expressions Numeric values in AMPS are always _typed_ as either integers or floating point values. All numeric types that are less than or equal to the `LONG_MAX` limit in AMPS are signed, otherwise, the numeric type is unsigned. AMPS message types convert the original numeric types (or original representation for message types that do not have typed values) into the internal AMPS type system for the purposes of expression evaluation. Within expressions, integer values are all numerals, with no decimal point, and can have a value in the same range as a 64-bit integer. For example: ``` 42 149 -273 18446744073709551610 ``` Within expressions, all numerals with a decimal point are floating-point numbers. AMPS interprets these numerals as double-precision floating point values. For example: ``` 3.1415926535 98.6 -273.0 ``` or, in scientific notation: ``` 31.4e-1 6.022E23 2.998e8 ``` AMPS automatically converts strings that contain numeric values to numbers when strings are used with an operator, function or comparison that expects a numeric value. ## Type Promotion for Numeric Types AMPS uses the following rules for type promotion when evaluating numeric expressions: 1. If any of the values in the expression is `NaN`, the result is `NaN`. 2. Otherwise, if any of the values in the expression is floating point, the result is floating point. 3. Otherwise, all of the values in the expression are integers, and the result is an integer. Notice that, for division in particular, the results returned are affected by the type of the values. For example, the expression `1 / 5` evaluates to `0` since the result is interpreted as an integer. In comparison, the expression `1.0 / 5` evaluates to `0.2` since the result is interpreted as a floating point value. When a function or operator that expects a numeric type is provided with a string, AMPS will attempt to convert string values to numeric types as necessary. When converting string values, AMPS recognizes the same numeric formats in message data as are supported in the AMPS expression language (see [Numeric Types and Literals](amps-data-types.md#numeric-types-and-literals-in-amps-expressions)). If the string is in an unrecognized format, AMPS converts the string as `NaN`. ## String Literals in AMPS Expressions When creating expressions for AMPS, string literals are indicated with single or double quotes. For example: ```sql /FIXML/Order/Instrmt/@Sym = 'IBM' ``` AMPS supports the following escape sequences within string literals: | Escape Sequence | Definition | | ------------------- | --------------------------------------------- | | \a | Alert | | \b | Backspace | | \t | Horizontal tab | | \n | Newline | | \f | Form feed | | \r | Carriage return | | \xHH | Hexadecimal digit where H is (0..9,a..f,A..F) | | \OOO | Octal Digit (0..7) | Additionally, any character which follows a backslash will be treated as a literal character. AMPS string operations have no restrictions on character set, and correctly handle embedded `NULL` characters (`\x00`) and characters outside of the 7-bit ASCII range. AMPS string operations are not unicode-aware. ## NULL, NaN and IS NULL XPath expressions are considered to be `NULL` when they evaluate to an empty or nonexistent field reference. `NULL` values follow SQL-92 semantics. This means that comparisons with `NULL` are never true (in other words, even if `/a` is `NULL`, `/a != NULL` is false and `/a == NULL` is also false). :::tip AMPS considers a zero-length string to be NULL. ::: In numeric expressions where the operands or results are not a valid number, the XPath expression evaluates to `NaN` (not a number). The rules for applying the `AND` and `OR` operators against `NULL` and `NaN` values are outlined in the tables below: | Operand1 | | Operand2 | Result | | ------------ | ----- | ------------ | ---------- | | TRUE | (AND) | NULL | NULL | | FALSE | (AND) | NULL | FALSE | | NULL | (AND) | NULL | NULL | | NULL | (AND) | TRUE | NULL | | NULL | (AND) | FALSE | NULL | | Operand1 | | Operand2 | Result | | ------------ | ---- | ------------ | ---------- | | TRUE | (OR) | NULL | TRUE | | FALSE | (OR) | NULL | NULL | | NULL | (OR) | NULL | NULL | | NULL | (OR) | TRUE | NULL | | NULL | (OR) | FALSE | NULL | Likewise, direct comparisons with `NULL` are not ever true (so, if `/b` is NULL, `/b == NULL` does not produce a true value, and neither does `/b != NULL`). AMPS, like SQL-92, provides an `IS NULL` predicate for testing whether a value is `NULL`, and an `IS NOT NULL` predicate for testing whether a value is not `NULL`. There also exists an `IS NAN` predicate for checking that a value is `NaN` (not a number.) :::warning To reliably check for existence of a `NULL` value, you must use the `IS NULL` predicate such as the filter: `/optionalField IS NULL` To reliably check that a value is not `NULL`, you must use the `IS NOT NULL` predicate or negate the value of an `IS NULL` test: `/optionalField IS NOT NULL` and `NOT /optionalField IS NULL` are equivalent. ::: AMPS also provides a `COALESCE()` function that accepts a set of values and returns the first value that is not NULL. For example, given the following filter expression: ```sql COALESCE(/userCategory, /employeeCategory, /vendorCategory, 'restricted') != 'restricted' ``` AMPS will return the first value that is not `NULL`, and compare that value to the constant string `'restricted'`. Notice that, to make the intent of the filter clear, this example provides a constant value for AMPS to return from the `COALESCE` if all of the field values are `NULL`. The `COALESCE` function, like other functions in AMPS, is not array-aware. This means that when one of the XPath expressions provided to `COALESCE` specifies an array in the original message, AMPS provides the _first item in the array_ to the `COALESCE` function. See [Working With Arrays](working-with-arrays) for details. ## Compound Types in AMPS Many messaging applications are designed for high performance and use a simplified message structure. For applications that use compound types, AMPS includes the ability to parse and filter on the contents of nested data structures. For performance, AMPS parses nested data structures into a set of values. As with single-valued (or scalar) values, the AMPS expression language refers to a parsed set of values that is common to all message types rather than the underlying data. The AMPS message types treat compound data types as a set of paths with corresponding scalar values. A field that only contains other fields is represented as a step in the path to the primitive values that it contains. AMPS parses compound types as follows: * Any field that contains a scalar value is represented as an identifier/value pair. * Any field that contains other fields is represented as a step in the path to that value. * Multiple values with identical paths are represented as an array. For more information on arrays in the AMPS expression language, see [Working With Arrays](working-with-arrays). The following JSON document is a simple example. ```json {"outer": {"middle": { "inner": 5 } } } ``` With this document, AMPS produces the following parsed value: | Path | Value | | --------------------- | ----- | | `/outer/middle/inner` | 5 | In the parsed representation, the `outer` and `middle` fields contain no data of their own. They serve only as containers for the `inner` field which contains data. Notice that the intermediate paths do not have an explicit scalar value. With a more complex document the parsed representation continues to follow the same principles, as shown in the following example. ```js showLineNumbers {"outer" : { "array" : ["a1", "a2", "a3"], "compound" : { "A" : "middle-A", "B" : "middle-B", "C" : [ {"C1":"first-C1", "D1":"first-D1"}, {"C1":"second-C1","D1":"second-D1"} ] } } } ``` The representation of the above message in the AMPS expression language would typically be as follows: | Path | Value | Notes | | ---------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `/outer/array` | `['a1', 'a2', 'a3]` |

    Elements in the array can be referred to directly with subscript notation.

    For example: `/outer/array[0]` is `'a1'`.

    | | `/outer/compound/A` | `'middle-A'` | | | `/outer/compound/B` | `'middle-B'` | | | `/outer/compound/C/C1` | `['first-C1', 'second-C1']` |

    Elements in the array can be referred to directly with subscript notation.

    For example: `/outer/compound/C/C1[0]` is `'first-C1'`.

    | | `/outer/compound/C/D1` | `['first-D1', 'second-D1']` |

    Elements in the array can be referred to directly with subscript notation.

    For example: `/outer/compound/C/D1[0]` is `'first-D1'`.

    | As with the first example, fields that do not directly contain a value do not have an explicit scalar value. Values with the same identifier are represented as an array of values with that identifier. --- # Arithmetic Operators AMPS supports the arithmetic operators `+`, `-`, `*`, `/`, `%`, and `MOD` in expressions. The result of arithmetic operators where one of the operands is `NULL` is undefined and evaluates to `NULL`. AMPS distinguishes between floating point and integral types. When an arithmetic operator uses two different types, AMPS will convert the integral type to a floating point value as described in [Numeric Types and Literals](amps-data-types.md#numeric-types-and-literals-in-amps-expressions). Examples of filter expressions using arithmetic operators: ```sql /6 * /14 < 1000 /Order/@Qty * /Order/@Prc >= 1000000 ``` AMPS numeric types are signed, and the AMPS arithmetic operators correctly handle negative numbers. The `MOD` and `%` operators preserve the sign of the first argument to the operator. That is, `-5 % 3` produces a result of `-2`, while `5 % -3` produces a result of `2`. :::warning When using mathematical operators in conjunction with filters, be careful about the placement of the operator. Some operators are used in the XPath expression as well as for mathematical operation (for example, the `'/'` operator in division). Therefore, it is important to separate mathematical operators with white space to prevent interpretation as an XPath expression. ::: --- # Comparison Operators The comparison operators can be loosely grouped into equality comparisons and range comparisons. The basic equality comparison operators, in precedence order, are `==`, `=`, `>`, `>=`, `<`, `<=`, `!=`, and `<>`. The `==` comparison and the `=` comparison are treated as the same operator and produce the same results. The `!=` and `<>` comparisons are also treated as the same operator and produce the same result. If these binary operators are applied to two operands of different types, AMPS attempts to convert strings to numbers. If conversion succeeds, AMPS uses the numeric values. If conversion fails because the string cannot be meaningfully converted to a number, strings are always considered to be greater than numbers. The operators consider an empty string to be `NULL`. The following table shows some examples of how AMPS compares different types. | Expression | Result | | -------------------- | ----------------------------------------------------- | | `1 < 2` | TRUE | | `10 < '2'` | FALSE, '2' can be converted to a number | | `'2.000' <> '2.0'` | TRUE, no conversion to numbers since both are strings | | `2 = 2.0` | TRUE, numeric comparison | | `10 < 'Crank It Up'` | TRUE, strings are greater than numbers | | `10 < ''` | FALSE, an empty string is considered to be NULL | | `10 > ''` | FALSE, an empty string is considered to be NULL | | `'' = ''` | FALSE, an empty string is considered to be NULL | | `'' IS NULL` | TRUE, an empty string is considered to be NULL | There are also set and range comparison operators. The `BETWEEN` operator can be used to check the range values. :::tip The range used in the `BETWEEN` operator is inclusive of both operands, meaning the expression `/A BETWEEN 0 AND 100` is equivalent to `/A >= 0 AND /A <= 100.` ::: For example: ```sql /FIXML/Order/OrdQty/@Qty BETWEEN 0 AND 10000 /FIXML/Order/@Px NOT BETWEEN 90.0 AND 90.5 (/price * /qty) BETWEEN 0 AND 100000 ``` The `IN` operator can be used to perform membership operations on sets of values. The `IN` operator returns true when the value on the left of the `IN` appears in the set of values in the `IN` clause. For example: ```sql /Trade/OwnerID NOT IN ('JMB', 'BLH', 'CJB') /21964 IN (/14*5, /6*/14, 1000, 2000) /customer IN ('Bob', 'Phil', 'Brent') ``` The `IN` operator returns true for the set of records that would be returned by an equivalent set of `=` comparisons joined by `OR`. The following two statements return the same set of records: ```sql /pet IN ('puppy', 'kitten', 'goldfish') ``` ```sql (/pet = 'puppy') OR (/pet = 'kitten') OR (/pet = 'goldfish) ``` This equivalence means that `NULL` values in either the field being evaluated, or the set of values provided to the `IN` clause, always return false. This also means that, for string values, the `IN` operator performs exact, case-sensitive matching. When using `NOT IN`, AMPS interprets this as a `NOT` unary operator applied to the `IN` operator. This means that the following expressions are equivalent: ```sql /data NOT IN (1,2,3) NOT /data IN (1,2,3) NOT ((/data == 1) OR (/data == 2) OR (/data == 3)) ``` :::tip When evaluating against a set of values, the `IN` operator typically provides better performance than using a set of `OR` operators. That is, a filter written as `/firstName IN ('Joe', 'Kathleen', 'Frank', 'Cindy', 'Mortimer')` will typically perform better than an equivalent filter written as `/firstName = 'Joe' OR /firstName = 'Kathleen' OR /firstName = 'Frank' OR /firstName = 'Cindy' OR /firstName = 'Mortimer'`. ::: --- # Conditional Operators AMPS contains support for a ternary conditional `IF` operator which allows for a Boolean condition to be evaluated to `true` or `false`, and will return one of the two parameters. The general format of the `IF` statement is: ```sql IF (BOOLEAN_CONDITIONAL, VALUE_TRUE, VALUE_FALSE) ``` In this example, the `BOOLEAN_CONDITIONAL` will be evaluated, and if the result is true, the `VALUE_TRUE` value will be returned otherwise the `VALUE_FALSE` will be returned. | Function or Operator | Parameters | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `IF` |

    Conditional expression

    Value to return if conditional expression is true

    Value to return if conditional expression is false

    |

    Evaluate the conditional expression and return one of the two input values based on the results of the expression.

    The AMPS expression engine can conditionally evaluate the terms provided to the `IF` statement in version 5.3.4 and greater.

    In previous versions of AMPS, all expressions provided to the `IF` statement were fully evaluated before the `IF` statement was evaluated.

    | For example: ```sql SUM( IF(( (/FIXML/Order/OrdQty/@Qty > 500) AND (/FIXML/Order/Instrmt/@Sym ='MSFT')), 1, 0 )) ``` The above example returns a count of the total number of orders that have been placed where the symbol is MSFT and the order contains a quantity more than 500. The `IF` operator can also be used to evaluate results to determine if results are `NULL` or `NaN`. This is useful for calculating aggregates where some values may be `NULL` or `NaN`. The `NULL` and `NaN` values are discussed in more detail in the [AMPS Data Types ](amps-data-types.md#null-nan-and-is-null)section. For example: ```sql SUM(/FIXML/Order/Instrmt/@Qty * IF( /FIXML/Order/Instmt/@Price IS NOT NULL, 1, 0)) ``` --- # Grouping and Order of Evaluation AMPS expressions allow you to group parts of the expression using parentheses. Parts of an expression inside parentheses are evaluated together. 60East recommends using parentheses to group independent parts of an expression to ensure that the expression is evaluated in the expected order. For example, in this expression: ```sql ( /counter % 3 ) == 0 ``` The clause `/counter % 3` is evaluated first, and the result of that evaluation is compared to `0`. Within a group, elements are evaluated left to right in precedence order. For example, given the filter below: ```sql (expression1 OR expression2 AND expression3) OR (expression4 AND NOT expression5) ... ``` AMPS evaluates `expression2`, then `expression3` (since `AND` has higher precedence than `OR`), and if they evaluate to false, then `expression1` will be evaluated. AMPS does not guarantee that all parts of an expression will be evaluated if the result of an expression can be determined after only evaluating part of the expression. For example, given the expression: ```sql A_FUNCTION(/a) OR B_FUNCTION(/b) ``` AMPS only guarantees that `B_FUNCTION(/b)` will be evaluated if `A_FUNCTION(/a)` returns `false`. --- # Identifiers AMPS identifiers use a subset of XPath to specify values in a message. AMPS identifiers specify the value of an attribute or element in an XML message, and the value of a field in a JSON, FIX or NVFIX message. Given that the identifier syntax is only used to specify values, the subset of XPath used by AMPS does not include wildcards, relative paths, array manipulation, predicates or functions. For example, when messages are in this XML format: ```xml showLineNumbers 12345 IBM 1000 ``` The following identifier specifies the `Symbol` element of an `Order` message: ```sql /Order/Symbol ``` The following identifier specifies the `update` attribute of an `Order` message: ```sql /Order/@update ``` For FIX and NVFIX, you specify fields using `/` and the tag name. AMPS interprets FIX and NVFIX messages as though they were an XML fragment with no root element. For example, to specify the value of FIX tag `55` (symbol), use the following identifier: ```sql /55 ``` Likewise, for JSON or other types that represent an object, you navigate through the object structure using the `/` to indicate each level of nesting. AMPS only guarantees support for field identifiers that are valid _step names_ in XPath. For example, AMPS does not guarantee that it can process or filter on a field named `Fits&Starts`. AMPS also supports an optional _bracketed field identifier_ syntax that extends the characters available for field names. For example, the following step name: ```sql [/Not Xpath Name] ``` refers to a field name of `Not Xpath Name` at the root level of the message. This syntax allows spaces to be used in field names in AMPS expressions, even though this is not a valid step name in XPath. Notice that not all message types support field names with embedded spaces or other special characters. For example, the `Not Xpath Name` identifier is not a valid element name in XML, nor would it be a valid field name in Google Protocol Buffers. AMPS checks the syntax of identifiers when parsing an expression. AMPS does not try to predict whether an identifier will match messages within a particular topic. It is not an error to submit an identifier that can never match due to the limitations of the message type. For example, AMPS allows you to use an identifier like `/OrderQty` in a filter submitted for a FIX connection, even though FIX messages only use numeric tags, or an identifier like `/DataPackage/RunDate` in a filter submitted for a BFlat connection, even though BFlat does not support nested elements. The message type is responsible for constructing a set of identifiers from a message. In most cases, the mapping is simple. However, see the documentation for the message type for details, or if the mapping is unclear. For example, a `composite-local` message type adds the number of the part to the beginning of each XPath within the part (so, a top-level field of `/name` in the first part of the message has an identifier of `/0/name`). --- # LIKE Operator AMPS also provides a regular expression comparison operator, `LIKE`, to provide regular expression matching on string values. A _pattern_ is used for the right side of the `LIKE` operator. A pattern must be provided as a literal, quoted value. For more on regular expressions and the `LIKE` comparison operator, please see the section on [Regular Expressions](regular-expressions). The string comparison operators described in the section called [String Comparison Functions](../builtin\_functions/string-comparison-functions) are usually more efficient than equivalent `LIKE` expressions, particularly when used to compare multiple literal patterns, or when the only purpose of the regular expression is to perform case-insensitive matching. Use `LIKE` operations when it is not practical to represent the filter condition with the string comparison operators. | Function or Operator | Parameters | Description | | -------------------- | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `LIKE` |

    The string to be compared

    The pattern to evaluate the string against

    |

    Case-sensitive

    Returns true if the string to be compared matches the pattern.

    For example, the following filter uses a PCRE backreference to return true for any message where the `/state` field contains two identical characters in a row.

    ` /state LIKE '(.)\1' `

    This operator is not unicode-aware.

    | --- # Logical Operators The logical operators are `NOT`, `AND`, and `OR`, in order of precedence. These operators have the usual Boolean logic semantics. ```sql /FIXML/Order/Instrmt/@Sym = 'IBM' OR /FIXML/Order/Instrmt/@Sym = 'MSFT' ``` As with other operators, you can use parentheses to group operators and affect the order of evaluation. ```sql /qty > 100 AND (/job = 'custom' OR /customerType = 'important') ``` --- # Performance Considerations This section describes general performance considerations for the AMPS expression language and content filters. The considerations here are aspects of AMPS performance to be aware of in the general case. However, since the AMPS expression language operates on specific data, the structure and size of the messages that your application uses may have more effect on overall performance than the specific expressions used. For example, parsing and filtering a 20MB XML document is inherently more expensive than parsing and filtering a 400 byte BFlat document. ## Use Short-Circuiting When clauses in an expression are joined by `OR`, AMPS will only evaluate the right side of an `OR` expression if the left side of the expression is false. Similarly, when clauses in an expression are joined by `AND`, AMPS will only evaluate the right side of an `AND` expression if the left side of the expression is true. When constructing an expression, this means that there can be a performance advantage to having relatively less expensive clauses on the left hand sides of the `OR`/`AND`. For example, in the following clause: ```sql /code = 'restricted' OR /notes LIKE 'restricted|limited' ``` The regular expression comparison is only evaluated if the comparison `/code = 'restricted'` is false. If the comparison is true, then the overall clause is true and there is no need to evaluate the regular expression. ## Avoid Redundant Expressions AMPS does not reorder or recombine complex expressions. Where feasible, your application can save work at the server by combining expressions. In particular, if an application is constructing a filter by reading options from various sources, performance can be improved by combining the queries. For example, in a filter like the following: ```sql /id = '12345' OR /id IN ('12345','23456','34567','45678') OR /id IN ('12345','45678','90909') ``` The comparison against `'12345'` will be evaluated three times in cases where the value of `/id` does not match any of the values in the filter. This filter is equivalent to: ```sql /id IN ('12345','23456','34567','45678','90909') ``` The same results are produced, but only evaluates the `/id` field against a given value one time. ## Use Specialized Operators for Simple Comparisons, Use LIKE when Necessary The `LIKE` operator offers access to full Perl-Compatible Regular Expressions within the AMPS expression language. This flexibility allows for very precise filtering, and the PCRE engine performs well. However, for comparisons for which AMPS provides a named function, the named function is highly-optimized and will perform somewhat better than the general-purpose regular expression engine. For example, given a choice between two equivalent expressions: ```sql /state BEGINS WITH('North') ``` and ```sql /state LIKE '^North' ``` The version that uses `BEGINS WITH` will typically perform slightly better than the version that uses the regular expression. This doesn't mean that regular expressions or the `LIKE` operator perform poorly. The `LIKE` operator can efficiently match patterns that would be difficult or impossible to match using the other operators. However, for very simple comparisons where AMPS provides a dedicated operator, that operator typically performs slightly better than a regular expression. The following table shows some examples of regular expressions and the AMPS operator equivalent. | Regular Expression | AMPS Operator Equivalent | | ------------------ | -------------------------------------- | | `^something` | `BEGINS WITH('something')` | | `something$` | `ENDS WITH ('something')` | | `something` | `INSTR(/field, 'something') != 0` | | `(?i)something` | `INSTR_I(/field, 'something') != 0` | | `(?i)^something$` | `STREQUAL_I(/field, 'something') != 0` | | `^a$` | `= 'a'` | ## Optimize for Partial Parsing Most AMPS message types have the ability to _partially parse_ messages. That is, rather than parsing the entire message, the message type can simply find the identifiers that will be used, and stop the parsing process as soon as those identifiers are found. This optimization is most useful for larger messages. For example, if the SOW key for a topic is based on the `/id` field of a message and there are active content filters that use both the `/id` field and the `/code` field, while no other field is being indexed, then, considering the message below: ```js {"id":24,"code":"A12347","notes":"entered on behalf of a sloth", // ... 100K of other data ... } ``` The AMPS parser can stop parsing after processing only the `/id` and the `/code` fields. In this case, halting the parsing after processing these two fields avoids the expense of parsing the remaining parts of the message. Notice that this optimization will only improve performance in cases where AMPS doesn't need to parse the entire message. For example, if there is a `delta_subscribe` active for the topic, or if the command being processed is a `delta_publish`, AMPS will parse the message completely to be able to calculate the deltas. Likewise, if any filter refers to a field that doesn't appear in the message, AMPS will parse the message completely to be able to determine that the field does not appear in the message. ## SOW Queries and Indexing Queries over topics in the State of the World (SOW) have additional performance considerations. AMPS maintains indexes over SOW topics to help locate messages in response to a query. * Queries over a topic in the SOW can use SOW topic indexes. Where possible, use an exact string match and create a hash index to take advantage of hash indexes. * When a query is submitted with an XPath identifier for which no index exists, AMPS will create and populate a memo index for that XPath identifier. This can add to the amount of time a query takes the first time a given XPath identifier is queried. You can specify that AMPS creates a memo index for a given identifier by using the `Index` configuration item in the `Topic` definition. Once an index is created, AMPS will continue to search for that XPath identifier in incoming messages for that topic to keep the index up to date. Notice that SOW topic indexes are only used for `sow` commands and during the `sow` portion of a `sow_and_subscribe` (or `sow_and_delta_subscribe`) command. Once the subscription to current updates begins, the subscription does not use a SOW topic index because there is no need to locate a message. During a subscription, filters are run against the current message. See the section on [Indexing for State of the World topics](../sow/sow\_indexing) for details. --- # Regular Expressions Regular expression matching provides precision, power, and flexibility for matching patterns. AMPS supports regular expression matching on topics and within content filters. Regular expressions are implemented in AMPS using the Perl-Compatible Regular Expressions (PCRE) library. For a complete definition of the supported regular expression syntax, please refer to: [http://perldoc.perl.org/perlre.html](http://perldoc.perl.org/perlre.html) To use regular expressions for topic matching, provide a regular expression pattern where you would normally provide a topic name. To use regular expressions in content filtering, compare strings to regular expressions using the `LIKE` operator. The syntax of the `LIKE` operator is: ```sql string LIKE pattern ``` In this context, a string is any expression that provides a string and pattern is a literal regular expression pattern. This chapter presents a brief overview of regular expressions in AMPS. However, this chapter is not exhaustive. For more information on regular expression matching, see the PCRE site mentioned above. ## Examples Here is an example of a content filter for messages that will match any message meeting the following criteria: * Regular expression match of symbols of 2 or 3 characters starting with “IB” * Regular expression match of prices starting with “90” * Numeric comparison of prices less than 91 The corresponding content filter would be: ```sql (/FIXML/Order/Instrmt/@Sym LIKE "^IB.?$") AND (/FIXML/Order/@Px LIKE "^90\..*" AND /FIXML/Order/@Px < 91.0) ``` The tables below contain a brief summary of special characters and constructs available within regular expressions. Here are more examples of using regular expressions within AMPS: Use `(?i)` to enable case-insensitive regular expression searching. For example, the following filter will be true regardless if `/client/country` contains “US” or “us”. ```sql (/client/country LIKE "(?i)ˆus$") ``` To match messages where tag 55 has a `TRADE` suffix, use the following filter: ```sql (/55 LIKE "TRADE$") ``` To match messages where tag 109 has a `US` prefix and a `TRADE` suffix, with case insensitive matching, use the following filter: ```sql (/109 LIKE "(?i)ˆUS.*TRADE$") ``` AMPS recognizes the following regular expression metacharacters: | Character | Meaning | | ------------- | ------------------------------ | | ^ | Beginning of string | | $ | End of string | | . | Any character except a newline | | * | Match previous 0 or more times | | ? | Match previous 0 or 1 times | | () | Grouping of expression | | [] | Set of characters | | \{\} | Repetition modifier | | \ | Escape for special characters | AMPS recognizes the following repetition constructs: | Construct | Meaning | | ------------- | -------------------------------------- | | `a*` | Zero or more _a_'s | | `a?` | Zero or one _a_'s | | `a{m}` | Exactly _m a_'s | | `a{m,}` | At least _m a_'s | | `a{m,n}` | At least _m_, but no more than _n a_'s | The table below lists some of the modifiers AMPS recognizes: | Modifier | Meaning | | ------------ | ----------------------------------------------------------------------------------------------------------------------- | | i | Case insensitive search | | m | Multi-line search | | s | Any character (including newlines) can be matched by a . character | | x | Unescaped white space is ignored in the pattern | | A | Constrain the pattern to only match the beginning of a string | | U | Make the quantifiers non-greedy by default (the quantifiers are greedy and try to match as much as possible by default) | ## Raw Strings AMPS additionally provides support for _raw strings_, which are strings prefixed by an 'r' or 'R' character. Raw strings use different rules for how a backslash escape sequence is interpreted by the parser. When a string literal is provided as a raw string, the characters in the raw string are matched exactly, even when those characters are special characters for a regular expression. In the example below, the raw string - noted by the `r` prefix of the string literal in the second operand of the `LIKE` predicate causes AMPS to search for the literal characters `++` in the results, without requiring those characters to be escaped. In this example we are querying for a string that contains the programming language named `C++`. In the regular string, we are required to escape the `'+'` character since it is also used in a regular expression as the “match previous 1 or more times” regular expression character. In the raw string we can use `r'C++'` to search for the string and not have to escape the special `'+'` character. An expression using the raw string capability would look like the following: ```sql /FIXML/Language LIKE r'C++' ``` This can be simpler and easier to read then the escaped equivalent, shown below: ```sql /FIXML/Language LIKE 'C\+\+' ``` ## Subscribing to a Set of Topics Using Regular Expressions As mentioned previously, AMPS supports regular expression filtering for topics, in addition to content filters. Regular expressions use the same grammar described in content filtering. Regular expression matching for topics is enabled in an AMPS instance by default. Subscriptions or queries that use a regular expression for the topic name provide all matching records from AMPS topics where the name of the topic matches the regular expression used for the subscription or query. For example, if your AMPS configuration has three SOW topics, `Topic_A`, `Topic_B` and `Topic_C` and you wish to search for all messages in all of your SOW topics for records where the `Name` field is equal to “Bob”, then you could use a `sow` command with a topic of `^Topic_.*` and a filter of `/FIXML/@Name='Bob'` to return all matching messages that match the filter in all of the topics that match the topic regular expression. AMPS interprets any topic name that contains regular expression characters as a regular expression. A subscription can specify that a topic name should not be interpreted as a regular expression by including `non_regex_topic` in the options for the subscription. Notice that, as with the `LIKE` expression, a regular expression will match at any position in the topic name. To anchor the match to the beginning of the string, use the `^` directive at the beginning of the regular expression. To anchor the match to the end of the string, use the `$` directive at the end of the string. For example, to match a topic with `"order"` anywhere in the topic name, you could use the regular expression `order.*` (the ending `.*` matches zero or more characters, but lets AMPS know to interpret this as a regular expression). To match only topics that start with `order`, you would use the regular expression `^order`. To match topics that end with `order`, you would use the regular expression `order$`. To match topics that match `order-na` or `order_northamerica` or `rollup-orders-northamerica` you could use a regular expression like `order.n`. :::info Results returned when performing a topic regular expression query will follow “configuration order” — meaning that the topics will be searched in the order that they appear in your AMPS configuration file. Using the above query example with `Topic_A`, `Topic_B` and `Topic_C`, if the configuration file has these topics in that exact order, the results will be returned first from `Topic_A`, then from `Topic_B` and finally the results from `Topic_C`. As with other queries, AMPS does not make any guarantees about the ordering of results within any given topic query. ::: --- # Syntax AMPS expressions are designed to work exactly as expected if you are familiar with XPath path specifiers and SQL-92 predicates. This section describes in detail how AMPS evaluates the syntax, operators, and functions available in the AMPS expression language. AMPS expressions combine the following elements: * _Identifiers_ specify a field in a message. When evaluating an expression, AMPS replaces identifiers with values from the message or set of messages being evaluated. * _Literal_ values are explicit values in an AMPS expression, such as `'IBM'` or `42.` * _Operators_ and _functions_ such as `=`, `<`, `>`, `*`, and `UNIX_TIMESTAMP().` Every AMPS expression produces a value. The way that AMPS uses the value depends on the context in which AMPS evaluates the expression. For example, if the expression is used for a filter, the message is considered to match the filter when the expression returns `true`. When an expression is used to project a field, the result of the expression is used as the value of the projected field. --- # Typed Value Construction Functions AMPS includes functions for explicitly constructing constant values of various types. |Function |Parameters |Description | |------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`FALSE_VALUE`|none |

    Returns a boolean false value.

    This function is most useful for constructing values in message types that have a distinct type for boolean values. In the AMPS expression language, false is equivalent to a literal 0.

    | |`TRUE_VALUE` |none |

    Returns a boolean true value.

    This function is most useful for constructing values in message types that have a distinct type for boolean values. In the AMPS expression language, true is typically represented with a literal 1.

    | |`NAN_VALUE` |none |Returns a NaN (not a number) value.| |`CHAR_VALUE` |integer (0-255) |

    Returns the character (byte) for the integer provided.

    This function is most useful for constructing values in message types that have a distinct type for char values. In the AMPS expression language, a single character value is equivalent to a string constructed with an escape, and constructing a string literal is more efficient. That is, `'\x01'` is more efficient in a filter or field construction than `CHAR_VALUE(1)`. However, to construct a character based on a field, use `CHAR_VALUE`; for example, `CHAR_VALUE(/code)`.

    | --- # Working with Arrays AMPS supports expressions that operate on arrays in messages. The principles behind how AMPS treats arrays are: 1. _Binary operators_ that yield `true` or `false` (for example, `=`, `<`, `LIKE`, `TOPIC_MATCH`) are _array aware_, as is the `IN` operator. These operators work on arrays as a whole, and evaluate every element in the array. 2. _Array reduce functions_, that is, functions that are intended to take an array and produce a single value, are array aware and produce a result over the full contents of the array. 3. _Arithmetic operators_, _functions_, _user-defined functions_ and other _scalar operators_, are _not_ array aware, and use the first element in the array. With these simple principles, you can predict how AMPS will evaluate an expression that uses an array. For any operator, an empty array evaluates to `NULL`. Let's look at some examples. For the purposes of this section, we will consider the following JSON document: ```javascript { "data" : [1, 2, 3, "zebra", 5], "other" : [14, 34, 23, 5] } ``` While these arrays are presented using JSON format for simplicity, the same principles apply to arrays in other message formats. Here are some examples of ways to use an array in an AMPS filter: #### _Determining if any element in an array meets a criteria_ To determine this, you provide the identifier for the array, and use a comparison operator. | Filter | Evaluates as | | ------------------ | ------------------------------------------------------- | | `/data = 1` | TRUE, `/data` contains `1` | | `/data = 'zebra'` | TRUE, `/data` contains `'zebra'` | | `/data != 'zebra'` | TRUE, `/data` contains an element that is not `'zebra'` | | `/data = 42` | FALSE, `/data` does not contain `42` | | `/data LIKE 'z'` | TRUE, a member of `/data` matches `'z'` | | `/other > 30` | TRUE, a member of `/other` is `> 30` | | `/other > 50` | FALSE, no member of `/other` is `> 50` | #### _Determine whether a specific value is at a specific position_ To determine this, use the subscript operator `[]` on the XPath identifier to specify the position, and use the equality operator to check the value at that position. | Filter | Evaluates as | | -------------------- | ---------------------------------------------- | | `/data[0] = 1` | TRUE, first element of `/data` is `1` | | `/data[3] = "zebra"` | TRUE, fourth element of `/data` is `'zebra'` | | `/data[1] != 1` | TRUE, second element of `/data` is not `1` | | `/other[1] LIKE '4'` | TRUE, second element of `/other` matches `'4'` | #### _Determine whether any value in one array is present in another array_ | Filter | Evaluates as | | ----------------- | ----------------------------------------------------------- | | `/data = /other` | TRUE, a value in `/data` equals a value in `/other` | | `/data != /other` | TRUE, a value in `/data` does not equal a value in `/other` | #### _Determine whether an array contains one of a set of values_ | Filter | Evaluates as | | ---------------------------------------- | --------------------------------------------------------------- | | `3 IN (/data)` | TRUE, `3` is a member of `/data` | | `/data IN (1, 2, 3)` | TRUE, a member of `/data` is in `(1, 2, 3)` | | `/data IN ("zebra", "antelope", "lion")` | TRUE, a member of `/data` is in `("zebra", "antelope", "lion")` | |

    `NOT /data IN ("zebra", "antelope", "lion")`

    `/data NOT IN ("zebra","antelope","lion")`

    | FALSE, a member of `/data` is in `("zebra", "antelope", "lion")` | These patterns and principles hold regardless of the original representation of the array in a document. When creating an expression that uses a field in a compound value, keep in mind that AMPS represents compound values as described in the section on [Compound Data Types in AMPS](amps-data-types.md#compound-types-in-amps). --- # AMPS Expressions AMPS includes an expression language that combines elements of XPath and SQL-92's `WHERE` clause. This expression language is used whenever the AMPS server refers to the contents of a message, including: * Content filtering * Constructing fields for message enrichment * Creating projected fields for views AMPS uses a common syntax for each of these purposes, and provides a common set of operators and functions. AMPS also provides special directives for message enrichment, and aggregation functions for projecting views. For example, when an expression is used as a content filter, any message for which the expression returns `true` matches the content filter. When an expression is used to construct a field for message enrichment or view projection, the expression is evaluated and the result that the expression returns is used as the content of the field. ## Expressions Overview The quickest way to learn AMPS expressions is to think of each as a combination of identifiers that tell AMPS where to find data in a message, and operators that tell AMPS what to do with that data. Each AMPS expression produces a value. The way AMPS uses that value depends on where the expression is used. For example, in a content filter, AMPS uses the value of the expression to determine whether a message matches the filter. When constructing a field, AMPS uses the value of the expression as the contents of the field. Consider a simple example of an expression used as a filter. Imagine AMPS receives the following JSON message: ```javascript {"name":"Gyro", "job":"kitten"} ``` Using an AMPS expression, you can easily construct a content filter that matches the message: ```sql /name = 'Gyro' ``` There are three parts to this expression. The first part, `/name`, is an _identifier_ that tells AMPS to look for the contents of the `name` field at the top level of the JSON document. The second part of the filter, `=`, is the equality _operator_, which tells AMPS to compare the values on either side of the operator and return `true` if the values match. The final part of the filter, `'Gyro'`, is a string _literal_ for the equality operator to use in the comparison. When an expression is used in a content filter, a message matches the filter when the expression returns `true`. The expression returns `true` for the sample message, so the sample message matches the filter. The identifier syntax is a subset of XPath, as described in the section on [Identifiers](amps-expressions/identifiers). The comparison syntax is similar to SQL-92. Notice that AMPS makes no rigid guarantees as to the number of times a given expression is evaluated or when that evaluation will take place. AMPS will evaluate the expression as needed. --- # AMPS Functions This section describes the functions installed by default in the AMPS server. Additional functions that ship with the AMPS server are provided in auxiliary modules, as described in the section on [Optional Functions](optional-modules/functions). --- # AMPS Statistics AMPS provides the ability to record the statistics gathered from the AMPS instance and the host machine. The AMPS statistics database is stored in sqlite3 format and can be used with any of the standard sqlite3 tools. This section assumes that you are using the standard `sqlite3` package installed on your local computer. While you may be able to run the SQL examples in this guide using other packages, this guide will assume that all SQL commands will be executed with `sqlite3`. Notice that the statistics subsystem is independent of the other subsystems in AMPS, and is the only part of AMPS that uses the sqlite3 format. You cannot use sqlite3 tools with SOW files, journal files or .ack files: these files use formats specifically designed for high performance messaging. Working with the AMPS statistics database is described in more detail in the following sections. ## Configuring AMPS to Persist Statistics By default, AMPS maintains statistics in memory. To configure AMPS to record the statistics to a file, the following configuration options are available in the AMPS configuration file to update the location and frequency of the statistics database file. ```xml showLineNumbers ... localhost:9090 ./stats.db 5s ... ``` In the example above, the AMPS administration interface is set to collect statistics every 5 seconds as indicated by the `` tag. The AMPS administration interface is additionally configured to save the statistics in the `stats.db` file, which will be created in the directory where AMPS was started. AMPS does not require that statistics are persisted. Persisting statistics enables information about instance performance, capacity, usage, and so on to be analyzed offline (rather than by using RESTful operations against a running instance). This also enables statistical information about the instance to be persisted when AMPS restarts. ## Introduction to SQLite3 This section is a quick reference to sqlite3. It is intended to help in getting started with examining the statistics provided by AMPS. While this guide will be sufficient to execute the examples listed, a more comprehensive guide of the sqlite3 command line tool is available at [http://www.sqlite.org/sqlite.html](http://www.sqlite.org/sqlite.html). ### Starting SQLite3 To start sqlite3 with the stats.db file simply type: ```sql $> sqlite3 ./stats.db ``` This will create a command prompt that looks like the following: ```sql $> sqlite3 ./stats.db SQLite version 3.7.3 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> ``` To exit the sqlite3 prompt at any time, use the Ctrl+d sequence. ### Simple SQLite3 Commands #### Tables To get a listing of all available tables in the sqlite database type the `.table` command. ```sql sqlite> .table HCPUS_DYNAMIC IMEMORY_CACHES_DYNAMIC HCPUS_STATIC IMEMORY_CACHES_STATIC HDISKS_DYNAMIC IMEMORY_DYNAMIC HDISKS_STATIC IMEMORY_STATIC HMEMORY_DYNAMIC IPROCESSORS_DYNAMIC HMEMORY_STATIC IPROCESSORS_STATIC HNET_DYNAMIC IQUEUES_DYNAMIC HNET_STATIC IQUEUES_STATIC ICLIENTS_DYNAMIC IREPLICATIONS_DYNAMIC ICLIENTS_STATIC IREPLICATIONS_STATIC ICONFLATEDTOPICS_DYNAMIC ISOW_DYNAMIC ICONFLATEDTOPICS_STATIC ISOW_STATIC ICONSOLE_LOGGERS_DYNAMIC ISTATISTICS_DYNAMIC ICONSOLE_LOGGERS_STATIC ISTATISTICS_STATIC ICPUS_DYNAMIC ISUBSCRIPTIONS_DYNAMIC ICPUS_STATIC ISUBSCRIPTIONS_STATIC IFILE_LOGGERS_DYNAMIC ISYSLOG_LOGGERS_DYNAMIC IFILE_LOGGERS_STATIC ISYSLOG_LOGGERS_STATIC IGLOBALS_DYNAMIC ITRANSPORTS_DYNAMIC IGLOBALS_STATIC ITRANSPORTS_STATIC ILIFETIMES_DYNAMIC IVIEWS_DYNAMIC ILIFETIMES_STATIC IVIEWS_STATIC ``` #### Schema To view the schema for any table, type the `.schema ` command where `
    ` is the name of the table to inspect. ```sql sqlite> .schema IFILE_LOGGERS_DYNAMIC CREATE TABLE IFILE_LOGGERS_DYNAMIC( timestamp integer, static_id integer, bytes_written integer, PRIMARY KEY( timestamp, static_id ) ); ``` ## Statistics Table Design This section describes the philosophy of how the AMPS tables are designed within the statistics database. This chapter also includes some examples of some useful queries which can give an administrator more information than just the raw data would normally give them. Such information can be a powerful tool in diagnosing perceived problems in AMPS. ### Table Naming Scheme Tables in the database use the following naming scheme: ```sql _ Where: I = AMPS instance statistics H = Host statistics STAT = The statistics that are collected (MEMORY, CPUs, SUBSCRIPTIONS, etc) STATIC = Attributes that rarely change for an object (such as client name, CPU #) DYNAMIC = Stats that are expected to change on every sample (rates, counters, and so on) ``` ### Example Queries To view which clients have fallen behind at one time, run: ```sql sqlite> SELECT s.client_name, MAX(d.queue_max_latency), MAX(queued_bytes_out) FROM ICLIENTS_DYNAMIC d JOIN ICLIENTS_STATIC s ON (s.static_id=d.static_id) GROUP BY s.client_name; ``` To view clients that are behind in the latest sample: ```sql sqlite> SELECT s.client_name, d.queue_max_latency, queued_bytes_out FROM ICLIENTS_DYNAMIC d JOIN ICLIENTS_STATIC s ON (s.static_id=d.static_id) WHERE d.timestamp = (SELECT MAX(d.timestamp) FROM ICLIENTS_DYNAMIC d) AND d.queue_max_latency > 0; ``` ## Using the amps-sqlite3 Utility The AMPS distribution includes a convenience utility, `amps-sqlite3`, for easily running queries against a statistics database. The utility takes two parameters, as shown below: | Parameter | Description | | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `database` | The sqlite3 database file to query. | | `query` | The query to run. Notice that the query must be enclosed in quotes, since this is a command-line program run by the Linux shell. | The `amps-sqlite3` utility joins the `STATIC` and `DYNAMIC` tables together, making a single table that is easier to query on. For example, the script joins the `ICLIENTS_DYNAMIC` and `ICLIENTS_STATIC` tables together into a single `ICLIENTS` table. The `amps-sqlite3` utility also provides a set of convenience functions that can be included in the query. | Option | Description | | --------------------------- | ---------------------------------------------------------------------- | | ISO8601(_timestamp_) | Convert `timestamp` to an ISO8601 format string. | | ISO8601\_local(_timestamp_) | Convert `timestamp` to an ISO8601 format string in the local timezone. | | timestamp(_string_) | Convert the provided ISO8601 format `string` to a timestamp. | To use `amps-sqlite3`, simply provide the file name of the database to query and the query to run. For example, the following query returns the set of samples AMPS has recorded for the `system_percent` consumed on each CPU while the instance has been running: ```bash $ amps-sqlite3 stats.db "SELECT iso8601(timestamp),system_percent FROM hcpus ORDER BY timestamp" ``` ## SQLite Tips and Troubleshooting This section includes information on SQLite tasks that may not be immediately obvious and troubleshooting information on SQLite. ### Converting AMPS Statistics Time to an ISO8601 Datetime This Python function shows how to convert an AMPS timestamp to an ISO8601 datetime. You can use the equivalent in your language of choice to convert between the timestamps recorded in the statistics database and ISO8601 timestamps. ```python showLineNumbers def iso8601_time(amps_time): """ Converts AMPS stats time into an ISO8601 datetime. """ pt = float(amps_time)/1000 - 210866803200 # subtract the Unix epoch it = int(pt) ft = pt-it return time.strftime("%Y%m%dT%H%M%S", time.localtime(it)) + ("%.6f" % ft)[1:] ``` ### Shrinking an AMPS Statistics Database If the retention policy for an AMPS statistics database has changed such that there is unused space in the file at the maximum retention size, it may be helpful to shrink the size of the statistics database. Running this procedure will only reduce the size of the database if the database has been truncated. To do this: 1. Take the AMPS instance offline. 2. (Optional, but recommended) Make a backup copy of the AMPS statistics database on another device. 3. Run the sqlite `VACUUM` command to shrink the database: ```bash sqlite3 stats.db 'VACUUM' ``` This operation may require free disk space equal to the current size of the statistics database. If the operation fails, the vacuum rolls back without changing the database. A database _should not_ be vacuumed while AMPS is running. ### Troubleshooting "Database Disk Image is Malformed" To repair this error, you need to extract the data from the SQLite datastore and create a new datastore. To do this: 1. Open the sqlite datastore. For example, if the database store is named `stats.db`, the command would be: ```bash sqlite3 stats.db ``` 2. Dump the data into a SQL script. ```bash .mode insert .output stats_data.sql .dump .exit ``` This creates a series of SQL commands that recreate the data in the database. 3. Make sure that the script commits updates (depending on the version of sqlite3 and the state of the database, the script may roll back the updates rather than committing them without this step). ```bash sed -i 's/^ROLLBACK;/COMMIT;/ig' stats_data.sql ``` 4. Now create a new database file using the SQL commands. ```bash sqlite3 good.db < stats_data.sql ``` Finally, adjust the configuration of the Admin server to use the new database (in this example, `good.db`) or copy the new database over the old database. --- # Aggregate Functions AMPS provides a set of aggregation functions that can be used in a `Field` constructor for a view and in the `projection` option of an aggregated subscription. These functions return a single value for each distinct group of messages, as identified by distinct combinations of values in the `Grouping` clause. These functions produce an aggregation over a literal value, an identifier directing AMPS to extract the value from the message, or the result of a function. For example, given a set of messages like the following: ```javascript {"id":1, "item":1,"qty":10, "oid":1, ...} {"id":2, "item":2,"qty":10, "oid":1, ...} {"id":3, "item":3,"qty":25, "oid":1, ...} ``` With a view definition that has a `Projection` clause and `Grouping` clause like the following: ```xml showLineNumbers /oid SUM(/qty) AS /totalOrderQty SUM(IF((/qty % 10) == 0,1,0)) AS /evenTensOrderCount /oid ``` AMPS will produce the following record: ```javascript {"oid":1,"totalOrderQty":45,"evenTensOrderCount":2} ``` Notice that the first `SUM()` function simply extracts the value of the /qty from each message, while the second `SUM()` function uses the output of the `IF` statement for each message. Since aggregate functions operate over groups of messages, these functions are only available when constructing fields for aggregate purposes, either in a view or an aggregated subscription. The functions described in this section are not available to filters, and are not available for constructing fields during SOW topic enrichment. The set of functions provided in AMPS have been chosen to be efficient to compute over high volumes of rapidly changing data. Null values are not included in aggregate expressions with AMPS, nor in ANSI SQL. `COUNT` will count only non-null values, `SUM` will add only non-null values, `AVG` will average only non-null values, and `MIN` and `MAX` ignore `NULL` values, and so on. `MIN` and `MAX` can operate on either numbers or strings, or a combination of the two. AMPS compares values using the principles described for comparison operators. For `MIN` and `MAX`, AMPS determines order based on these rules: * Numbers sort in numeric order. * String values sort in ASCII order. * When comparing a number to a string, convert the string to a number, and use a numeric comparison. If that is not successful, the value of the string is higher than the value of the number. For example, given a field that has the following values across a set of messages: ```sql 24, 020, 'cat', 75, 1.3, 200, '75', '42' ``` `MIN` will return `1.3`, `MAX` will return `'cat'`. Note that different message types may have different support for converting strings to numeric values: AMPS relies on the parsing done by the message type to determine the numeric value of a string. ## AVG --- ```sql AVG(expression) ``` Averages an expression. **Parameters** * `expression`: The expression to average. **Returns** The mean value of the values specified by the expression. ## ANY --- ```sql ANY(expression) ``` Returns _one_ of the set of values in the expression. **Parameters** * `expression`: The expression to evaluate. **Returns** One of the set of values in the expression. ## COUNT --- ```sql COUNT(expression) ``` Counts the values in an expression. **Parameters** * `expression`: The expression to count values from. **Returns** The number of values specified by the expression. ## COUNT_DISTINCT --- ```sql COUNT_DISTINCT(expression) ``` Counts the number of distinct values in an expression, ignoring `NULL`. **Parameters** * `expression`: The expression to count distinct values from. **Returns** The number of distinct values in the expression. AMPS type conversion rules apply when determining distinct values. ## GROUP_CONCAT --- ```sql GROUP_CONCAT(expression, [delimiter]) ``` Creates a list of the distinct values in the specified expression, using the second argument as the delimiter. **Parameters** * `expression`: The expression that provides the values to concatenate. * `delimiter`: Optional. The delimiter to use between values. If no delimiter is provided, the delimiter defaults to `,` (a comma). **Returns** A string that contains the distinct values from the expression, separated by the delimiter. This function returns a string, regardless of the types of the values in the expression. The order of the values within the string is not guaranteed. **Example** To create a list of the distinct values in the `/names` column for the group delimited by a `|` character, you would use: ```sql GROUP_CONCAT(/names, '|') ``` ## MIN --- ```sql MIN(expression) ``` Returns the minimum value out of the values specified by the expression. **Parameters** * `expression`: The expression to find the minimum value from. **Returns** The minimum value. ## MAX --- ```sql MAX(expression) ``` Returns the maximum value out of the values specified by the expression. **Parameters** * `expression`: The expression to find the maximum value from. **Returns** The maximum value. ## STDDEV_POP --- ```sql STDDEV_POP(expression) ``` Calculates the population standard deviation of an expression. **Parameters** * `expression`: The expression for which to calculate the standard deviation. **Returns** The calculated standard deviation. ## STDDEV_SAMP --- ```sql STDDEV_SAMP(expression) ``` Calculates the sample standard deviation of an expression. **Parameters** * `expression`: The expression for which to calculate the standard deviation. **Returns** The calculated standard deviation. ## SUM --- ```sql SUM(expression) ``` Calculates the summation over an expression. **Parameters** * `expression`: The expression to sum. **Returns** The total value of the values specified by the expression. ## UNIQUE --- ```sql UNIQUE(expression) ``` Determines if all of the values in a given field match within the group. **Parameters** * `expression`: The expression to check for uniqueness. **Returns** If all of the values match, returns the value. Otherwise, returns `NULL`. --- # AMPS Function Overview In the AMPS expression language, a function can be used in any place an _identifier_ or _literal value_ can be used. All AMPS functions return a single value. During evaluation of an AMPS expression, AMPS calls the functions in the expression and uses the results to evaluate the expression. A function may perform type conversion as needed to evaluate the expression. The results of a call to an AMPS function can be used as the parameter to an AMPS function. For example, the following is a valid expression: ```sql REVERSE(SUBSTR('fandango',5)) == 'ogna' ``` In this case, AMPS first evaluates the `SUBSTR` function, which requests the subset of the string `fandango`, starting at position `5`. That function returns `ango`. AMPS then uses the string `ango` as the input to the `REVERSE` function, which returns the result `ogna`. The following table lists the available functions by category: | Category | Function Types Provided | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | String | [String Comparison Functions](string-comparison-functions) | | | [Converting Arrays to Strings](array-reduce-functions#array_to_string)
    (see `ARRAY_TO_STRING`) | | | [Concatenating Strings](concatenating-strings) | | | [Managing String Case](managing-string-case) | | | [Replacing Text in Strings](replacing-text-in-strings) | | | [String Manipulation Functions](string-manipulation-functions) | | Date and Time | [Date and Time Functions](date-and-time-functions)
    See also [Legacy Messaging Compatibility Functions](../optional-modules/legacy-messaging-functions) | | Numeric and Math | [Numeric and Math Functions](numeric-functions) | | Array Reduce | [Array Reduce Functions](array-reduce-functions) | | Geospatial | [Geospatial Functions](geospatial-functions) | | Identifier and Checksum | [Identifier and Checksum](identifier-and-checksum-functions) | | Message | [Current Message Functions](message-functions) | | Client | [Client Information Functions](client-functions) | | AMPS Information | [AMPS Information Functions](amps-information-functions) | | NULL and Typed Value Creation | [Finding Non-NULL Values](coalesce-function) | | | [Typed Value Creation](typed-value-creation) | | Aggregate | [Aggregate Functions](aggregate_functions) | | Field Construction | [Constructing Fields](constructing-fields) | ## Deterministic and Non-Deterministic Functions The AMPS server distinguishes between functions that produce a consistent value for the same message (_deterministic_ functions) and functions that may produce a different value each time it is called, even if the function has the same input and is called for the update to the same message (_non-deterministic_ functions). There are no restrictions on the use of deterministic functions, since each time that they are called for a given message (or a given update to a message), they will return a consistent result. Some features of AMPS rely on being able to evaluate an expression in a consistent way for a given message. A function that can produce a different value each time that it is called cannot be used in those situations: otherwise, AMPS could produce incorrect (or meaningless) results. In practice, this means that a non-deterministic function: * Cannot be used in the filter of a subscription that requests out of focus (oof) notifications. * Cannot be used in the filter of an aggregated subscription (although a non-deterministic function is allowed in the filter of an aggregated _query_, since the filter will only be evaluated once per message). * Cannot be used in an aggregate function (aggregate functions are available in views, aggregated subscriptions, and aggregated queries). * Cannot be used in the filter for a `sow_and_subscribe` command that uses pagination (that is, a command that specifies `top_n`/`skip_n`/`OrderBy`). * Cannot be used in the filter for a queue or the barrier expression for a queue. * Cannot be used in the filter for a view or conflated topic. * Cannot be used in a replication filter. In this release, `LAST_READ`, `UNIX_TIMESTAMP`, `UUID7` and `VALUE_LOOKUP` are non-deterministic. The other functions provided with AMPS (both built in and provided through auxiliary modules) are deterministic. --- # AMPS Information Functions AMPS includes a pair of functions that provide the instance name and group name of the current instance. ## AMPS_INSTANCE_NAME --- ```sql AMPS_INSTANCE_NAME() ``` Returns the instance name of this AMPS instance. **Parameters** None. **Returns** The instance name of this AMPS instance. **Example** ```sql AMPS_INSTANCE_NAME() IN ('INSTANCE-A', 'INSTANCE-B') ``` ## AMPS_GROUP_NAME --- ```sql AMPS_GROUP_NAME() ``` Returns the group name of this AMPS instance. **Parameters** None. **Returns** The group name of this AMPS instance. **Example** ```sql AMPS_GROUP_NAME() == 'GROUP-1' ``` ## VERSION --- ```sql VERSION() ``` Returns the version of the running AMPS server. **Parameters** None. **Returns** The version of your running AMPS server instance. **Example** ```sql SUBSTR(VERSION(), 1, 5) == '5.3.5' ``` --- # Array Reduce Functions AMPS includes a set of functions designed to operate over an array element in a message and produce a value. These functions take an array within a single message as input, and reduce that array to a single value as output. See [Compound Types in AMPS](/docs/amps-user-guide/amps-expressions/amps-data-types#compound-types-in-amps) and [Working with Arrays](/docs/amps-user-guide/amps-expressions/working-with-arrays) for more info on how arrays are parsed and used in AMPS. ## ARRAY_COUNT --- ```sql ARRAY_COUNT(array) ``` Returns the number of elements in the array. **Parameters** * `array`: The array to count the elements of. **Returns** The number of elements in the array. **Example** ```sql ARRAY_COUNT(/values) AS /count ``` ## ARRAY_MAX --- ```sql ARRAY_MAX(array) ``` Returns the largest value in the array. **Parameters** * `array`: The array to find the maximum value of. **Returns** The largest value in the array, using the standard AMPS `>` comparison. **Example** ```sql ARRAY_MAX(/values) AS /max ``` ## ARRAY_MIN --- ```sql ARRAY_MIN(array) ``` Returns the minimum value in the array. **Parameters** * `array`: The array to find the minimum value of. **Returns** The minimum value in the array, using the standard AMPS `<` comparison. **Example** ```sql ARRAY_MIN(/values) > 0 ``` ## ARRAY_SUM --- ```sql ARRAY_SUM(array) ``` Returns a number produced by adding all of the elements in the array. Non-numeric values in the array are ignored. **Parameters** * `array`: The array to sum. **Returns** The sum of all elements in the array. **Example** ```sql ARRAY_SUM(/orders/orderQty) >= 1000 ``` ## ARRAY_TO_STRING --- ```sql ARRAY_TO_STRING(array, delimiter, null_replacement) ``` Returns a string comprised of the elements of the array, separated by the provided delimiter. `NULL` values in the array are replaced with the provided `null_replacement` value. **Parameters** * `array`: The array to convert to a string. * `delimiter`: The delimiter to use between array elements. * `null_replacement`: The value to use for `NULL` elements in the array. **Returns** A string representation of the array. **Example** The following example shows a JSON message, an enrichment field, and the resulting message. ```sql {"id": 123, "childOrder": [456, 789]} ARRAY_TO_STRING(/childOrder, ",", "") AS /childOrders {"id": 123, "childOrder": [456, 789], "childOrders": "456,789"} ``` --- # Client Functions AMPS includes functions that return information about the currently connected client. As with the message functions, these functions return information about the client that prompted the operation, if one is present. ## CLIENT_NAME --- ```sql CLIENT_NAME() ``` Returns the name of the currently connected client. **Parameters** None. **Returns** The name of the currently connected client. #### Available For * Subscriptions to a topic or conflated topic. * Enrichment and preprocessing. ## USER --- ```sql USER() ``` Returns the user ID of the currently connected client. **Parameters** None. **Returns** The user ID of the currently connected client. #### Available For * Subscriptions to a topic or conflated topic. * Enrichment and preprocessing. ## REMOTE_ADDRESS --- ```sql REMOTE_ADDRESS() ``` Returns the remote address of the currently connected client. **Parameters** None. **Returns** The remote address of the currently connected client. #### Available For * Subscriptions to a topic or conflated topic. * Enrichment and preprocessing. ## CLIENT_VERSION --- ```sql CLIENT_VERSION() ``` Returns the version string reported by the currently connected client. **Parameters** None. **Returns** The version string reported by the currently connected client. #### Available For * Subscriptions to a topic or conflated topic. * Enrichment and preprocessing. ## CONNECTION_NAME --- ```sql CONNECTION_NAME() ``` Returns the connection name of the currently connected client. **Parameters** None. **Returns** The connection name of the currently connected client. #### Available For * Subscriptions to a topic or conflated topic. * Enrichment and preprocessing. --- # Coalesce Function AMPS includes a function that accepts any number of arguments and returns the first argument that is not NULL. ## COALESCE --- ```sql COALESCE(value, [value, ...]) ``` Returns the first `value` that is not `NULL`. **Parameters** * `value`: One or more values to check. **Returns** The first value that is not `NULL`. If all values are `NULL`, returns `NULL`. **Example** ```sql COALESCE(/driverLicense, /passport, /militaryID, "NO ID") AS /id COALESCE(/value OF CURRENT, /value OF PREVIOUS, 0) AS /value ``` --- # Concatenating Strings AMPS provides the `CONCAT` function, that can be used for constructing strings. ## CONCAT --- ```sql CONCAT(string1, [string2, ...]) ``` Takes any number of parameters and returns a string constructed from those parameters. The function can accept both XPath identifiers and literal values. **Parameters** * `string1`: The first string. * `string2`: Optional. Additional strings to concatenate. **Returns** A single string that is the concatenation of all the parameters. **Example** The `CONCAT` function can be used in any AMPS expression that uses a string. For example, you could `CONCAT` in a filter as follows: ```sql CONCAT(/firstName, " ", /lastName) = 'George Orwell' ``` `CONCAT` can be combined with other expressions, including conditional expressions. A `mailingAddressName` field in a view could be constructed as follows: ```xml CONCAT(/firstName, " ", /lastName, IF(/suffix NOT NULL, CONCAT(", ", /suffix), "") ) AS /mailingAddressName ``` --- # Constructing Fields For views, aggregated subscriptions, and SOW topic enrichment, AMPS allows you to construct new fields based on existing data. When you construct a field, there are two components required: 1. A _source expression_ that produces a value. This expression can include XPath identifiers that extract values from a message, literal values, operators, and functions. 2. A _destination identifier_ that specifies the identifier where the message type will serialize the value produced by the source expression. The source expression and the destination identifier are separated by the `AS` keyword. The format for a field construction expression is as follows: ```xml AS ``` For example, to create a field in a view that calculates the total value of an order by multiplying the `/price` field times the `/qty` field, construct the field as shown below: ```xml /price * /qty AS /total ``` This constructs a field using `/price * /qty` as the source expression. Both `/price` and `/qty` are taken from the incoming message. When the result of this expression is computed, the value will be produced with the XPath identifier `/total` as the destination. That value will then be serialized to a message (with the exact format and syntax determined by the message type). Notice that the grammar for constructing fields does not specify precisely how the field is represented in the message. AMPS constructs the value and provides the XPath identifier to the message type. The message type itself is responsible for serializing the value into the correct representation and structure for that message type. All of the AMPS operators and functions that are available for filters are available to use in source expressions, including any user-defined functions loaded into the instance. Depending on the context for field construction, there are additional capabilities available when constructing fields, as described in the following sections. ## Constructing Preprocessing Fields Preprocessing field constructors operate on a single message and construct fields based on that message. The results of the preprocessing field constructor are merged into the incoming message. Any field in the source message that is not changed or removed during preprocessing is left unchanged, so it is not necessary to include all fields in the message in the `Preprocessing` block. Since preprocessing fields apply to a specific message, preprocessing fields cannot specify the topic or message type in an XPath identifier. All identifiers in the source expression are evaluated as identifiers in the message being preprocessed. Preprocessing fields are evaluated during the preprocessing phase, so they cannot refer to the previous state of a message. ### Using HINT to Control Field Construction Preprocessing can be used to remove fields from a message. By default, AMPS serializes any field that has an empty string or `NULL` value after preprocessing. Preprocessing fields can include a directive that specifies that a field that contains a `NULL` value should be removed from the set of fields rather than serialized with a `NULL` value. The directive `HINT OPTIONAL` applied to the XPath identifier specifies that if the result of the source expression is `NULL`, AMPS does not provide the value for the message type to serialize. For example, the following field constructor removes the `/source` field from the message if the value provided is not in a specific list of values: ```xml IF(/source IN ('a','e','f'), /source, NULL) AS /source HINT OPTIONAL ``` By default, AMPS considers the results of field construction (the processed message) to be distinct from the current message. AMPS rewrites the current message _after_ preprocessing is completed. This means that, by default, the results of fields constructed during preprocessing are not available to other fields within preprocessing. The `HINT SET_CURRENT` option immediately inserts or updates values in the current message, which makes the new value available to all subsequent `Field` declarations. In the sample below, AMPS enriches the message by performing an expensive operation (implemented as a user-defined function) on two input fields, and immediately updates the current message with the output of that operation. AMPS then sets other fields in the processed message using the updated value in the current message. ```xml EXPENSIVE_UDF_CALL(/dataSet1, /dataSet2) AS /processedData HINT SET_CURRENT IF(/processedData > 1000000, 'A', 'B') AS /resultClass ``` Notice that using `HINT SET_CURRENT` requires AMPS to process `Field` declarations in order, which may prevent future optimizations. Hints can be combined as follows: ```xml EXPENSIVE_UDF_CALL(/dataSet1, /dataSet2) AS /processedData HINT SET_CURRENT,OPTIONAL ``` In this case, if the projected field would be `NULL`, the field is removed from the current message. ## Constructing Enrichment Fields Enrichment field constructors operate on a single message and construct fields based on that message. Enrichment expressions operate on the current message and change the current message. The results of the enrichment directives are merged into the incoming message. Any field in the source message that is not changed or removed during preprocessing is left unchanged, so it is not necessary to include all fields in the message in the `Enrichment` directive. Since enrichment fields apply to a specific message, enrichment fields cannot specify the topic or message type in an XPath identifier. All identifiers in the source expression are evaluated as identifiers in the message being enriched. Enrichment fields are constructed during the enrichment phase, so enrichment fields can refer to the previous state of a message. Within an enrichment expression, AMPS provides two special modifiers for XPath identifiers that specify whether an XPath identifier refers to the current incoming message or the previous state of the message. These modifiers apply only to the source expression, and cannot be used in the destination identifier. The modifiers are as follows: | Modifier | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OF CURRENT` | Specify that the XPath identifier refers to the incoming message. | | `OF PREVIOUS` |

    Specify that the XPath identifier refers to the previous state of the message in the SOW.

    If there is no record in the SOW for this message, all identifiers that specify `OF PREVIOUS` return `NULL`.

    *`Enrichment` on `ConflatedTopic` does not support `OF PREVIOUS`*

    | ### Using HINT to Control Field Construction Enrichment can be used to remove fields from a message. By default, AMPS serializes any field that has an empty string or `NULL` value after enrichment. Enrichment `Field` elements can include a directive that specifies that a field that contains a `NULL` value should be removed from the message rather than serialized with a `NULL` value. The directive `HINT OPTIONAL` applied to the XPath identifier specifies that if the result of the source expression is `NULL`, AMPS does not provide the value for the message type to serialize. For example, the following field constructor removes the `/source` field from the message if the value provided is not in a specific list of values: ```xml IF(/source IN ('a','e','f'), /source, NULL) AS /source HINT OPTIONAL ``` By default, AMPS considers the results of field construction (the enriched message) to be distinct from the current message. AMPS rewrites the current message _after_ enrichment is completed. This means that, by default, the results of fields constructed during enrichment are not available to other fields within enrichment. The `HINT SET_CURRENT` option immediately inserts or updates values in the current message, which makes the new value available to all subsequent `Field` declarations. In the sample below, AMPS enriches the message by performing an expensive operation (implemented as a user-defined function) on two input fields, and immediately updates the current message with the output of that operation. AMPS then sets other fields in the processed message using the updated value in the current message. ```xml EXPENSIVE_UDF_CALL(/dataSet1, /dataSet2) AS /processedData HINT SET_CURRENT IF(/processedData > 1000000, 'A', 'B') AS /resultClass ``` Notice that using `HINT SET_CURRENT` requires AMPS to process `Field` declarations in order, which may prevent future optimizations. Hints can be combined as follows: ```xml EXPENSIVE_UDF_CALL(/dataSet1, /dataSet2) AS /processedData HINT SET_CURRENT,OPTIONAL ``` In this case, if the projected field would be `NULL`, the field is removed from the current message. ## Constructing View Fields View field constructors operate over groups of messages, and construct a single output message for each distinct group, as specified by the `Grouping` element in the `View` configuration. When constructing a field in a view, all identifiers used in the source expression must be in one of the underlying topics for the view. When the view uses a `Join`, the identifiers must include the topic identifier. If the topics in the `Join` are of different message types, the identifiers must include both the message type and the topic identifier. For example, the following `Field` definition multiplies the `/quantity` from the NVFIX topic `orders` by the `/price` from the JSON topic `items`, and projects the result into the `/total` field of the view. ```xml [nvfix].[orders]./quantity * [json].[items]./price AS /total ``` --- # Date and Time Functions AMPS includes functions for working with date and time values. This section covers functions loaded into AMPS by default. AMPS also includes functions for working with date and time in the Legacy Messaging Compatibility layer. ## STRFTIME --- ```sql STRFTIME(format_string, timestamp) ``` Produces a string that contains a representation of the provided `timestamp`, formatted as specified in the provided `format_string`. **Parameters** * `format_string`: The format string, using the same format specifiers as the standard `strftime(3)` function. This function also supports the additional format specifier `%f` to format microseconds, and the format specifier `%03f` to format milliseconds. * `timestamp`: The UNIX timestamp to format. **Returns** A formatted string representation of the timestamp in UTC. The length of the string produced for the time is limited to 128 bytes. **Example** ```sql STRFTIME("%Y-%m-%d %H:%M:%S", UNIX_TIMESTAMP()) ``` ## STRPTIME --- ```sql STRPTIME(time_string, format_string) ``` This function interprets the `time_string` provided as a UTC timestamp, with the `format_string` specifying how to interpret the `time_string`. **Parameters** * `time_string`: The string containing a timestamp to parse, interpreted as UTC. * `format_string`: The format string, using the same format specifiers as the standard `strptime(3)` function. This function also supports the additional format specifier `%f` to parse microseconds, and the format specifier `%03f` to parse milliseconds. **Returns** A double representing the parsed timestamp in UNIX time. If conversion fails, `-1.0` is returned. **Example** ```sql STRPTIME("2020-10-31", "%Y-%m-%d") ``` ## UNIX_TIMESTAMP --- ```sql UNIX_TIMESTAMP() ``` Returns the current timestamp as a double, represented in seconds (including parts of a second as a decimal). A UNIX timestamp is seconds elapsed since 00:00 on January 1, 1970 in UTC and is independent of the timezone of the local system. The underlying system call used for this function has microsecond resolution, subject to any hardware or host limitations. :::info This function is non-deterministic, and cannot be used in contexts that require a deterministic function. ::: **Parameters** None. **Returns** A double representing the current UNIX timestamp. **Example** ```sql UNIX_TIMESTAMP() AS /timestamp ``` --- # Extracting Matching Text in Strings AMPS includes a function, `REGEXP_MATCH`, that returns the first match of text within a string. ## REGEXP_MATCH --- ```sql REGEXP_MATCH(string_to_transform, pattern_or_text_to_match) ``` Returns the first occurrence of the `pattern_or_text_to_match`. This can be either a literal match or PCRE pattern. **Parameters** * `string_to_transform`: The string to search within. * `pattern_or_text_to_match`: The literal text or PCRE pattern to match. **Returns** The first matching text, or `NULL` if there is no match. **Example** The following expressions all evaluate as true: ```sql REGEXP_MATCH('sheepdog', 'dog') > 0 REGEXP_MATCH("30dogs", "^[0-9]*") == "30" ``` The following snippet shows `REGEXP_MATCH` constructed as a field: ```xml COALESCE(REGEXP_MATCH(/petDescription, "cat|dog|bear"), "back away slowly, I don't know what that is") as /animal_type ``` With the following data: ```json {"petDescription": "tabby cat"} {"petDescription": "havanese dog"} {"petDescription": "grizzly bear"} {"petDescription": "blue gerbil"} ``` Would result in: ```json {"petDescription":"tabby cat","animal_type":"cat"} {"petDescription":"havanese dog","animal_type":"dog"} {"petDescription":"grizzly bear","animal_type":"bear"} {"petDescription":"blue gerbil","animal_type":"back away slowly, I don't know what that is"} ``` --- # Geospatial Functions AMPS includes a function for calculating the distance from a signed latitude and longitude. ## GEO_DISTANCE --- ```sql GEO_DISTANCE(first_latitude, first_longitude, second_latitude, second_longitude) ``` Returns a double that contains the distance between the point identified by `first_latitude`, `first_longitude` and `second_latitude`, `second_longitude` in meters. AMPS uses the haversine formula when computing distances. **Parameters** * `first_latitude`: The latitude of the first point. * `first_longitude`: The longitude of the first point. * `second_latitude`: The latitude of the second point. * `second_longitude`: The longitude of the second point. **Returns** A double representing the distance between the two points in meters. **Example** Given a home point and a message containing `/lat` and `/long` fields, you could use the following expression to calculate the distance from home. ```sql GEO_DISTANCE( /lat, /long, 40.786337, -119.206508) ``` --- # Identifier and Checksum AMPS includes functions that generate identifiers and compute checksums. These functions are useful for creating unique values, creating a shortened representation of a string, or providing input for a `MOD` calculation. ## UUID7 --- ```sql UUID7() ``` Generates a unique identifier (UUID) in the version 7 format specified by RFC 9562. :::info This function is non-deterministic, and cannot be used in contexts that require a deterministic function. ::: **Parameters** None. **Returns** The identifier is returned as a formatted string. **Example** ```sql UUID7() AS /uuid ``` ## MD5 --- ```sql MD5(string) ``` Returns the MD5 checksum of the provided string. **Parameters** * `string`: The string to checksum. **Returns** The MD5 checksum of `string` given as a string of hexadecimal values. If a `NULL` value is provided, this function returns a constant value. **Example** ```sql MD5(/data) ``` ## CRC32 --- ```sql CRC32(string) ``` Returns a 32-bit integer calculated as a checksum of the provided string. This function uses CRC32-C to create the result. If a NULL value is provided, this function returns a constant value. **Parameters** * `string`: The string to calculate the checksum for. **Returns** A 32-bit integer checksum. ## CRC64 --- ```sql CRC64(string) ``` Returns a 64-bit integer calculated as a checksum of the provided string. This function uses a polynomial of `0x95AC9329AC4BC9B5` to create the result. If a NULL value is provided, this function returns a constant value. **Parameters** * `string`: The string to calculate the checksum for. **Returns** A 64-bit integer checksum. --- # Managing String Case AMPS provides the `UPPER` and `LOWER` functions to produce a string in a specific case. This can be useful when constructing fields, or when an expression needs case-insensitive comparisons against a group of values using the `IN` clause. As described above in [String Comparison Functions](string-comparison-functions), AMPS provides `INSTR_I` and `STREQUAL_I` functions for performing case-insensitive comparisons. In some cases, particularly when using strings with the `IN` clause, it is more efficient to simply convert the string to a known case. The `UPPER` and `LOWER` functions are not unicode-aware; these functions will not produce the correct data when used with multibyte characters. ## UPPER --- ``` sql UPPER(string_to_transform) ``` Returns the input string, transformed to uppercase. This function is not unicode aware. **Parameters** * `string_to_transform`: The string to transform. **Returns** The uppercase version of the string. **Example** You might compare an incoming field of unknown case to a set of known values as follows: ```sql UPPER(/ticker) IN ('MSFT', 'IBM', 'RHAT', 'DIS') ``` ## LOWER --- ``` sql LOWER(string_to_transform) ``` Returns the input string, transformed to lowercase. This function is not unicode aware. **Parameters** * `string_to_transform`: The string to transform. **Returns** The lowercase version of the string. **Example** You might compare an incoming field of unknown case to a set of known values as follows: ```sql LOWER(/ticker) IN ('msft', 'ibm', 'rhat', 'dis') ``` --- # Message Functions AMPS includes functions that can be used to refer to the current message being processed. When used in view construction or aggregate definition, these functions refer to the _incoming message_ that is prompting the update to the view or aggregate, not to the constructed message that is the result of the update. For example, a `Field` like this in a view projection: ```xml showLineNumbers ... TOPIC_NAME() AS /theTopic ... ``` will return the topic name of the topic that prompted the update to the view, _not_ the name of the view itself. ## MESSAGE_SIZE --- ```sql MESSAGE_SIZE() ``` Returns the size of the payload of the current message, in bytes. **Parameters** None. **Returns** The size of the message payload in bytes. #### Available For All messages. ## CORRELATION_ID --- ```sql CORRELATION_ID() ``` Returns the correlation ID of the current message as a string. **Parameters** None. **Returns** The correlation ID of the current message, or `NULL` if there is no correlation ID. #### Available For All messages. ## LAST_UPDATED --- ```sql LAST_UPDATED() ``` Returns a timestamp for the last time that a message in the SOW was updated, as a double. For a subscription (including the subscription part of a `sow_and_subscribe` command), the `LAST_UPDATED` value will be the current timestamp. This function is most useful for queries of a topic in the SOW. Notice that this field is set based on when the local instance has updated the message. For replicated topics, this means that a given message will have different values on different instances. **Parameters** None. **Returns** A timestamp for the last time the message was updated. #### Available For Queries of a SOW topic. ## TOPIC_NAME --- ```sql TOPIC_NAME() ``` Returns the topic name for the message currently being processed. When used in a filter for a message being delivered from a queue that has multiple underlying topics, returns the name of the underlying topic **Parameters** None. **Returns** A string containing the topic name for the message. #### Available For All messages ## BOOKMARK --- ```sql BOOKMARK() ``` Returns the bookmark for the current message, if one is available. Messages retrieved from a SOW topic using a query return `NULL` for `BOOKMARK`, since the SOW does not store the bookmark of a message. Bookmarks are assigned using a combination of an identifier derived from the client name and a sequence number. When working with bookmarks, 60East recommends treating bookmarks as opaque identifiers. In particular, bookmarks are not guaranteed to sort in any particular order between different publishers. AMPS only assigns bookmarks when a message is stored in the transaction log. Messages that are not in the transaction log do not have bookmarks assigned. **Parameters** None. **Returns** The bookmark for the current message. #### Available For * Subscriptions to a transaction-logged topic * Bookmark subscriptions to a transaction-logged topic * SOW delete by filter for a transaction-logged topic ## LAST_LEASED --- ```sql LAST_LEASED() ``` For a message in a queue, returns a timestamp for the last time this message was leased from this instance, as a double. Returns `NULL` for a message that is not in a queue. Notice that this timestamp is set based on when the local instance leased the message. This counter is reset when the instance restarts. **Parameters** None. **Returns** A timestamp for the last time this message was leased. #### Available For * Queries of a message queue * Subscription to a message queue * SOW delete by filter for a message queue ## LEASE_COUNT --- ```sql LEASE_COUNT() ``` For a message in a queue, returns the number of times the message has been leased from this instance as a long. Returns `NULL` for a message that is not in a queue. Notice that this counter is set based on leases from the local instance. This counter is reset when the instance restarts, and does not track leases from other instances. **Parameters** None. **Returns** The number of times the message has been leased, as a long. Returns `NULL` for a message that is not in a queue. #### Available For * Queries of a message queue * Subscription to a message queue * SOW delete by filter for a message queue ## SOW_KEY --- ```sql SOW_KEY() ``` For a message in a SOW topic, returns the current SOW key of the message. This will typically be the AMPS-generated identifier for the message. For topics that use explicit publisher-provided keys (that is, where the SOW key must be set on the header of the publish command rather than derived from the data), this will be the publisher-provided key. This function is designed for use in enrichment. In a query, subscription, or delete command, using the `SowKeys` header with the key or keys of interest is more efficient. **Parameters** None. **Returns** The SOW key that AMPS uses for the message, either as provided by the publisher or as used internally by AMPS, as a string. #### Available For * Queries or enrichment of a SOW topic * Subscription to a SOW topic ## SOW_KEY_HASH --- ```sql SOW_KEY_HASH() ``` For a message in a SOW topic, returns the AMPS-generated identifier for the message. For topics that do not use explicit publisher-provided keys, this function will return the same value as `SOW_KEY`, but as an unsigned long. For topics that use explicit publisher-provided keys (that is, where the SOW key must be set on the header of the publish command rather than derived from the data), this will return the hash value AMPS uses internally for the message rather than the publisher-provided key. This function is designed for use in enrichment. In a query, subscription, or delete command, using the `SowKeys` header with the key or keys of interest is more efficient. **Parameters** None. **Returns** The SOW key hash that AMPS uses internally for the message, as an unsigned long. #### Available For * Queries or enrichment of a SOW topic * Subscription to a SOW topic ## LAST_READ --- ```sql LAST_READ() ``` Returns a timestamp for the last time that this message was read from the SOW. This function only returns a value for messages in a topic in the SOW. This value is not persisted across restarts or shared across instances. **Parameters** None. **Returns** A double indicating the last time that this message was read from the SOW on the local instance. This value is not persisted across restarts or shared across instances. #### Available For * Queries of a SOW topic --- # Numeric and Math Functions AMPS includes the following functions for working with numbers. ### ABS --- ```sql ABS(number) ``` Returns the absolute value of a number. **Parameters** * `number`: The number to find the absolute value of. **Returns** The absolute value of the number. **Example** The following filter will be TRUE when the difference between `/a` and `/b` is greater than 5, regardless of whether `/a` or `/b` is larger. ```sql ABS(/a - /b) > 5 ``` ### GREATEST --- ```sql GREATEST(number, [number, ...]) ``` Returns the largest of the provided values. **Parameters** * `number`: A list of numbers to compare. **Returns** The largest of the provided values. **Example** ```sql GREATEST(/requestedQty, /availableQty, /reserveQty) ``` ### LEAST --- ```sql LEAST(number, [number, ...]) ``` Returns the smallest of the provided values. **Parameters** * `number`: A list of numbers to compare. **Returns** The smallest of the provided values. **Example** ```sql LEAST(/requestedQty, /availableQty, /reserveQty) ``` ### CEILING --- ```sql CEILING(number) ``` Returns the value rounded upward to the next greatest integer. **Parameters** * `number`: The number to round. **Returns** The rounded-up integer as a double. Returns an integer unchanged (still converts to double). **Example** ```sql CEILING(/estimatedDelta) ``` ### FLOOR --- ```sql FLOOR(number) ``` Returns the value rounded downward to the next lower integer. **Parameters** * `number`: The number to round. **Returns** The rounded-down integer as a double. Returns an integer unchanged (still converts to double). **Example** ```sql FLOOR(/estimatedReturn) ``` ### EXP --- ```sql EXP(exponent) ``` Returns *e* raised to the power of the provided exponent. **Parameters** * `exponent`: The exponent to use. **Returns** The value of *e* raised to the power of the exponent. **Example** ```sql EXP(3.14) ``` ### LN --- ```sql LN(number) ``` Returns the natural logarithm of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The natural logarithm of the number. Returns `NaN` if `number` is outside the domain of logarithmic functions. **Example** ```sql LN(/time) ``` ### LOG2 ```sql LOG2(number) ``` --- Returns the base-2 logarithm of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The base-2 logarithm of the number. Returns `NaN` if `number` is outside the domain of logarithmic functions. **Example** ```sql LOG2(/recordSize) ``` ### LOG10 ```sql LOG10(number) ``` --- Returns the base-10 logarithm of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The base-10 logarithm of the number. Returns `NaN` if `number` is outside the domain of logarithmic functions. **Example** ```sql LOG10(/displayData) ``` ### POWER --- ```sql POWER(base, exponent) ``` Returns the value of `base` raised to the power of `exponent`. **Parameters** * `base`: The base number. * `exponent`: The exponent. **Returns** The value of the base raised to the power of the exponent. POSIX-compliant edge cases. **Example** ```sql POWER(/number, 3) ``` ### SQRT --- ```sql SQRT(number) ``` Returns the square root of the provided number. **Parameters** * `number`: The number to find the square root of. **Returns** The square root of the number. Returns `NaN` if `number` is outside the domain of the square root function. **Example** ```sql SQRT(2304) ``` ### SIGN --- ```sql SIGN(number) ``` Returns the sign of the provided number. **Parameters** * `number`: The number to check. **Returns** If the number is less than `0`, returns `-1`. If the number is greater than `0`, returns `1`. Otherwise, the number is `0` and the function returns `0`. **Example** ```sql SIGN(/result) ``` ### ROUND --- ```sql ROUND(number, [decimal_places]) ``` Returns a number rounded to the specified number of decimal places. **Parameters** * `number`: The number to round. * `decimal_places`: Optional. The number of decimal places to round to. Defaults to 0. Can be positive or negative. **Returns** The rounded number. **Example** You could use the following expression in a view to limit the precision of the `/price` field of the source topic to 2 decimal places. ```sql ROUND(/price, 2) AS /price ``` ### WIDTH_BUCKET --- ```sql WIDTH_BUCKET(expression, min, max, bucket_count) ``` The `bucket_count` argument specifies the number of buckets to create over the range defined by `min` and `max`. `min` is inclusive, while `max` is not. The value from `expression` is assigned to a bucket, and the function returns a corresponding bucket number. **Parameters** * `expression`: The expression to evaluate. * `min`: The minimum value of the range. * `max`: The maximum value of the range. * `bucket_count`: The number of buckets to create. **Returns** The bucket number. When `expression` falls outside the range of buckets, the function returns either `0` or `max + 1`, depending on whether `expression` is lower than `min` or greater than or equal to `max`. **Example** ```sql WIDTH_BUCKET(/percentage, 0, 100, 10) AS /displayBar ``` ## Trigonometric Functions ### ACOS --- ```sql ACOS(number) ``` Returns the arccosine of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The arccosine of the number. Returns `NaN` if `number` is outside the domain of the arccosine function. **Example** ```sql ACOS(/x) ``` ### ASIN --- ```sql ASIN(number) ``` Returns the arcsine of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The arcsine of the number. Returns `NaN` if `number` is outside the domain of the arcsine function. **Example** ```sql ASIN(/x) ``` ### ATAN --- ```sql ATAN(number) ``` Returns the arctangent of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The arctangent of the number. **Example** ```sql ATAN(/x) ``` ### ATAN2 --- ```sql ATAN2(y, x) ``` Returns the arctangent of the provided numbers. POSIX-compliant edge cases. **Parameters** * `y`: The y-coordinate. * `x`: The x-coordinate. **Returns** The arctangent of the numbers. **Example** ```sql ATAN2(/y,/x) ``` ### COS --- ```sql COS(number) ``` Returns the cosine of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The cosine of the number. **Example** ```sql COS(/x) ``` ### COSH --- ```sql COSH(number) ``` Returns the hyperbolic cosine of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The hyperbolic cosine of the number. **Example** ```sql COSH(/x) ``` ### COT --- ```sql COT(number) ``` Returns the cotangent of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The cotangent of the number. Returns `NaN` if `number` is 0. **Example** ```sql COT(/x) ``` ### DEGREES --- ```sql DEGREES(radians) ``` Returns the provided number converted from radians to degrees. **Parameters** * `radians`: The number in radians. **Returns** The number converted to degrees. **Example** ```sql DEGREES(/rad) ``` ### RADIANS --- ```sql RADIANS(degrees) ``` Returns the provided number converted from degrees to radians. **Parameters** * `degrees`: The number in degrees. **Returns** The number converted to radians. **Example** ``` sql RADIANS(/deg) ``` ### SIN ```sql SIN(number) ``` --- Returns the sine of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The sine of the number. **Example** ```sql SIN(/x) ``` ### SINH ```sql SINH(number) ``` --- Returns the hyperbolic sine of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The hyperbolic sine of the number. **Example** ```sql SINH(/x) ``` ### TAN --- ```sql TAN(number) ``` Returns the tangent of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The tangent of the number. **Example** ```sql TAN(/x) ``` ### TANH --- ```sql TANH(number) ``` Returns the hyperbolic tangent of the provided number. **Parameters** * `number`: The number to operate on. **Returns** The hyperbolic tangent of the number. **Example** ```sql TANH(/x) ``` --- # Replacing Text in Strings AMPS provides a pair of functions, `REPLACE` and `REGEXP_REPLACE`, that replace text within strings. ## REPLACE --- ```sql REPLACE(string_to_transform, string_to_match, replacement_text) ``` Returns the input string, with all occurrences of the `string_to_match` replaced with the `replacement_text`. This function does a literal match of the string to be replaced. **Parameters** * `string_to_transform`: The string to transform. * `string_to_match`: The string to search for. * `replacement_text`: The text to replace the matched string with. **Returns** The transformed string. **Example** ```sql REPLACE('fandango', 'dan', 'din') == 'fandingo' ``` ## REGEXP_REPLACE --- ```sql REGEXP_REPLACE(string_to_transform, pattern_to_match, replacement_text) ``` Returns the input string, with all occurrences of the `pattern_to_match` replaced with the `replacement_text`. This function uses a PCRE pattern to find the string to be replaced. **Parameters** * `string_to_transform`: The string to transform. * `pattern_to_match`: The PCRE pattern to search for. * `replacement_text`: The text to replace the matched string with. **Returns** The transformed string. **Example** ```sql REGEXP_REPLACE('fandango','n.*n', 'r') == 'fargo' ``` --- # String Comparison Functions AMPS includes several types of string comparison operators: * **Case-Sensitive Exact Matches** - The `IN`, `=`, `BEGINS WITH`, `ENDS WITH`, and `INSTR` operators do literal matching on the contents of a string. These operators are case-sensitive. * **Case-Insensitive Exact Matches** - AMPS also provides two case-insensitive operators: `INSTR_I`, a case-insensitive version of `INSTR`, and a case-insensitive equality operator, `STREQUAL_I`. * **Regular Expression Matches** - AMPS also provides full regular expression matching using the `LIKE` operator, described in [Regular Expressions](../amps-expressions/regular-expressions). The `=` operator tests whether a field exactly matches the literal string provided. ```sql /status = 'available' /orderId = 'F327AC' ``` `BEGINS WITH` and `ENDS WITH` test whether a field begins or ends with the literal string provided. The operators return `TRUE` or `FALSE`. ```sql /Department BEGINS WITH ('Engineering') /path NOT BEGINS WITH ('/public/dropbox') /filename ENDS WITH ('txt') /price NOT ENDS WITH ('99') ``` AMPS allows you to use set comparisons with `BEGINS WITH` and `ENDS WITH`. In this case, the filter matches if the string in the field begins or ends with any of the strings in the set. ```sql /Department BEGINS WITH ('Engineering', 'Research', 'Technical') /filename ENDS WITH ('gif', 'png', 'jpg') ``` The `INSTR` operator allows you to check to see if one string occurs within another string. For this operator, you provide two string values. If the second string occurs within the first string, `INSTR` returns the position at which the second string starts, or 0 if the second string does not occur within the first string. Notice that the first character of the string is 1 (not 0). For example, the expression below tests whether the string `critical` occurs within the `/eventLevels` field. ```sql INSTR(/eventLevels, "critical") != 0 ``` AMPS also provides `INSTR_I` and `STREQUAL_I` functions for performing case-insensitive comparisons. ```sql STREQUAL_I(/couponCode, 'QED') == 1 INSTR_I(/symbolList, 'MSFT') != 0 ``` Following are the list of string comparison functions and operators in AMPS. ## = (Equals) --- ```sql string1 = string2 ``` *Case-sensitive.* Returns true if `string1` is identical to `string2`. **Parameters** * `string1`: The first string. * `string2`: The second string. **Returns** `true` if the strings are identical, `false` otherwise. **Example** ```sql /state = 'Ohio' ``` ## BEGINS WITH ```sql string BEGINS WITH (substring1, [substring2, ...]) ``` *Case-sensitive.* Returns true if `string` begins with any of the substrings in the list. **Parameters** * `string`: The string to check. * `substring1`: The first substring to check for. * `substring2`: Optional. Additional substrings to check for. **Returns** `true` if the string begins with one of the provided substrings, `false` otherwise. **Example** ```sql /state BEGINS WITH ('North', 'South') ``` ## ENDS WITH ```sql string ENDS WITH (substring1, [substring2, ...]) ``` *Case-sensitive.* Returns true if `string` ends with any of the substrings in the list. **Parameters** * `string`: The string to check. * `substring1`: The first substring to check for. * `substring2`: Optional. Additional substrings to check for. **Returns** `true` if the string ends with one of the provided substrings, `false` otherwise. **Example** ```sql /state ENDS WITH ('Dakota', 'Carolina') ``` ## INSTR ```sql INSTR(string, substring) ``` *Case-sensitive.* Returns the 1-based position at which `substring` starts in `string`, or `0` if `substring` does not occur within `string`. This function is not unicode-aware. **Parameters** * `string`: The string to search within. * `substring`: The substring to search for. **Returns** The 1-based starting position of the substring, or `0` if not found. **Example** ```sql INSTR(/state, 'i') != 0 ``` ## INSTR_I ```sql INSTR_I(string, substring) ``` *Case-insensitive.* Returns the 1-based position at which `substring` starts in `string`, or `0` if `substring` does not occur within `string`. This function is not unicode-aware. **Parameters** * `string`: The string to search within. * `substring`: The substring to search for. **Returns** The 1-based starting position of the substring, or `0` if not found. **Example** ```sql INSTR_I(/state, 'i') != 0 ``` ## STREQUAL_I ```sql STREQUAL_I(string1, string2) ``` *Case-insensitive.* Returns true if, when both strings are transformed to the same case, `string1` is identical to `string2`. This function is not unicode-aware. **Parameters** * `string1`: The first string. * `string2`: The second string. **Returns** `true` if the strings are identical (case-insensitively), `false` otherwise. **Example** ```sql STREQUAL_I(/state, 'oHIO') != 0 ``` ## LENGTH ```sql LENGTH(string) ``` Returns the length of the provided string. **Parameters** * `string`: The string to measure. **Returns** The length of the string. Returns `NULL` when length is 0. **Example** ```sql LENGTH(/streetAddress) > 50 ``` --- # String Manipulation Functions AMPS provides a set of functions for manipulating strings. ## SUBSTR --- ```sql SUBSTR(string, starting_position, [length]) ``` Returns a portion of the input string, starting at the `starting_position` and ending after the specified `length`. **Parameters** * `string`: The string to process. * `starting_position`: The 1-based position to start the substring. A negative number counts backward from the end of the string. * `length`: Optional. The length of the substring. If not provided, returns the rest of the string. **Returns** The extracted substring. If `starting_position` \> length of `string`, returns `NULL`. If `starting_position` \<= negative length of `string` or `starting_position == 0`, returns `string`. **Example** The following expressions are all `TRUE`: ```sql SUBSTR("fandango", 4) == "dango" SUBSTR("fandango", 1) == "fandango" SUBSTR("fandango", -2) == "go" SUBSTR("fandango", 1, 3) == "fan" ``` ## TRIM --- ```sql TRIM(string, [characters_to_trim]) ``` Returns the input string, with all leading and trailing characters in the set of `characters_to_trim` removed. **Parameters** * `string`: The string to transform. * `characters_to_trim`: Optional. The set of characters to trim. Defaults to a space. **Returns** The trimmed string. **Example** The following expressions are all `TRUE`: ```sql TRIM(" Lancelot ") == "Lancelot" TRIM(" aabaa ", "a ") == "b" TRIM("aaa", "a") IS NULL ``` ## LTRIM --- ```sql LTRIM(string, [characters_to_trim]) ``` Returns the input string, with all leading characters in the set of `characters_to_trim` removed. **Parameters** * `string`: The string to transform. * `characters_to_trim`: Optional. The set of characters to trim. Defaults to a space. **Returns** The left-trimmed string. **Example** The following expressions are all `TRUE`: ```sql LTRIM(" Lancelot ") == "Lancelot " LTRIM(" aabaa ", "a ") == "baa " LTRIM("aaa", "a") IS NULL ``` ## RTRIM --- ```sql RTRIM(string, [characters_to_trim]) ``` Returns the input string, with all trailing characters in the set of `characters_to_trim` removed. **Parameters** * `string`: The string to process. * `characters_to_trim`: Optional. The set of characters to trim. Defaults to a space. **Returns** The right-trimmed string. **Example** The following expressions are all `TRUE`: ```sql RTRIM(" Lancelot ") == " Lancelot" RTRIM(" aabaa ", "a ") == " aab" RTRIM("aaa", "a") IS NULL ``` ## LEFT --- ```sql LEFT(string, number_of_characters) ``` Returns the leftmost `number_of_characters` from the provided string. **Parameters** * `string`: The string to process. * `number_of_characters`: The number of characters to return from the left. **Returns** The leftmost characters of the string. If `number_of_characters` is greater than or equal to the length of `string`, returns `string`. **Example** ```sql LEFT("fandango", 3) == "fan" ``` ## RIGHT --- ```sql RIGHT(string, number_of_characters) ``` Returns the rightmost `number_of_characters` from the provided string. **Parameters** * `string`: The string to process. * `number_of_characters`: The number of characters to return from the right. **Returns** The rightmost characters of the string. If `number_of_characters` is greater than or equal to the length of `string`, returns `string`. **Example** ```sql RIGHT("fandango", 2) == "go" ``` ## REVERSE --- ```sql REVERSE(string) ``` Returns the provided string in reverse. **Parameters** * `string`: The string to process. **Returns** The reversed string. **Example** ```sql REVERSE("fandango") == "ognadnaf" ``` --- # Typed Value Creation AMPS includes functions for explicitly constructing constant values of various types. As stated in [AMPS Data Types](/docs/amps-user-guide/amps-expressions/amps-data-types.md), AMPS operators and functions will automatically convert values to compatible types, but these functions can be used for explicit typing, value construction, and type casting. ## FALSE_VALUE --- ```sql FALSE_VALUE() ``` Returns a boolean false value. This function is most useful for constructing values in message types that have a distinct type for boolean values. In the AMPS expression language, false is equivalent to a literal `0`. **Parameters** None. **Returns** A boolean false value. **Example** ```sql IF(/price < 0.8 * /recentPeak, TRUE_VALUE(), FALSE_VALUE()) AS /possibleDowntrend ``` ## TRUE_VALUE --- ```sql TRUE_VALUE() ``` Returns a boolean true value. This function is most useful for constructing values in message types that have a distinct type for boolean values. In the AMPS expression language, true is typically represented with a literal `1`. **Parameters** None. **Returns** A boolean true value. **Example** ```sql IF(/price > 1.2 * /recentPeak, TRUE_VALUE(), FALSE_VALUE()) AS /possibleUptrend ``` ## NAN_VALUE --- ```sql NAN_VALUE() ``` Returns a NaN (not a number) value. **Parameters** None. **Returns** A NaN value. **Example** ```sql IF(/number IS NULL, NAN_VALUE(), /number) ``` ## CHAR_VALUE --- ```sql CHAR_VALUE(integer) ``` Returns the character (byte) for the integer provided (0-255). This function is most useful for constructing values in message types that have a distinct type for char values. In the AMPS expression language, a single character value is equivalent to a string constructed with an escape, and constructing a string literal is more efficient. That is, `'\x01'` is more efficient in a filter or field construction than `CHAR_VALUE(1)`. However, to construct a character based on a field, use `CHAR_VALUE`. **Parameters** * `integer`: An integer between 0 and 255. **Returns** The character corresponding to the integer. If `integer` is less than 0 or greater than 255, `'?'` is returned. **Example** ```sql CHAR_VALUE(/code) ``` --- # Configuration Options Quick Reference Below you'll find links to detailed pages for each configuration element. Click on any of the cards to quickly navigate to the specific configuration options you need for your AMPS instance.

    Actions

     Used to configure AMPS to run scheduled tasks or respond to events.

    Authentication

     Used to specify the module to use for validating user identity.

    Conflated Topics

     Used to define a Conflated Topic in a SOW.

    Entitlement

     Used to specify the module to use for validating permissions to resources within AMPS.

    Instance-Level Configuration

     Elements used to define settings that apply to the entire AMPS instance.

    Logging

     Used to define logging targets for messages.

    Message Types

     Used to define the message types supported by the AMPS instance.

    Modules

     Used to load, configure and define any plug-in modules.

    Monitoring / Galvanometer

     Used to control the behavior of the administration server and statistics collection.

    Protocols

     Used to define the format of the commands that clients use to communicate with the server.

    Queues

     Used to define a Queue in a SOW.

    Replication

     Used to define the replication of messages between instances.

    Replication Destinations

     Used to define outgoing replication connections.

    Replication Transports

     Used to define incoming replication connections.

    State of the World (SOW)

     Used to define the configuration for the AMPS SOW.

    Topics

     Used to define a Topic in a SOW.

    Transaction Log

     Used to keep a journal of messages published to an AMPS instance.

    Transports

     Used to define how AMPS communicates with publishers and subscribers, and how it accepts replication connections.

    Views

     Used to define a View in a SOW. --- # Configuring AMPS When the AMPS server starts, it reads the configuration file, fully expands any [environment variables](/docs/shared/units-intervals-and-environment#environment-variables-in-amps-configuration), and fully processes any [include directives](/docs/amps-user-guide/configuring-amps/include) in the file. The AMPS server stores this configuration file in memory, and the fully expanded version of the file is what is provided by the Administrative Console and Galvanometer. Changes made to the configuration file on disk after the server has started will not take effect. To apply updates to the configuration, you must restart the AMPS server. All AMPS configuration parameters are detailed in this guide and can be found in their appropriate sections. This section provides a walkthrough of the minimal AMPS configuration file, explains the available unit abbreviations, and demonstrates how to use environment variables to set AMPS behavior at startup. It also covers command-line options for validating and expanding configuration files, or for generating a simple configuration outline, and options for composing an AMPS configuration from other files. --- # Getting Started with AMPS Configuration This section describes a typical process for developing an AMPS configuration file. 1. **Add mandatory information** An AMPS configuration starts with an `AMPSConfig` element at the outermost level of the configuration. Every AMPS instance must have a `Name`. This name identifies the instance, and must be unique within a set of instances. In this case, we will define the name `test-AMPS-1` for the instance, as shown below: ```xml showLineNumbers test-AMPS-1 ``` With this configuration, AMPS will start but will not allow any incoming connections to process messages. This instance also does not provide any monitoring or logging. --- 2. **Describe how AMPS will receive connections from applications** AMPS has no default settings for receiving commands from applications or for delivering messages to applications. To enable applications to connect to AMPS, specify a `Transports` element that defines one or more incoming `Transport` definitions. These are described in detail in the [Configuring Transports](/docs/amps-user-guide/transports/configuring-transports) section. For example, it's common to define one transport for application connections, and one transport for the use of web applications (JavaScript / TypeScript) and data queries or subscriptions from the Galvanometer admin tool. In this case we'll allow browser-based applications (such as Galvanometer) to connect to port `9008`, and applications that use the other client libraries to connect to port `9007`. After this step, the configuration might look something like this: ```xml showLineNumbers test-AMPS-1 any-tcp tcp 9007 json amps any-websocket tcp 9008 json websocket ``` --- 3. **Enable administrative monitoring** AMPS does not enable administrative monitoring by default. However, all instances of AMPS should enable this, as described in the [Monitoring AMPS](../monitoring) section. For sample purposes, we will use port 8085 to host the administrative interface and Galvanometer. This configuration also tells Galvanometer to use the `any-ws` transport for the embedded JavaScript client. ```xml showLineNumbers test-AMPS-1 any-tcp tcp 9007 json amps any-ws tcp 9008 json websocket 8085 any-ws ``` --- 4. **Enable error and event logging** Last, but not least, it's important to be notified of any errors or important events in AMPS. AMPS does not enforce any default for this logging, but instead requires this logging to be explicitly configured. For a production instance, AMPS would typically log messages to a file. For a quick sample, though, we can just log anything at `warning` level or above to `stdout`. ```xml showLineNumbers test-AMPS-1 any-tcp tcp 9007 json amps any-ws tcp 9008 json websocket 8085 any-ws stdout warning ``` --- 5. **Define advanced messaging behavior** The configuration in the four sections above is all you need for basic subscribe and publish functionality. For more advanced functionality, AMPS can be configured to provide the necessary capabilities. For example, to configure a last-value-cache for a topic, you would define a `Topic` in the [State of the World (SOW)](/docs/amps-user-guide/sow/configuring-a-sow). To enable full, message-by-message replay for a topic or set of topics, you would define a [Transaction Log](/docs/amps-user-guide/txlog/configuring-a-transaction-log). All instances of AMPS, regardless of the application they support, should follow this minimum outline: define the instance `Name`, the instance `Transports`, define monitoring through the `Admin` console, and specify the `Logging` for the instance. Most applications also take advantage of the advanced messaging capabilities of AMPS, as described in detail in this guide. The [Scenario and Feature Reference](/docs/intro-guide/feature_guide) section in the [Introduction to AMPS](/docs/intro-guide/intro) guide provides helpful starting points for different scenarios. --- # Including External Files For production applications, AMPS configuration files can become large and complicated. In many cases, different instances of an AMPS server need to reuse the same definitions. For example, both servers in a High-Availability pair may need to use the same queue and SOW definitions. To help you manage complicated configurations and more easily keep configuration consistent on different servers, AMPS allows you to include external files in the configuration file by using the `Include` directive. For example, you could use this for a High-Availability pair to include a file that defines the queue, transaction log, and topic definitions. Both instances could include exactly the same file for those definitions, while having different instance names and port numbers. When AMPS loads a configuration file that contains an Include directive, AMPS follows this process: * Load and parse the configuration file * If the file contains any `Include` directives, load and parse the files specified by those directives. If the included files contain Include directives, load and parse the files specified by those directives (and so forth until all Include directives have been processed). * Once all files have been loaded and parsed, replace the Include directives in the original files with the parsed files. AMPS does not process the configuration file until all of the `Include` directives have been resolved and the files have been parsed. A file may not be included by any file that it includes, or it is impossible for AMPS to complete the parsing process. Since each file is individually parsed, XML entities defined in a file are not defined for the files that are listed within the `Include` tag by that file. To make it easier to identify which elements of the complete AMPS configuration file have been inserted through the `Include` mechanism, AMPS can include comments in the assembled file that indicate the source file for configuration elements. By default, this feature is off, and XML content is included verbatim. To change the default for the instance, use the `ConfigIncludeCommentDefault` configuration element to enable comments, by default, for every `Include` in the instance. To override commenting behavior for an individual `Include`, use the `comment` attribute. ## Example Consider a configuration file with the following `Logging` element defined: ```xml showLineNumbers ... filetarget.xml ... ``` After parsing the configuration file, AMPS loads and parses the `filetarget.xml` file and replaces the `Include` element with the contents of that file. Suppose `filetarget.xml` contains the following `Target` directive: ```xml showLineNumbers file /var/log/amps-log-%n.log info ``` The configuration that AMPS uses will be effectively the same as if the configuration file contained the following XML: ```xml showLineNumbers ... file /var/log/amps-log-%n.log info ... ``` :::tip Include directives are processed at startup, when AMPS loads the configuration file. Changing the included files after AMPS starts has no effect. ::: --- # Instance-Level Configuration The following sections outline the configuration elements used to define settings that apply to the entire AMPS instance. ## AMPS Process Options Described below are the options available at the instance level of the AMPS configuration file. Expand each option for more details.
    `Name` (required) This element defines the name of your AMPS instance. The instance name is used to uniquely identify this instance for replication purposes, to generate file names for use by the AMPS instance, that is shown in log statements and used for other administrative purposes. 60East recommends that the name be short and meaningful, and that each instance in your AMPS installation have a distinct name. When creating a name, the name should not contain special characters such as spaces, path separator characters (`/` or `\\`), or characters that will be interpreted by the Linux shell (`$` or `~`). 60East recommends that the name of the instance stay the same for as long as the application will be connected to the same replication partners or retain transaction log data. This name is used in the files created by the instance as part of the transaction log, in the replication path for replicated instances, and in the client names created for replication connections. If your AMPS installation will use replication, the `Name` of each instance must be unique within the set of replicated instances. This element is required, and there is no default. Example: ```xml showLineNumbers .... AMPS Sample-AMPS .... ```
    `Group` Identifies the replication group for this instance. If no `Group` element is present, the replication group for this instance is set to the `Name` of the instance. Set the group parameter when being able to refer to a set of instances that should be treated as identical for replication purposes makes replication configuration easier. Instances that have the same `Group` value should be intended to be exactly equivalent for the purposes of message contents and failover. If two instances are intended to be treated differently for the purposes of failover or replication, or are intended to have different contents, they should be given different `Group` names. Replication passthrough uses the group name to specify which instances to provide passthrough for. See the [Replicating Messages Between Instances](/docs/amps-user-guide/replication) section for a discussion of replication, including passthrough. When the `Group` is not set explicitly, the `Group` defaults to the instance `Name`. Example: ```xml showLineNumbers .... AMPS Sample-AMPS .... ```
    `ProcessName` Specifies the process name to set for this instance. When this element is present, AMPS uses the name specified as the process name. Otherwise, the process name uses the default set by Linux, which is the executable name (`ampServer` or `ampServer-compat` unless the executable has been renamed). This element is most useful for systems that host multiple AMPS instances and want to be able to quickly tell the instances apart based on the process name. This element is optional. If not present, the AMPS executable does not change the process name.
    `Description` This element is used to provide a description of the AMPS instance for monitoring tools (including the AMPS Galvanometer). AMPS provides the contents of this element in the admin interface, but does not use the description for any other purpose.
    `Environment` This element is used to provide information about the environment of the AMPS instance to monitoring tools (including the AMPS Galvanometer). AMPS provides the contents of this element in the admin interface, but does not use this element for any other purpose.
    `SuggestedMinimumVersion` The suggested minimum AMPS version to use this configuration file. If the AMPS instance that loads this configuration file has a version number less than the suggested minimum version, AMPS issues a warning. This option can be useful when upgrading a set of AMPS instances, or when the AMPS instance will see improved performance from a particular feature. For example, an application that will run correctly without hash indexes, but would see improved performance with hash indexes, could provide a `SuggestedMinimumVersion` of 4.3.1.0. Default: When no value is provided, AMPS does not check the configuration file against the version number of the instance.
    `RequiredMinimumVersion` The required minimum AMPS version to use this configuration file. If the AMPS instance that loads this configuration file has a version number less than the suggested minimum version, AMPS issues an error and will not start. This option can be useful for enforcing an upgrade on a set of AMPS instances, or when the AMPS instance must support a particular feature. For example, an application that uses message queues could provide a `RequiredMinimumVersion` of 5.0. Default: When no value is provided, AMPS does not check the configuration file against the version number of the instance.
    `ConfigIncludeCommentDefault` Sets the default for how `Include` directives indicate the source of the content. When this option is set to `true` or `enabled`, content inserted through an `Include` directive is surrounded by comments indicating the source of the content. Default: `false`, which specifies that AMPS will not surround included content with comments.
    `ConfigCycleDetectionThreshold` Sets the maximum size to allow for an expanded configuration file. This setting is intended to prevent cycles of include files (for example, where file A includes file B and file B includes file A) from consuming all of the memory on the system before failing. Default: `5MB`
    `UserDefinedFunctions` Use the `UserDefinedFunctions` element to register custom scalar functions that can be used anywhere AMPS evaluates an expression (filters, projections, enrichment, actions, views, and so on). AMPS loads these functions after loading the [`Modules`](../optional-modules) section, so each function references a module that contains the compiled implementation. This element is optional. When present, it contains one or more `Function` entries: | Element | Description | | ------- | ----------- | | `Function/Name` | Name exposed to expression authors. Names are case-insensitive at runtime, but defining a clear, uppercase name makes expressions easier to read. | | `Function/Module` | The `Name` of a module defined in the `` section. AMPS loads that shared object before resolving the function. | | `Function/Symbol` | Exported symbol that implements the UDF inside the module. Declare the function with `extern "C"` (for C++) so the symbol name is not mangled. | | `Function/ParameterCount` | Number of arguments the UDF expects. Omit this element (or leave it empty) to accept a variable number of arguments. Internally AMPS treats a missing value as `AMPS_UDF_VARIADIC_PARAMETER_COUNT`. | AMPS validates each function while the configuration loads. If the module cannot be found, or the symbol does not exist in that module, AMPS reports a configuration error and stops startup. Example: ```xml pricing-udf libpricing_functions.so FWD_PRICE pricing-udf amps_udf_forward_price 3 NORMALIZE_TAGS pricing-udf amps_udf_normalize_tags ``` After AMPS starts with this configuration, both `FWD_PRICE()` and `NORMALIZE_TAGS()` are available in expressions. Reloading or changing UDF implementations requires restarting the instance so the module and symbol table can be refreshed.
    ## Authentication and Entitlements Management AMPS authentication and entitlements are managed for each `Transport.` You can set the default to use for all transports at the instance level. See the [Configuring Authentication](/docs/amps-user-guide/securing/configuring-authentication) section for details on setting identity management for the instance, and the [Configuring Entitlement](/docs/amps-user-guide/securing/configuring-entitlement) section for details on setting permissions for the instance. ## Slow Client Policies AMPS includes a set of parameters that specify how the instance should manage slow clients. Sometimes, AMPS can publish messages faster than an individual client can consume messages, particularly in applications where the pattern of messages includes "bursts" of messages. Clients that are unable to consume messages faster or equal to the rate messages are being sent to them are ”slow clients”. By default, AMPS queues messages for a slow client in memory to grant the slow client the opportunity to catch up. However, scenarios may arise where a client can be over-subscribed to the point that the client cannot consume messages as fast as messages are being sent to it. In particular, this can happen with the results of a large SOW query, where AMPS generates all of the messages for the query much faster than the network can transmit the messages. Slow client management is one of the ways that AMPS prevents slow clients from disrupting service to the instance. 60East recommends enabling slow client management for instances that serve high message volume or are mission critical. Slow client policies for all `Transports` in the instance are set at the root level of the configuration file. A `Transport` can override any of these settings, or choose to use the instance-wide settings. Details on slow client handling are available in the [Slow Client Management and Capacity Limits](/docs/amps-user-guide/ha/slow-client-management-and-capacity-limits) section. **Instance-Wide Options** Described below are the policies applied based on the total resource consumption of clients. Expand each option for more details.
    `MessageMemoryLimit` The total amount of memory to allocate to messages before offlining messages (that is, beginning to buffer messages to disk). This value applies to all clients. For example, setting a value of `500MB` means that all clients that this limit applies to will share `500MB` for all buffered messages to those clients. This option is specified in bytes, and accepts the standard AMPS notation (for example, `10GB` or `250MB`). Default: The default value is calculated when AMPS starts as 10% of total host memory or 10% of the amount of host memory AMPS is allowed to consume (as reported by `ulimit -m` ), whichever is lowest. For example, if a host has 250GB of memory, and `ulimit -m` for the AMPS process is unlimited, the default value for AMPS when started on that system is `25GB`.
    `MessageDiskLimit` The total amount of disk space to allocate to messages before disconnecting clients. This option is specified in bytes, and accepts the standard AMPS notation (for example, `10GB` or `250MB`). Default: `1GB` or the amount specified in the `MessageMemoryLimit`, whichever is highest.
    `MessageDiskPath` The path to use to write offline files. Default: `/var/tmp`
    **Per-Client Options** Described below are the slow client policies that are applied based on the behavior of an individual client. Expand each option for more details.
    `ClientMessageAgeLimit` The maximum amount of time for the client to lag behind. If the oldest message buffered in AMPS for a client has been held longer than this time, that client will be disconnected. This parameter is an AMPS time interval (for example, `30s` for 30 seconds, or `1h` for 1 hour). Default: No age limit
    `ClientMaxCapacity` The amount of available capacity a single client can consume. Before a client is offlined, this limit applies to the `MessageMemoryLimit`. After a client is offlined, this limit includes the `MessageDiskLimit`. This parameter is a percentage of the total limit available to the instance. This limit is set as a percentage of the total amount of capacity available. Default: `50%` in this version of AMPS. Note that versions of AMPS previous to 5.3.4 default to `100%`
    ## Minidump Settings AMPS minidumps contain information on the current state of the AMPS program execution, which is useful for support and diagnostics. AMPS will generate a minidump file on any crash event, or a minidump file can be generated at any point in time through the monitoring interface (see the [AMPS Monitoring Guide](/docs/amps-monitoring-guide)). AMPS allows you to set the directory in which minidump files will be created and the permissions mask for minidump files. Described below are the options for setting the directory and permissions mask. Expand each option for more details.
    `MiniDumpDirectory` Location to store AMPS minidump files. Default is `/tmp`. If the directory does not exist, AMPS creates the directory. The special value `disabled` configures AMPS not to produce minidumps.
    `MiniDumpFileMask` Permissions mask for minidump files. The value of the mask is an octal number (by convention, four digits) in the same format as the standard `chmod` command, and AMPS applies this mask exactly as the `chmod` command would. This is the mask AMPS will apply to the file after it's created. The file is created with the user and group that the AMPS server process runs under.
    • `0444` File is readable by owner, group, and any user.
    • `0440` File is readable by owner and members of the owner's group.
    • `0400` File is readable by owner only.
    • `0664` File is readable and writable by owner and members of the owner's group. File is readable by any user.
    • `0644` File is readable and writable by owner. File is readable by members of the owner's group and any user.
    Default: `0640` File is readable and writable by the file owner and readable by members of the owner's group.
    ```xml showLineNumbers ... /var/tmp 0644 ... ``` ## Tuning The `Tuning` section of the configuration file sets instance-level parameters for tuning the performance of AMPS. In many cases, AMPS self-tunes to take advantage of the hardware and environment. However, explicitly setting tuning parameters is sometimes necessary in cases where an AMPS instance cannot determine the best value. For example, if multiple AMPS servers are running on the same system, 60East recommends disabling NUMA. :::warning Use the `Tuning` element with care. Options in the `Tuning` element can affect AMPS performance, and the behavior of `Tuning` options may be version-specific. ::: Described below are the options for AMPS performance tuning. Expand each option for more details.
    `NUMA/Enabled` Setting this to `disabled` will turn off AMPS NUMA tuning. The default is `enabled`, which affinitizes certain AMPS threads to specific processors. The default value of `enabled` can produce significantly better performance when a single instance of AMPS is running on a given system. However, if multiple instances of AMPS are running on the same system, setting this value to `disabled` for all of the instances on the system can reduce contention among the instances and produce better overall performance. Likewise, if the system that hosts AMPS runs other CPU-intensive processes, setting this option to `disabled` can improve overall performance. When AMPS runs in a virtual machine, 60East recommends setting this option to `disabled`. When AMPS runs in a container on a NUMA host use the guidance in [Host Guidance](../../amps-user-guide/operation/host-guidance#containers). Leave this option set to `enabled` in a container on when every condition in that section is true, including workload isolation, visible host NUMA topology, compatible runtime CPU and memory constraints, and performance testing of the complete container and host environment. This option can also be set by setting the `AMPS_NUMA` environment variable. Default: `enabled` Example: ```xml showLineNumbers .... enabled .... ```
    `Replication/MinSyncDestinations` Setting this to a value will limit the number of replication destinations that AMPS will allow an action to downgrade to use `async` acknowledgment. When set, AMPS actions will not downgrade a destination if doing so would cause the number of destinations using `sync` acknowledgment to be less than the number set in this value. Notice that this value will not cause AMPS to upgrade a destination if a `sync` destination disconnects. :::important The `MinSyncDestinations` parameter affects only whether AMPS will downgrade a destination. If the instance falls below the `MinSyncDestinations` value due to a `sync` destination going offline, the instance *will not* upgrade a destination. ::: When no value is set, AMPS actions will downgrade any destination that meets the criteria in the downgrade action. Default: Unset Example: ```xml showLineNumbers .... AMPS_A .... ```
    `Queue/QueueDeliveryFlushInterval` This sets the maximum interval for a message queue delivery thread to wait to send messages to a client. The default value typically gives a good balance between latency and throughput. Decreasing the `QueueDeliveryFlushInterval` value may decrease the latency of message delivery from a queue at the expense of lower overall throughput. Default: `250us` Minimum: `1us` Example: ```xml showLineNumbers .... 250us .... ```
    `Statistics/Indexing/Enabled` When enabled, this setting will create an index on `static_id` for all `DYNAMIC` tables when the statistics database is persisted to a file. This setting can speed up statistics truncation. :::warning It is important to note that this could increase AMPS server start up times when first enabled. Therefore, it is recommended to enable this option on a new statistics database. ::: If the `Statistics` `Indexing` `Enabled` option is removed, the created indexes are not removed. Note: There can be a cost to other functions, such as insertion of records when statistics are collected. Default: Unset. By default, AMPS does not create an index on `static_id` for all `DYNAMIC` tables. Example: ```xml showLineNumbers .... enabled .... ```
    ## Externals The AMPS server depends on external libraries for some functionality. The `Externals` configuration item allows you to control the exact shared object loaded for some of these external libraries, particularly those related to security. :::info Although AMPS ships with SSL and Crypto libraries that are current at the point of server release, it's recommended that you load your own OpenSSL and Crypto libraries to more easily respond to any issues at your own patching cadence. ::: Described below are the options for configuring external libraries. Expand each option for more details.
    `SSL/Library` The path and shared object name of the SSL library to use for this instance. AMPS requires an SSL library that is compatible with OpenSSL 1.1. By default, AMPS specifies the object name, and uses the standard shared object loading mechanism to resolve the object name. With this configuration option, you can direct AMPS to load a specific shared object. Default: `libopenssl.so`
    `Crypto/Library` The path and shared object name of the cryptography library ("crypto library") to use for this instance. By default, AMPS specifies the object name, and uses the standard shared object loading mechanism to resolve the object name. With this configuration option, you can direct AMPS to load a specific shared object. Default: `libcrypto.so`
    `Curl/Library` The path and shared object name of the `libcurl` shared object ("curl library"). By default, AMPS specifies the object name and loads the version of libcurl included with the AMPS distribution as necessary. With this configuration option, you can direct AMPS to load a specific shared object. Default: `libcurl.so`
    ```xml showLineNumbers /opt/audited/libopenssl.so /opt/audited/libcrypto.so /opt/resolver/lib/libcurl.so ``` ## Specialized Options This section describes options that are configured at the instance level for specialized purposes. Expand each option for more details.
    `SOWStatisticsInterval` AMPS can publish SOW statistics for each SOW topic which has been configured. The `SOWStatisticsInterval` is specified as an interval between updates to the `/AMPS/SOWStats` topic. Set this option if an application will subscribe to the `/AMPS/SOWStats` topic or if that topic is included in the SOW.
    `RegexTopicSupport` Sets whether this instance supports regular expression topic matching. When this option is `true`, clients can register subscriptions using regular expressions and receive messages for all matching topics. When this option is `false`, regular expression characters are interpreted as literal characters. Likewise, when this option is `true`, `Topic` specifications in replication configuration, transaction log configuration, and so on can use regular expressions. 60East recommends leaving this option set to the default unless there is a specific reason to change it, and unless the configuration and applications have been reviewed to ensure no regular expression topics are used. Default: `true`
    `ConfigValidation` Sets whether AMPS validates the configuration file when starting. Setting this to `disabled` will turn off AMPS configuration validation. The default is `enabled`, ensuring that the current AMPS configuration meets valid parameter ranges and data types. When this option is set to `disabled`, AMPS may start with an inconsistent or invalid configuration, which may have unpredictable effects, including data loss or AMPS unexpectedly exiting. This option is included in cases where it may be necessary to start with a file that is known to be invalid, such as when testing applications that generate configuration files.
    --- # Production Configuration To create a production configuration of AMPS, you configure the instance to meet the needs of the application (or applications) that will use the instance. An overview of the most commonly used features is available in the [Introduction to AMPS](/docs/intro-guide/intro) guide. This guide, the _AMPS User Guide_ provides detailed descriptions of those features, including the required and optional configurations for each. Typically, all instances of AMPS will configure: * The instance [`Name`](/docs/amps-user-guide/configuring-amps/instance-configuration) (this is required). * The [Admin interface](/docs/amps-user-guide/monitoring/monitor-configuration) for the instance, to make monitoring available. This typically includes setting a path to persist the instance statistics database. * [Logging](/docs/amps-user-guide/logging/logging-configuration) for the instance (at a minimum of `info` level for production instances, typically at `trace` level for development, testing, or UAT instances). * One or more [Transports](/docs/amps-user-guide/transports/configuring-transports) to allow incoming connections to the AMPS server. * [Administrative actions](/docs/amps-user-guide/actions) to create a [scheduled maintenance plan](/docs/amps-user-guide/actions/on-elements/on-schedule) for the [statistics database](/docs/amps-user-guide/actions/do-elements/do-manage-stats) and the logs. The `ampServer` binary will produce a minimal sample configuration to `stdout` if it is run with the `--sample-config` flag that shows a minimum configuration. Options that require site-specific information (for example, the path to the statistics database or log files) are commented out in the sample. Instances of AMPS may then add configuration to take advantage of advanced messaging features (such as the [State of the World (SOW)](/docs/amps-user-guide/sow), [Aggregation and Analytics](/docs/amps-user-guide/views), the ability to [Record and Replay Messages](/docs/amps-user-guide/txlog), and so on), to add resiliency by [Replicating Messages Between Instances](/docs/amps-user-guide/replication) (typically required for [Highly Available AMPS Installations](/docs/amps-user-guide/ha)), and so on. --- # Working with Configuration Files AMPS provides a command line option to help an administrator quickly set up an AMPS server. In addition to the quick setup discussed in [Installing and Starting AMPS](/docs/amps-user-guide/installing-and-starting), AMPS also provides the following command line options to create a basic XML configuration file. Running the following command will create a configuration file named `config.xml`. The generated file is a bare-bones configuration that allows AMPS to start, process JSON messages, and provide monitoring through the admin interface. ``` ampServer --sample-config > config.xml ``` The AMPS server also provides the ability to perform basic validation of the config file, using the `--verify-config` flag. ``` ampServer --verify-config config.xml ``` The validation process checks for errors in the configuration that would prevent AMPS from starting, and reports warnings and informational messages about the configuration file. However, the validation process does not ensure that the configuration file provided is suitable for any particular purpose. When a configuration file uses the `Include` directive or uses environment variable substitution, it can be useful to produce a fully expanded file. AMPS provides a `--dump-config` flag for this purpose. The command produces the fully expanded file to standard output. ``` ampServer --dump-config config.xml > expanded.xml ``` --- # Configuring Conflated Topics in a SOW This section lists the parameters for defining a `ConflatedTopic` within the SOW section of an AMPS configuration file (also called _topic replicas_ in previous releases of AMPS). For compatibility with previous AMPS versions, AMPS allows you to use `ReplicaDefinition` as a synonym for `ConflatedTopic`. Described below are the configuration items for defining a `ConflatedTopic`. Expand each item for more details.
    `Name` (required) String used to define the name of the conflated topic. While AMPS doesn't enforce naming conventions, it can be convenient to name the conflated topic based on the underlying topic name. For example, if the underlying topic is `orders`, it can be convenient to name the conflated topic `orders-C`. If no `Name` is provided, AMPS accepts `Topic` as a synonym for `Name` to provide compatibility with versions of AMPS previous to 5.0.
    `UnderlyingTopic` (required) String used to define the SOW topic which provides updates to the conflated topic. This must exactly match the name of a SOW topic. When the underlying topic is a regular expression topic, this must match the `Name` of the topic rather than the `Pattern` of the topic. In this case, the conflated topic will conflate all of the topics in the underlying regular expression topic into individual topics, with the names of the individual topics set using the `TopicFormat` element.
    `MessageType` (required) The message format of the underlying topic. This `MessageType` must be the `MessageType` of the provided `UnderlyingTopic`.
    `Interval` The frequency at which AMPS updates the data in the conflated topic. Default: `5s` Minimum: `100ms`
    `Filter` Content filter that is applied to the underlying topic. Only messages that match the content filter are stored in the conflated topic.
    `HashIndex` AMPS provides the ability to do fast lookup for SOW records based on specific fields. When one or more `HashIndex` elements are provided, AMPS creates a hash index for the fields specified in the element. These indexes are created on startup, and are kept up to date as records are added, removed, and updated. The `HashIndex` element contains a `Key` element for each field in the hash index. AMPS uses a hash index when a query uses exact matching for all of the fields in the index. AMPS does not use hash indexes for range queries or regular expressions. AMPS automatically creates a hash index for the set of fields specified in the set of `Key` fields for the `UnderlyingTopic`, if that topic provides `Key` fields.
    `Enrichment` When present, specifies the message enrichment to be performed on the messages from the underlyingTopic. The `Enrichment` element must contain one or more `Field` elements that specify the enrichment to perform. For more information on constructing enrichment fields, see [Constructing Enrichment Fields](../builtin\_functions/constructing-fields.md#constructing-enrichment-fields). *`Enrichment` on ConflatedTopic does not support `OF PREVIOUS` in a `Field`*
    If the underlying topic of a `ConflatedTopic` is a set of logical topics defined with the `Pattern` element, an additional configuration parameter is required to specify the topic name that AMPS will use for each conflated topic. Expand the item for more details.
    `TopicFormat` (required if `UnderlyingTopic` uses a `Pattern`) String used to format the name of the conflated topics when the underlying topic uses a `Pattern` to create a group of logical topics within a single physical topic. When the underlying topic is a regular expression topic, AMPS will create a conflated topic for each topic within the regular expression topic. This element specifies how AMPS will construct the name of the conflated topics produced. The `TopicFormat` must contain the string `%n`, which is replaced with the name of the topic in the underlying regular expression topic. For example, if the `TopicFormat` is `%n-C`, when the underlying regular expression topic contains a topic named `/my/orders`, AMPS will produce a conflated topic with the name `/my/orders-C`. There is no default for this element. This element cannot contain characters that are significant for regular expressions (such as `.`, `^`, and so on).
    Below is an example that shows different approaches to defining a `ConflatedTopic`: ```xml showLineNumbers FastPublishTopic-C nvfix FastPublishTopic 5s /region = 'A' LongIntervalTopic-C json FastPublishTopic 120s /order/status ConflatedEnrichmentTopic json market 1s UPPER(/ticker) as /ticker ConflateUnderlyingRegex bflat TheRegexTopic 20s ``` --- # Conflated Topics AMPS provides the ability for the server to _conflate_ updates to a SOW topic by defining a _conflated topic_ for that SOW topic. A conflated topic will retain messages published to the underlying topic for a certain period of time, and provide the latest update for each distinct message in the underlying topic at the end of that period of time. In effect, AMPS guarantees that a subscriber will receive _no more than_ one update for a given message per conflation interval. A conflated topic provides a way to reduce the bandwidth and processing for subscribers in cases where the subscriber needs periodic updates with the current state of the message rather than a rapid set of updates with each individual change to the message. For example, an application that presents a user interface to display rapidly changing data often uses conflation, since the value of a record may change more rapidly than the user interface is able to be refreshed. For this application, creating a conflated topic and then having the application subscribe to that topic can reduce network traffic, reduce processing load, and provide a more responsive user interface than if the application were to subscribe to the underlying topic and try to process every update. AMPS provides the ability to conflate messages for an individual subscription, as described in the [Conflated Subscriptions](pub-sub/conflation) section. When a single subscriber requires conflation, requesting conflation for that subscription is a reasonable approach to take. In cases where all instances of an application can benefit from conflation, _conflated topics_ are a more efficient and scalable approach. A conflated topic is a copy of one SOW topic into another with the ability to control the update interval. In this case, AMPS maintains conflation for the entire topic. There is no need for subscribers to independently request conflation, and AMPS does not need to spend resources processing conflation for each subscriber individually. The underlying topic for a conflated topic can be a `Topic`, a `View`, or another `ConflatedTopic`. To better see the value in a conflated topic, imagine a SOW topic called `ORDER_STATE` exists in an AMPS instance. `ORDER_STATE` messages are published frequently to the topic. Meanwhile, there are several subscribing clients that are watching updates to this topic and displaying the latest state in a GUI front-end. If this GUI front-end only needs updates in five second intervals from the `ORDER_STATE` topic, then more frequent updates would be wasteful of network and client-side processing resources. To reduce network congestion, a conflated topic for the `ORDER_STATE` topic can be created which will contain a copy of `ORDER_STATE` updated in five second intervals. Only the changed records from `ORDER_STATE` will be copied to the conflated topic and then sent to the subscribing clients. Those records with multiple updates within the time interval will have their latest updated values copied to the conflated topic, and only those conflated values are sent to the clients. This results in substantial savings in bandwidth for records with high update rates. This can also result in substantial savings in processing overhead for a client. AMPS treats the conflated topic as a conflated version of the underlying topic. Applications cannot publish directly to the conflated topic. Likewise, AMPS does not recalculate the SOW key for messages delivered from the conflated topic: these messages have the same SOW key as the corresponding message in the underlying topic. AMPS indexes conflated topics in the same way that it indexes the underlying topic in the SOW. When a query uses a given field, AMPS will automatically create a memo index for that field. A configuration can also declare one or more `HashIndex` indexes for a conflated topic. AMPS uses the following conflation strategy: * When a message arrives for a given key, if no message is already pending for that key, begin the conflation interval. * If an update arrives during the conflation interval for a given key, _replace_ the pending message with the update for a subscription, or merge the update into the pending message for a delta subscription. All of the metadata on the message is fully replaced. * If the update is an `oof` notification, and the key has not previously been delivered to the subscription, note that the message should not be delivered. * At the end of the conflation interval, deliver the current version of the message unless the current message is an `oof` that should not be delivered. Remove the message from the set of messages being conflated. This means that, when conflation is used, the first update for a given key will arrive after the conflation interval. Further, the application must expect that any given update may be delayed by up to the conflation interval. The following section provides the configuration details for conflated topics. --- # Configuring the Service When running as a service, the following considerations apply to the configuration file: ## AMPS Logging 60East recommends logging the most important AMPS messages to syslog when running as a service. For example, the following configuration file snippet logs messages of warning level and above to the system log: ```xml showLineNumbers syslog warning amps LOG_CONS,LOG_NDELAY,LOG_PID LOG_USER ``` 60East does not recommend logging a level lower than warning to syslog, since an active AMPS instance can produce a large volume of messages. ## File Paths When running as a service, file paths in the configuration file also require attention. In particular: - For simplicity, use absolute paths for all file paths in the configuration file. - Consider startup order, and ensure that any devices that AMPS uses are mounted before AMPS starts. - As with any other AMPS installation, it's also important to estimate the amount of storage space AMPS requires and ensure that the device where AMPS stores files has the needed capacity. ## Configuration File Location The AMPS service scripts require the configuration file to be located at: `/opt/etc/amps/config.xml`. --- # Installing the Service AMPS includes a shell script that installs the service. The shell script is included in the `bin` directory of your AMPS installation. Run the script with root permission, as follows: ```bash $ sudo ./install-amps-daemon.sh ``` This script does the following installation work: - Installs the AMPS distribution into `/opt/amps`. - Creates the `/opt/etc/amps` directory if it does not already exist. By default, the daemon uses an AMPS configuration file at `/opt/etc/amps/config.xml`. - Installs the service management scripts. Depending on the init system the script detects on your system, this will either be a System V style script located at `/etc/init.d/amps` or a SystemD service definition file named `amps.service` installed under `/usr/lib/systemd/`. - Updates the service management infrastructure to register AMPS as a service and configure the service to start on startup. The exact steps that the script takes to do this depend on the init system detected. In addition, you must copy the AMPS configuration file for the instance to: `/opt/etc/amps/config.xml`. You can only run one instance of AMPS as a service on a system at a given time using this script. AMPS does not enforce any restriction on how many instances can be run on the system at the same time through other means, but this script is designed to manage a single instance running as a service. --- # Managing the Service The scripts that AMPS installs provide management functions for the AMPS service. The scripts are used in the same way scripts for other Linux services are used. ## Starting the AMPS Service To start the AMPS service, use the following command if your system uses System V init scripts: ```bash sudo /etc/init.d/amps start ``` Many systems that use System V init scripts also provide convenience commands (such as `service`) to locate and run commands for working with daemons. Check your distribution's documentation for details. If your system uses SystemD, you can use a command like: ```bash sudo systemctl start amps ``` ## Stopping the AMPS Service To stop the AMPS service, use the following command if your system uses System V init scripts: ```bash sudo /etc/init.d/amps stop ``` Many distributions that use System V init scripts also provide convenience commands (such as the `service` program) for working with daemons. Check your distribution's documentation for details. If your system uses SystemD, you can use a command like: ```bash sudo systemctl stop amps ``` ## Restarting the AMPS Service To restart the AMPS service, use the following command if your system uses System V init scripts: ```bash sudo /etc/init.d/amps restart ``` Many distributions that use System V init scripts also provide convenience commands (such as the `service` program) for working with daemons. Check your distribution's documentation for details. If your distribution uses SystemD, you can use a command like: ```bash sudo systemctl restart amps ``` ## View Status for the AMPS Service To see the status of the AMPS service, use the following command if your system uses System V init scripts: ```bash sudo /etc/init.d/amps status ``` Many distributions that use System V init scripts also provide convenience commands (such as the `service` program) for working with daemons. Check your distribution's documentation for details. If your distribution uses SystemD, you can use a command like: ```bash sudo systemctl status amps ``` --- # Uninstalling the Service AMPS includes a script that uninstalls AMPS as a service. The script reverses the changes that the install script makes to your system. Run the script with root permission, as follows: ```bash $ sudo ./uninstall-amps-daemon.sh ``` The uninstall script does not remove the configuration file or any files or data that AMPS creates at runtime. --- # Running AMPS as a Linux Service AMPS is designed to be able to easily integrate into your existing infrastructure. AMPS includes all of the dependencies it needs to run and is configured easily with a single configuration file. Some deployments integrate AMPS into a third-party service management infrastructure. For those deployments, the needs of that infrastructure determine how to install AMPS. More typically, AMPS runs as a Linux service. This chapter describes how to install AMPS as a service. --- # Delta Publish Support To accept delta publishes, the message type and the topic must both support delta publish. When this is not the case, AMPS accepts the publish, but may not produce the expected results. All of the basic message types provided with AMPS support delta publish with the exception of the `binary` message type and protobuf message types that use protobuf version 3 below version 3.15. AMPS supports delta publish for protobuf 3 messages that use Protobuf 3.15 or greater and the `optional` field annotation as specified in the official Protobuf documentation and the AMPS user guide section on protobuf. greater Composite message types support delta publish if they use the `composite-local` definition, as described in the section on composite message types. Types that do not support delta publish, produce the full, literal message provided with a delta publish command (rather than merging the publish into the previous state of the message). When a topic uses the `composite-local` message type, parts of the composite that are provided as empty (that is, zero-length) are considered to be unchanged, and the merged message contains the existing contents of that part. This provides a convenient way to update only one part of a composite message, without having to republish data that has not changed. For example, a `composite-local` type contains a JSON part and a binary part can modify the JSON part without having to republish the full binary part. AMPS queues support delta publish to an underlying topic, if that underlying topic maintains a SOW. The merged message is provided to the AMPS queue. All other AMPS topic types that are based on a SOW and accept publish commands support delta publish. AMPS topics that do not use a SOW do not support delta publish, so publishing a delta message to those topics produces the full, literal message from the publish command rather than a merged message. Without a SOW configured for the topic, AMPS does not track the current value of a message, and therefore does not have a way to merge the publish into an existing message. ## Transaction Log Replay and Delta Publish The AMPS transaction log stores the fully merged message. This means that, when the topic is also recorded in the transaction, a replay from the transaction log can provide all of the information in the message, not simply the delta. This can be particularly useful when one or more fields in the message is intended to be unchanged over a period of time: there is no need for a transaction log replay to reconstruct the message from an original publish that could be hours, days, or months in the past. This also means that AMPS replication, since it is based on the transaction log, replicates the *fully merged message*. In general, this means that publishers that will operate on the same message should be grouped on the same AMPS instance, or should take care to sequence their updates so as not to overwrite other delta publishes. --- # Understanding Delta Publish When AMPS receives a delta publish request, AMPS first performs any preprocessing or enrichment specified for the incoming message. AMPS then fully parses both the incoming message and the destination message, and then merges the contents of the incoming update into the existing message. The mechanism that merges the messages is message-type independent, and does not rely on the syntax or semantics of any particular message type. The message is merged before the message is persisted to the SOW, recorded in the transaction log, replicated, or delivered to subscribers. Since delta publish uses a very simple, compact syntax -- a partial message -- for updates, AMPS makes certain assumptions about the intent of an update: 1. AMPS replaces the contents of the existing message based on the parsed identifiers in the update. An update to an anonymous element or the root level of a subdocument is a _full replacement_ of that value. For example, consider the following JSON document: ```json {"id":42, "contents":{"packages":[{"box":"chocolates"}, {"bowl":"noodles"}]}} ``` An update to the `packages` field of the document _replaces_ the subdocument, so providing the following `delta_publish`: ```json {"id":42, "contents":{"packages":[{"basket":"eggs"}]}} ``` Replaces the previous value of `packages`, and produces the message: ```json {"id":42, "contents":{"packages":[{"basket":"eggs"}]}} ``` 2. A missing field in a delta publish means that the content of that field is unchanged. Since AMPS treats the absence of a field in a delta update to mean that the previous value is unchanged, a `delta_publish` cannot be used to remove fields from a document. For example, consider the following JSON document: ```json {"id":42, "flowers":"roses"} ``` A `delta_publish` that attempts to remove the `flowers` field by simply removing it will be treated as though there is no update to that field. In other words, a `delta_publish` of: ```json {"id":42} ``` will produce no change to the `flowers` field, and result in a document of: ```json {"id":42, "flowers":"roses"} ``` To remove a field from a message, republish the full message. 3. AMPS does not evaluate whether a field has a different value than the previous message or has a republish of the previous value when processing a `delta_publish` -- any field present in the delta publish will replace the corresponding field in the existing message. For example, consider an application that advertises whether a given system is accepting requests with messages along the lines of: ```json {"systemId":"A34-Astro", "status":"available", "commandAccepted":["print","scan","fax"]} ``` Now, the system sends a delta publish update that does not change the status: ```json {"systemId":"A34-Astro", "status":"available"} ``` Even though no value in the message has changed, the publish is still processed, and the publish will still be delivered to subscribers. The fact that it replaces an existing value with the same value does not matter for AMPS. This is still considered an update. (An individual subscriber can choose not to receive updates that do not change the value of a field by using the `no_empties` option with a delta subscription as described in the [Receiving Only Updated Fields](../delta-subscribe) section.) 4. AMPS uses the fully parsed values of the existing message and the updated message to perform the merge. This means that nested elements are processed as described in [Compound Types in AMPS](../amps-expressions/amps-data-types.md#compound-types-in-amps). Use care when combining delta publish with nested elements, particularly nested elements within arrays or nested elements that contain arrays. --- # Using Delta Publish Since delta messages must be processed and merged into the existing SOW record, AMPS provides a distinct command for delta publish. | Command | Result | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `delta_publish` |

    Publish a delta message.

    If no record exists in the SOW, add the message to the SOW.

    If a record exists in the SOW, merge the data from this record into the existing record.

    | --- # Incremental Message Updates AMPS allows a publisher to update and add fields within a message that is stored in a State-of-the-World `Topic` using the `delta_publish` command. This can be important in high-performance messaging, it can be important to conserve bandwidth by sending the smallest possible update over the network. An incremental update may improve performance in environments where bandwidth is at a premium. Since an incremental update requires that AMPS parse, merge, and re-serialize messages, an incremental update can consume somewhat more CPU on the AMPS server than a simple publish, particularly for large messages with a complex structure (such as deeply-nested documents). To be able to incrementally update a message, the message type for the `Topic` must support delta messages. All of the included AMPS message types, except for `binary` and `struct`, support delta messages, with the limitations described in each section below. For custom message types, contact the message type implementer to determine whether delta support is provided. AMPS also supports the ability of a subscriber to receive only the changed parts of a message, described in the section on [Receiving Only Updated Fields](delta-subscribe). While these features are often used together, the features are independent. For example, a subscriber can request a regular subscription even if a publisher is publishing deltas. Likewise, a subscriber can request a delta subscription even if a publisher is publishing full messages. --- # Conflated Subscriptions and Delta Subscribe AMPS provides subscription conflation on delta subscriptions. When conflation is enabled, each delta message during the conflation interval is merged into the conflated message. The message that is delivered is the merge of all of the deltas that arrived during the conflation interval. Since AMPS combines successive delta messages into a single update, a delta subscription that uses conflation may receive values that are identical to the previous values. For example, consider the following record in a SOW that uses `/id` as the key: ```javascript { "id": 99, "status":"open", "notes":"none", "xref":82 } ``` Assume that the following updates to the record are published during the conflation interval: ```javascript {"id": 99, "status":"questioned", "notes":"none", "xref":82} {"id": 99, "status":"questioned", "notes":"jcarlo hold", "xref":82} {"id": 99, "status":"cleared", "notes":"none", "xref":82} {"id": 99, "status":"open", "notes":"none", "xref":82} ``` At the end of the conflation interval, the subscription will receive the delta message: ```javascript { "id": 99, "status":"open", "notes":"none" } ``` The `/id` field is included because that field is the key of the SOW, and all of the delta messages produced during the conflation interval contained that key. The `/status` and `/notes` fields are included because there were changes to these values during the conflation interval. The delta messages produced during the conflation interval contained changed values, so the merged update contains those fields and the state of the values at the end of the conflation interval. The `/xref` field is not included, because none of the delta messages produced during the conflation interval contained that field. --- # Identifying Changed Records When an application that uses delta subscriptions receives a message, that message can either be a new record or an update to an existing record. AMPS offers two strategies for an application to tell whether the record is a new record or an existing record and identify which record has changed if the message is an update to an existing record. The two basic approaches are as follows: 1. By default, each message delivered through a delta subscription contains a SowKey header field. This field is the identifier that AMPS assigns to track a distinct record in the SOW. If the application has previously received a SowKey with that value, then the new message is an update to the record with that SowKey value. If the application has not previously received a SowKey with that value, then the new message contains a new record. 2. Delta messages can also contain the key fields from the SOW in the body of the message. This is controlled by the `send_keys` option on the subscription, which is always enabled as of AMPS 4.0. With this approach, the application parses the body of the message to find the key. If the application has previously received the key, then the message is an update to that existing record. Otherwise, the message contains a new record. In either case, AMPS delivers the information the application needs to determine if the record is new or changed. The application chooses how to interpret that information, and what actions to take based on the changes to the record. AMPS also supports out-of-focus notification for delta subscriptions, as described in [Out-of-Focus (OOF)](../oof) messages. If your application needs to know when a record is deleted, expires, or no longer matches a subscription, you can use out-of-focus messages to be notified. ## Receiving Only Changes that Update Values In some cases, an application needs to know if the record has been updated, even if the update preserves the value of the fields the application is interested in. This is default behavior of a delta subscription. In other cases, an application may only want to receive an update if the value of fields have changed. AMPS accepts a subscription option for `delta_subscribe` and `sow_and_delta_subscribe` to request that the subscription receive only updates where the value of a field has changed. This option is `no_empties`. Include this option in the `options` header of the command to request that updates that do not change the value of fields (that is, in cases where the delivered message would only have the `Key` fields for the topic) are not delivered to the subscription. ## Conflated Subscriptions and Delta Subscribe AMPS provides subscription conflation on delta subscriptions. When conflation is enabled, each delta message during the conflation interval is merged into the conflated message. The message that is delivered is the merge of all of the deltas that arrived during the conflation interval. Since AMPS combines successive delta messages into a single update, a delta subscription that uses conflation may receive values that are identical to the previous values. For example, consider the following record in a SOW that uses `/id` as the key: ```javascript { "id": 99, "status":"open", "notes":"none", "xref":82 } ``` Assume that the following updates to the record are published during the conflation interval: ```javascript {"id": 99, "status":"questioned", "notes":"none", "xref":82} {"id": 99, "status":"questioned", "notes":"jcarlo hold", "xref":82} {"id": 99, "status":"cleared", "notes":"none", "xref":82} {"id": 99, "status":"open", "notes":"none", "xref":82} ``` At the end of the conflation interval, the subscription will receive the delta message: ```javascript { "id": 99, "status":"open", "notes":"none" } ``` The `/id` field is included because that field is the key of the SOW, and all of the delta messages produced during the conflation interval contained that key. The `/status` and `/notes` fields are included because there were changes to these values during the conflation interval. The delta messages produced during the conflation interval contained changed values, so the merged update contains those fields and the state of the values at the end of the conflation interval. The `/xref` field is not included, because none of the delta messages produced during the conflation interval contained that field. --- # Options for Delta Subscribe The delta subscribe command accepts several options that are unique to delta subscriptions. These options control the precise behavior of delta messages: | Option | Result | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `no_empties` |

    Do not send messages if no data fields have been updated.

    By default, AMPS will publish a delta for every publish to the record, even if the data has not changed. By specifying this option, AMPS will only send messages when there is changed data.

    | | `no_sowkey` |

    Do not include the AMPS generated SowKey with messages.

    By default, AMPS includes this key to help you identify unique records within the SOW.

    | | `send_keys` |

    Include the SOW key fields in the message.

    Since the SOW key fields indicate which message to update, without this option, updates to delta messages will never contain the SOW key fields. For views, the SOW key fields are the fields specified in the `Grouping` element.

    AMPS accepts this option for backward compatibility. This behavior is the default if `no_empties` is not provided.

    | | `oof` |

    AMPS will deliver out of focus messages on this subscription.

    When focus tracking is enabled, AMPS will also deliver the full message to a subscription when a previously out-of-focus message comes into focus.

    | Delta subscriptions also support the options provided for regular subscriptions, including the timestamp option and the conflation options described in the section on [Conflated Subscriptions](../pub-sub/conflation). --- # Select Lists and Delta Subscribe When a `delta_subscribe` or `sow_and_delta_subscribe` command provides a select list and the `no_empties` option, only changes to fields included in the select list will prompt a publish to the client. Changes to fields that the subscriber does not receive do not produce a publish. --- # Using Delta Subscribe Since a client must process delta subscriptions using substantially different logic than regular subscriptions, delta subscription is implemented as a separate set of AMPS commands rather than simply as an option on subscribe commands. AMPS supports two different ways to request a delta subscription: |Command |Result | |------------------------------------|-------------------------------------------------------------------------| |`delta_subscribe` |Register a delta subscription, starting with newly received messages. | |`sow_and_delta_subscribe`|Replay the state of the SOW and atomically register a delta subscription.| Applications most commonly use `sow_and_delta_subscribe` to receive the current state of messages in the SOW before they begin receiving deltas. --- # Receiving Only Updated Fields AMPS allows a subscriber to a `Topic`, `View`, or `ConflatedTopic` in the State of the World to request that the server provide only incremental updates to messages using the `sow_and_delta_subscribe` command. This can reduce the size of messages received, and reduce the need to parse or process parts of a message that have not changed when other fields are updated. When a delta subscription is active, AMPS compares the new state of the message to the old state of the message, creates a message for the difference, and sends the difference message to subscribers. For example, consider a SOW that contains the following messages, with the `order` field as the key of the SOW topic: Now, consider an update that changes the status of order number 3: ```javascript { "order":3, "customer":"Patrick", "status":"pending", "qty":1000, "ticker":"MSFT" } ``` For a regular subscription, subscribers receive the entire message. With a delta subscription, subscribers receive just the key of the SOW topic and any changed fields: This can significantly reduce the amount of network traffic, and can also simplify processing for subscribers, since the only information sent is the information needed by the subscriber to take action on the message. When the `oof` option is specified on a delta subscription, AMPS will deliver the full message when a previously out-of-focus message comes into focus, even if only some of the fields in the message have changed. AMPS also supports the ability of a publisher to incrementally update the message, as described in the section on [Incremental Message Updates](delta-publish). While these two capabilities work together well, they are completely independent. It is not necessary for a publisher to use `delta_publish` for a subscriber to receive incremental updates. ## Delta Subscribe Support To produce delta messages, the message type and the topic must both support delta subscribe. When this is not the case, AMPS accepts the subscription, but provides full messages rather than delta messages. All of the basic message types provided with AMPS support delta subscribe with the exception of the `binary` message type, `struct` message types, and protobuf message types that use protobuf version 3 less than version 3.15. AMPS supports delta subscribe for protobuf 3 messages that use protobuf 3.15 or greater and the `optional` field annotation as a specified in the official protobuf documeantion and the AMPS user guide section on protobuf. Composite message types support delta subscribe if they use the `composite-local` definition, as described in the section on [Composite Message Types](message-types/composite-messages). AMPS queues do not support delta subscribe. AMPS accepts a delta subscription for a queue, but produces full messages from the queue. Bookmark subscriptions do not support delta subscribe. AMPS does not accept a delta bookmark subscription. All other AMPS topic types that are based on a SOW support delta subscribe. AMPS topics that do not use a SOW do not support delta subscribe, and instead produce full messages. ## Multiple Subscriptions and Delta Subscribe When a single connection to AMPS has multiple subscriptions, AMPS sends the message to that client once, with information on the set of subscriptions that match. AMPS sends a message that will include the requested data for all of the matching subscriptions. For example, if a message matches one subscription that requests full messages and another subscription from the same connection that requests deltas, both subscriptions will receive a full message. If your application depends on receiving deltas, take care that the application does not issue non-delta subscriptions for the same set of messages on the same connection. --- # AMPS Distribution Layout This appendix lists the layout of the AMPS distribution, with special focus on the binaries present in the layout. Use this appendix to plan your AMPS deployment. 60East recommends that all AMPS deployments contain the full contents of the `/bin` and `/lib` directories. For development installations that are extending the AMPS server, your installation should contain the `/api` and `/sdk` directories (as well as the *AMPS Server SDK*, available through 60East support). The AMPS distribution contains the following items at the top level: |Item |Description | |------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`/bin` |AMPS binaries: the AMPS server, daemon deployment scripts, AMPS utilities, and `spark`.| |`/docs`|AMPS base documentation. Current versions of the documentation and additional guides are available from the 60East website.| |HISTORY |Information on the AMPS revision history. In current distributions, this provides a link to the entry for this release of AMPS within the full revision history on the 60East web site.| |`/lib` |Libraries used by the AMPS binary.| |LICENSE |The AMPS license. | |README |The README file for AMPS.| |`/sdk` |Headers used for modules that extend AMPS.| ## /bin directory |Item |Description | |--------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------| |`amps_bio_perf_test` |Diagnostic tool for testing the performance of I/O systems. | |`amps_clients_ack_dump` |Utility for showing the contents of the AMPS clients.ack file, containing persistent per-client information.| |`ampserr` |Utility for looking up details on AMPS log file items. | |`ampServer` |The AMPS server binary. | |`ampServer-compat` |The downward compatible version of the AMPS server binary. This version avoids using some of the hardware capabilities present in newer CPU architectures.| |`amps_file` |A utility for identifying the type of AMPS files and the file format that the file uses.| |`amps-init-script` |Part of the AMPS service installation. This script is installed into the init.d directory when the AMPS service is installed.| |`amps_journal_dump` |Utility for extracting the contents of AMPS transaction log journal files.| |`amps_mt_perf_test` |Diagnostic tool for performance testing of the AMPS engine parsing infrastructure.| |`amps_sow_dump` |Utility for extracting the contents of AMPS SOW files. | |`amps-sqlite3` |Convenience wrapper for querying an AMPS statistics database. | |`amps_upgrade` |Utility for upgrading data files from previous versions of AMPS to the current version.| |`install-amps-daemon.sh` |Installation script for installing AMPS as a Linux service. | |`/lib` |Directory containing the libraries used by the `spark` utility.| |`spark` |Utility that provides a command-line interface to AMPS. | |`uninstall-amps-daemon.sh`|Installation script for removing the AMPS Linux service from the system.| --- # State of the World Message Enrichment Topics recorded to the State of the World (SOW) can provide inline message enrichment for messages published to the topic. This capability is especially useful for applications that do consistent, simple transformations on incoming data. For example, you can use this capability to automatically add a calculated price to an incoming order, to map abbreviated data such as status codes to easier-to-understand values, or even to compute the value of a field used for a SOW key. AMPS provides two distinct stages of message enrichment: _preprocessing_ and _enrichment_. The _preprocessing_ stage occurs before AMPS calculates the SOW key for the message. Fields that are added or updated in the preprocessing stage can be used as the SOW key for the message. Given that this stage occurs before the SOW key is generated, this stage does not have access to the previous state of the message in the SOW. The enrichment stage occurs after AMPS calculates the SOW key. _Enrichment_ performed at this stage has access to the previous state of the SOW. If entitlement for the instance uses content filters for publish entitlements, these filters are applied to the incoming message _before_ either enrichment stage runs. For more details on the steps involved in enrichment, see the sequence of operations in [SOW Update and Enrichment Processing](enrichment#sow-update-and-enrichment-processing). Message enrichment only affects the message data, not the metadata on the message. In other words, while enrichment can change any field in the data, you cannot change metadata properties such as the topic the message was published to, the acknowledgments requested on the message, or the authenticated username for the publish command. Message enrichment rewrites the message before the messages are stored in AMPS or delivered to subscribers. AMPS also provides the ability to aggregate or analyze messages while preserving the original state of the message, as described in the chapter on [Aggregation and Analysis](views). If a subscriber only needs a subset of data in a message, AMPS provides the ability for that subscriber to provide a [select list](pub-sub/select-list) to retrieve only the needed data. When an instance receives a message over replication, any enrichment on the source instance will already have run. Messages received over replication are not enriched again on the destination instance. ## Preprocessing Messages The preprocessing stage of AMPS enrichment allows you to alter a message before the SOW key is calculated. This gives you the ability to easily add or transform fields that are used in the SOW key. Use this stage to enrich messages when the enriched field should be used as part of the SOW key. To specify preprocessing for a topic, you add a `Preprocessing` directive to the `Topic` configuration for the SOW topic. :::tip Use `Preprocessing` when you need to change the value of a field that is part of the `Key` for the topic. Otherwise, use `Enrichment`. ::: Preprocessing field directives operate on a single message and construct fields based on that message. The results of the preprocessing expression are merged into the incoming message. Any field in the source message that is not changed or removed during preprocessing is left unchanged, so it is not necessary to include all fields in the message in the `Preprocessing` block. Since preprocessing fields apply to a specific message, preprocessing fields cannot specify the topic or message type in an XPath identifier. By default, AMPS serializes fields with a NULL value in the preprocessing result. Preprocessing fields can include a directive that specifies that if a field contains a NULL value, it should be removed from the set of fields rather than serialized. The directive `HINT OPTIONAL` applied to the XPath identifier specifies that if the result of the source expression is `NULL`, AMPS does not provide the value for the message type to serialize. For example, use the following directive to remove a `/source` field if the value provided is not in a specific list of values: ```xml IF(/source IN ('a','e','f'), /source, NULL) AS /source HINT OPTIONAL ``` For more information on constructing preprocessing fields, see [Constructing Preprocessing Fields](builtin_functions/constructing-fields#constructing-preprocessing-fields). ## Enriching Messages AMPS enrichment operates on a message after the SOW key is computed, but before an incoming delta publish is merged to an existing message, or the incoming message is written to the transaction log, stored to the SOW, used to update views, or delivered to subscribers. Use this enrichment stage when the enrichment process depends on the previous values of the message, or when the updated fields will not be used in the SOW key. To specify enrichment for a topic, you add an `Enrichment` directive to the configuration for the SOW topic. Enrichment field directives operate on a single message and construct fields based on that message. Enrichment expressions operate on the current message and change the current message. The results of the enrichment directives are merged into the incoming message. Any field in the source message that is not changed or removed during enrichment is left unchanged, so it is not necessary to include all fields in the message in the `Enrichment` directive. Since enrichment fields apply to a specific message, enrichment fields cannot specify the topic or message type in an XPath identifier. Within an enrichment expression, AMPS provides two special modifiers for XPath identifiers that specify whether an XPath identifier refers to the current incoming message or the previous state of the message. These modifiers apply only to the source expression, and cannot be used in special modifiers. They are: | Modifier | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OF CURRENT` | Specify that the XPath identifier refers to the incoming message. | | `OF PREVIOUS` |

    Specify that the XPath identifier refers to the previous state of the message in the SOW.

    If there is no record in the SOW for this message, all identifiers that specify `OF PREVIOUS` return `NULL`.

    | By default, AMPS serializes fields with a NULL value during enrichment. Enrichment fields can include a directive that specifies that if a field contains a NULL value, it should be removed from the set of fields rather than serialized. The directive `HINT OPTIONAL` applied to the XPath identifier specifies that if the result of the source expression is `NULL`, AMPS does not include the value in the set of XPath identifiers for the message type to serialize. For example, use the following directive to use remove a `/source` field if the value provided is not in a specific list of values: ```xml IF(/source IN ('a','e','f'), /source, NULL) AS /source HINT OPTIONAL ``` For more information on constructing enrichment fields, see [Constructing Enrichment Fields](builtin\_functions/constructing-fields.md#constructing-enrichment-fields). ## SOW Update and Enrichment Processing The following diagram presents a simplified, high-level view of the update process for an individual message. For the purposes of this diagram, views and conflated topics can be considered listeners on the SOW topic, while applications that connect to AMPS and the `on-publish` and `on-deliver` actions can be considered subscribers. It's important to keep in mind the following aspects of the SOW update sequence: * If the publish is disallowed due to topic-based entitlements or the publish filter specified for entitlements, there is no change to the state of the SOW. The entitlement filter (if one exists), is applied to the incoming message _before_ preprocessing, enrichment, or delta merge occurs. * AMPS records the enriched message in the transaction log and SOW file. When AMPS is configured for enrichment or your application performs a delta publish, the transaction log and SOW do _not_ preserve a record of the original message received by AMPS. Instead, they record the enriched and merged message. * Content filtering for subscriptions, views, and so forth is done on the final enriched and merged message, not on the original message as published. * Replication replicates the enriched message, since the enriched message is stored in the transaction log. Messages received over replication are recorded in the transaction log as they are received. Messages received over replication are not enriched again. ## SOW Preprocessing and Enrichment Examples This section shows some simple examples of SOW Preprocessing and Enrichment. ### Add A Field To add a field to every message in the topic, you just add an `Enrichment` directive to the configuration for the topic. For example, the `Enrichment` directive in the configuration below adds the `Name` of the AMPS instance that receives the publish as the `/publishSource` field of the message: ```xml showLineNumbers enrichment-example json /id ./sow/%n.sow AMPS_INSTANCE_NAME() as /publishSource ``` ### Set a Field to a Default Value The following example sets a field to a default value if there is no value in an existing record and no value for the incoming publish. ```xml showLineNumbers enrichment-example-default-value json /id ./sow/%n.sow COALESCE(/important OF CURRENT, /important OF PREVIOUS, 'default') as /important ``` The `Field` construction expression uses the `COALESCE` function, which takes a list of values and returns the first value that isn't `NULL`. If the current incoming publish has a value for `/important`, that value is used. Otherwise, if there is an existing record in the topic for the `/id`, and that record has a value for `/important`, that value is used. Finally, if neither the incoming publish or an existing record have a value for `/important`, the expression uses the value `'default'` for the `/important` field of the message. ### Create a Key Field The following example uses `Preprocessing` to create a field that is used to create the SOW key. The `Preprocessing` directive must be used in this case, since the intent is to modify a `Key` field. ```xml showLineNumbers key-creation-example json /bucket ./sow/%n.sow CRC32(CONCAT(/name, /orderId)) % 10 as /bucket ``` This topic maintains the last update to each of 10 "bucket" values. The `Preprocessing` directive constructs the bucket value using the /name and /orderId to divide records into buckets. --- # Client Status Events The AMPS engine will publish client status events to the internal `/AMPS/ClientStatus` topic whenever a client connects, issues a `logon` command, disconnects, enters or removes a subscription, queries a SOW or issues a `sow_delete`. AMPS sends a message if a client fails authentication. In addition, upon a disconnect, a client status message will be published for each subscription that the client had registered at the time of the disconnect. This mechanism allows any client to monitor what other clients are doing and is especially useful for publishers to determine when clients subscribe to a topic of interest. To help identify clients, it is recommended that clients send a `logon` command to the AMPS engine and specify a meaningful client name. This client name is used to identify the client within client status event messages, logging output and information on clients within the monitoring console. The client name must be unique if a transaction log is configured for the AMPS instance. Each message published to the client status topic will contain an `event` and a `client_name`. Depending on the event type, the message will contain other relevant fields. For example, the following JSON document demonstrates the message for a SOW query: ```JS showLineNumbers { "ClientStatus":{ "timestamp":"20250909T171919.976304Z", "event":"sow", "client_name":"test_client", "connection_name":"AMPS-Sample-any-tcp-9-242891694350073019", "correlation_id":null, "query_id":"1", "topic":"order", "filter":"/item/qty > 50", "options":"send_empties", "sub_id":"1", "auth_id":null, "entitlement_filter":null } } ``` The table below defines the header fields which may be returned as part of the subscription messages to the `/AMPS/ClientStatus` topic. | FIX | XML | JSON/BSON/MsgPack | Description | | ------- | ------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | 20062 | `Reason` | `reason` | Reason for the event (if applicable). This is present on client disconnect events to record the reason that the client disconnected. | | 20065 | `Timestamp` | `timestamp` | Timestamp at which AMPS processed the message. | | 20066 | `Event` | `event` | Command executed by the client. | | 20067 | `ClientName` | `client_name` | Client Name | | 20068 | `Tpc` | `topic` | Topic for the event (if applicable). | | 20069 | `Filter` | `filter` | Filter (if applicable). | | 20070 | `SubId` | `sub_id` | Subscription ID (if applicable). | | 20071 | `ConnName` | `connection_name` | Internal AMPS connection name. | | 20072 | `Options` | `options` | The options for the subscription (if applicable). | | 20073 | `QId` | `query_id` | The identifier for the query (if applicable). | | 20074 | `CorrelationID` | `correlation_id` | The correlation ID of the command (if applicable). | | 20080 | `ClientAddr` | `client_address` | The remote address of the client. | | 20081 | `AuthId` | `auth_id` | The authenticated identity of the client (if applicable). | | 20082 | `EntitlementFilter` | `entitlement_filter` | The entitlement filter for the command (if applicable). | Notice that for regular expression commands, options such as the `entitlement_filter` may vary by topic, and the topics that the command applies to may not be evaluated until messages are available for those topics. This means that these options may not available at the time status message is produced, and so they will not be provided. :::info Since the client status messages must be able to be serialized based on data in AMPS, the `/AMPS/ClientStatus` topic is not available for message types that rely on a fixed schema, such as `protobuf` or `struct`, and is not available for messages that have no fixed serialization such as `binary` or composite message types. ::: --- # Persisting Event Topics By default, AMPS event topics are not persisted to the SOW. However, because AMPS event topic messages are treated the same as all other messages, the event topics can be persisted to the SOW. Providing a topic definition with the appropriate `Key` definition can resolve that issue by instructing AMPS to persist the messages. The `Key` definition you specify must match the field name used for the message type specified in the SOW topic. That is, to track distinct records by client name for a SOW that uses `json`, you would use the following key: ```xml /client_name ``` While to track distinct records by client name for a SOW that uses `fix`, you would use the following key: ```xml /20067 ``` For example, to persist the last `/AMPS/SOWStats` message for each topic in `fix`, `json` and `xml` format, the following `Topic` sections could be added to the `SOW` section of the AMPS configuration file: ```xml showLineNumbers /AMPS/SOWStats ./sow/sowstats.fix.sow fix /20066 /AMPS/SOWStats ./sow/sowstats.json.sow json /topic /AMPS/SOWStats ./sow/sowstats.xml.sow xml /Topic ``` Every time an update occurs, AMPS will persist the `/AMPS/SOWStats` message and it will be stored three times, once to the `fix` SOW topic, once to the `json` SOW topic, and once to the `xml` SOW topic. Each update to the respective SOW topic will overwrite the record with the same `20066`, `topic` or `Topic` tag value. Doing this allows clients to now query the `SOWStats` topic instead of actively listening to live updates. --- # SOW Statistics Events AMPS can publish SOW statistics for each SOW topic which has been configured. To enable this functionality, specify the `SOWStatsInterval` in the configuration file. The value provided in `SOWStatsInterval` is the time between updates to the `/AMPS/SOWStats` topic. For example, the following would be a configuration that would publish `/AMPS/SOWStats` event messages every 5 seconds. ```xml showLineNumbers ... 5s ... ``` When a `SOWStatsInterval` is provided, AMPS publishes a message for each topic defined in the SOW at the requested interval. These messages contain basic information about the topic. The format of the `/AMPS/SOWStats` messages matches the message type of the connection that has requested the messages. For example, the following message provides information in JSON format about a topic named `a-sample-topic`. That topic is of message type `bflat`. ```JS showLineNumbers { "SOWStats":{ "message_type":"bflat", "topic":"a-sample-topic", "record_count":15021, "timestamp":"20161108T225452.280650Z" } } ``` In the `SOWStats` message above, `message_type` provides the name of the message type, `topic` specifies the name of the topic, `record_count` shows the number of records currently in the topic and `timestamp` includes the time the event was generated. The table below defines the header fields which may be returned as part of the subscription messages to the `/AMPS/SOWStats` topic. | FIX | XML | JSON/BSON/MsgPack | Definition | | ------- | ------------- | --------------------- | -------------------------------------------- | | 20007 | `MessageType` | `message_type` | The message type of the topic. | | 20065 | `Timestamp` | `timestamp` | Timestamp in which AMPS sent the message. | | 20066 | `Topic` | `topic` | Topic that statistics are being reported on. | | 20067 | `Records` | `record_count` | Number of records in the SOW topic. | For compatibility with systems that expect a consistent set of FIX tags across messages, AMPS provides a set of FIX tags that are unified with the tags used in the `/AMPS/ClientStatus` topic. To use the unified FIX tags, set the `AMPSVersionCompliance` configuration element to `5`. The following table lists the unified FIX tags: | FIX | Definition | | ----- | -------------------------------------------- | | 20007 | The message type of the topic. | | 20065 | Timestamp in which AMPS sent the message. | | 20068 | Topic that statistics are being reported on. | | 20075 | Number of records in the SOW topic. | :::warning The `/AMPS/SOWStats` topic is not available for `protobuf`, composite, `binary` or `struct` message types. AMPS will report statistics for topics of those types, but the format for the messages that AMPS generates must be a message type that can accept arbitrary fields, not a message type that requires a single schema. ::: --- # Event Topics AMPS publishes specific events to internal topics that begin with the `/AMPS/` prefix, which is reserved for AMPS only. For example, all client connectivity events are published to the internal `/AMPS/ClientStatus` topic. This allows all clients to monitor for events that may be of interest. :::info You can subscribe to an event topic using a content filter, just like any other topic within AMPS. ::: A client may subscribe to event topics on any connection with a message type that supports views. This includes all of the default message types and `bson`, but does not include the `binary` message type. Messages are delivered as the message type for the connection. For example, if the connection uses JSON messages, the event topic messages will be JSON. A connection that uses FIX will receive FIX messages from an event topic. --- # File Format Versions This chapter includes a simple cross-reference to the file format versions in recent AMPS releases. Using this guide, you can quickly determine whether your installation will need to run `amps_upgrade` when changing versions. |AMPS version|SOW version |Journal version|Acks version|Queue Metadata version| |------------|------------|---------------|------------|----------------------| |3.8 |v1.0 |v3 |v1.0 | | |3.9 |v1.0 |v3 |v1.0 | | |4.0 |v3.0 |v7 |v1.0 | | |4.3 |v3.0 |v7 |v1.0 | | |5.0 |v3.0 |v8 |v1.0 | | |5.2 |v3.0 |v8 |v1.0 | | |5.2.1 |v3.0 |v8 |v1.0 | | |5.2.2 |v3.0 |v8 |v1.0 | | |5.2.3 |v3.0 |v8 |v1.0 | | |5.2.4 |v3.0 |v8 |v1.0 | | |5.3.0 |v3.0 |v8 |v1.0 | | |5.3.1 |v3.0 |v8 |v1.0 | | |5.3.2 |v3.0 |v8 |v1.0 | | |5.3.3 |v3.0 |v8 |v1.0 | | |5.3.4 |v3.0 |v8 |v1.0 |v001 | |5.3.5 |v3.0 |v8 |v1.0 |v001 | An `amps_upgrade` tool is available for versions of AMPS 5.0 and earlier. The `amps_upgrade` tool updates the format of the data in the files that AMPS maintains. If there has been no version change to the format of the data, it is not necessary to run the tool to upgrade an AMPS instance. Notice that, while this chart describes the format of the files, AMPS versions may also change or extend the content of the files. In these cases, AMPS is guaranteed to be forward-compatible. That is, a given version of AMPS can process a file of the same file format version created on an earlier version of AMPS. However, later versions of AMPS may include changes in the content of the file that are not recognized by earlier versions of AMPS. This means that downgrading a file between major/minor versions of AMPS is not supported. :::warning The *contents* of AMPS files are not guaranteed to be fully backward compatible across changes to the major or minor version number. That is, while an AMPS instance of version X.4 can successfully load a SOW file created by version X.3, the version X.3 instance may not be able to process a file created or updated by the X.4 version. Contact 60East support for guidance if you intend to downgrade an instance of AMPS to an earlier version. ::: --- # Example: Regional Distribution AMPS is well suited for replicating messages to different regions, so clients in those regions are able to quickly receive and publish messages to a local instance. In this case, each region replicates all messages on the topic of interest to the other region(s). A variation on this strategy is to use a region tag in the content, and use content filtering so that each replicates messages intended for use in the other regions or worldwide. For this scenario, an AMPS instance in each region replicates to an instance in the other region. To reduce the memory and storage required for publishers, replication between the regions uses `async` acknowledgment, so that once the instance in one region has persisted the message, the message is acknowledged back to the publisher. In this case, clients in each region connect _only_ to the AMPS instance in that region. Bandwidth within regions is conserved, because each message is replicated once to the region, regardless of how many subscribers in that region will receive the message. Further, publishers are able to publish the message once to a local instance over a relatively fast network connection rather than having to publish messages multiple times to multiple regions. To configure this scenario, the AMPS instance in each region is configured to forward messages to known instances in the other region. --- # Example: Pair of Instances for Failover One of the most common scenarios is for two AMPS instances to replicate to each other. This replication is synchronous, so that both instances persist a message before AMPS acknowledges the message to the publisher. This makes a hot-hot pair. In the figure below, any messages published to the `orders` topic are replicated across instances, so both instances have the messages for `orders`. Two connections are shown in the diagram to demonstrate the required configuration. However, because these instances replicate to each other, AMPS can optimize this replication topology to use a single network connection (although AMPS treats this as two one-way connections that happen to share a single network connection.) Since AMPS replication is always peer-to-peer, clients can connect to either instance of AMPS when both are running. With this configuration, clients are configured with Instance 1 and Instance 2 as equivalent server addresses. If a client cannot connect to one instance, it tries the other. Given that both instances contain the same messages for `orders`, there is no functional difference in which instance a client connects to from the point of view of a publisher or subscriber. Each instance will contain the same messages for replicated topics. Messages can be published to either instance of AMPS at any time, and those messages will be replicated to the other instance. Since these instances are intended to be equivalent message sources (that is -- a client may fail over from one instance to another instance), these instances are configured to use `sync` acknowledgment to publishers. What that means is that, when a message is published to one of these instances, that instance does not acknowledge the message to the publisher as persisted until _both_ instances have written the message to disk (although the message can be delivered to subscribers once it is persisted locally). This means that a publisher using a publish store can fail over to either of these servers without risk of message loss. When a subscriber uses a bookmark store to manage bookmark replay, that subscriber can fail over safely between instances along any set of replication links that use `sync` replication without risk of message loss. However, a subscriber that uses bookmark replay should not fail over along a path that includes a replication link that uses `async` acknowledgment, since an instance of AMPS will not consider that link when determining if a message is persisted. --- # Example: Regional Distribution with HA Combining the first two scenarios allows your application to distribute messages as required and to have high availability in each region. This involves having two or more servers in each region, as shown in the figure below. Each region is configured as a `Group`, indicating that the instances within that region should be treated as equivalent, and are intended to have the same topics and messages. Within each group, the instances replicate to each other using `sync` acknowledgments, to ensure that publishers and subscribers can fail over between the instances. Since a client in a given region does not connect to a server outside the region, we can configure the replication links between the regions to use `async` acknowledgment, which could potentially reduce the amount of time that an application publishing to AMPS must store outgoing messages before receiving an acknowledgment that a given message is persisted. (Setting these links to use `async` acknowledgment does not affect the speed of replication or change the behavior of replication in any other way -- this setting only specifies when an instance of AMPS acknowledges the message as persisted.) The instances in each region are configured to be part of a `Group` for that region, since these instances are intended to have the same topics and messages. Within a region, the instances replicate to each other using `sync` acknowledgment. Replication connections to instances at the remote site use `async` acknowledgment. :::danger In a configuration like the one above, an application must only be allowed to fail over to other instances in its own region. Since replication to other regions uses `async` acknowledgments, a publisher may have received an acknowledgment that a given message is persisted before it is stored in instances in the other regions, or a subscriber may have received a persisted acknowledgment for a message that has not yet been persisted in other regions. ::: The instances are configured to downgrade (either via whatever monitoring/server health system is in use or via an AMPS action) to ensure that publishers do not retain an unworkably large number of messages in the event that one of the instances goes offline for an extended period of time. As with all connections where instances replicate to each other, this replication must be configured to have a connection in each direction, from `New York 1` to `New York 2` as well as from `New York 2` to `New York 1`. (AMPS may optimize this to a single network connection if possible.) Each instance at a site ensures that it provides passthrough replication to the other instance for both the local group and the remote groups. To optimize bandwidth, the instances at a site _may_ only provide passthrough to the remote instance for the local group. This ensures that once a message arrives at the local group (either from a remote group or over replication from a remote group), it is fully distributed to the local group. To optimize bandwidth, at the risk of increasing the chances of message loss if an entire region goes offline, each instance at a site only passes through messages from the local group to remote sites. This configuration balances fault-tolerance and performance and attempts to minimize the bandwidth consumed between the sites. Each instance at a site replicates to the remote sites. The instance specifies one `Destination` for each remote site, with the servers at the remote site listed as failover equivalents for the remote site. With the passthrough configuration, this ensures that each message is delivered to each remote site exactly once. Whichever server at the remote site receives the message, distributes it to the other server using passthrough replication. Notice that some features of AMPS, such as distributed queues (though not `LocalQueue` or `GroupLocalQueue`), require full passthrough to ensure correct delivery of messages. With this configuration, publishers at each site publish to a local AMPS instance. Subscribers subscribe to messages from their local AMPS instances. Both publishers and subscribers use the high availability features of the AMPS client libraries to ensure that if the primary local AMPS instance fails, they automatically fail over to the other instance. Replication is used to deliver both high availability and disaster recovery. In the table below, each row represents a replication destination. Servers in brackets are represented as sets of `InetAddr` elements in the `Destination` definition. | Server | Group | Destinations | PassThrough | | --------- | ------- | ---------------------------------------------------- | ----------- | | NewYork 1 | NewYork |
    • NewYork 2 /
      sync ack
    | `.*` | | | |
    • [London 1, London 2] / async ack
    | NewYork | | NewYork 2 | NewYork |
    • NewYork 1 /
      sync ack
    | `.*` | | | |
    • [London 1, London 2] / async ack
    | NewYork | | London 1 | London |
    • London 2 /
      sync ack
    | `.*` | | | |
    • [NewYork 1, NewYork 2] / async ack
    | London | | London 2 | London |
    • London 1 /
      sync ack
    | `.*` | | | |
    • [NewYork 1, NewYork 2] / async ack
    | London | --- # Details of High Availability AMPS High Availability, which includes multi-site replication and the transaction log, is designed to provide long uptimes and speedy recovery from disasters. Replication allows deployments to improve upon the already rock-solid stability of AMPS. Additionally, AMPS journaling provides the persisted state necessary to make sure that client recovery is fast, painless, and error free. ## Guaranteed Publishing An interruption in service while publishing messages could be disastrous if the publisher doesn't know which message was last persisted to AMPS. To prevent this from happening, AMPS has support for _guaranteed publishing_. With guaranteed publishing, the AMPS client library is responsible for retaining and retransmitting the message until the server acknowledges that the message has been successfully persisted to the server and has been acknowledged as persisted by any replication destinations that are configured for synchronous replication. This means that each message always has at least one part of the system (either the client library or the AMPS server) responsible for persisting the message, and if failover occurs, that part of the system can retain and recover the message as necessary. An important part of guaranteed publishing is to be able to uniquely identify messages. In AMPS, the unique identifier for a message is a _bookmark_, which is formed from a combination of a number derived from the client name and a _sequence number_ managed by the client. A sequence number is simply an ever-increasing number assigned by a publisher to any operation that changes the state of persistent storage in AMPS (that is, `publish` or `sow_delete` commands). The AMPS clients automatically manage sequence numbers when applications use the named methods or the `Command` interface and a `PublishStore` is set on the client object. The libraries set the sequence number on each published message, ensure that the sequence number increases as appropriate, and initialize the sequence number at `logon` using information retrieved from the server acknowledgment of the `logon` command. The sequence number is also used for acknowledgments. The `persisted` acknowledgment returned in response to a `publish` command contains the sequence number of the last message persisted rather than the `CommandId` of the publish command message (for more details see [Acknowledgment Conflation and Publish Acknowledgments](../acks/publish-acks)). The `logon` command supports a `processed` acknowledgment message, which will return the `Sequence` of the last record that AMPS has persisted. When the `processed` acknowledgment message is returned to the publisher, the `Sequence` corresponds to the last message persisted by AMPS. The publisher can then use that sequence to determine if it needs to 1) re-publish messages that were not persisted by AMPS, or 2) continue publishing messages from where it left off. Acknowledging persisted messages across logon sessions allows AMPS to guarantee publishing. The HAClient classes in the AMPS clients manage sequence numbers, including setting a meaningful initial sequence number based on the response from the `logon` command, automatically. :::info Connections should request a `processed` acknowledgment message with every `logon` command. This ensures that the `Sequence` returned in the acknowledgment message matches the publisher's last published message. The 60East AMPS clients do this automatically when using the named logon methods. If you are building the command yourself or using a custom client, you may need to add this request to the command yourself. ::: In addition to the acknowledgment messages, AMPS also keeps track of the published messages from a client based on the client's name. The client name is set during the `logon` command, so to set a consistent client name, it is necessary for an application to log on to AMPS. A logon is required by default in AMPS versions 5.0 and later, and optional by default in AMPS versions previous to 5.0. :::danger All publishers must set a unique client name field when logging on to AMPS. This allows AMPS to correlate the sequence numbers of incoming publish messages to a specific client, which is required for reliable publishing, replication, and duplicate detection in the server. In the event that multiple publishers have the same client name, AMPS can no longer reliably correlate messages using the publish sequence number and client name. When a transaction log is enabled for AMPS, it is an error for two clients to connect to an instance with the same name. ::: ## Durable Publication and Subscriptions The AMPS client libraries include features to enable durable subscription and durable publication. In this chapter we've covered how publishing messages to a transaction log persists them. We've also covered how the transaction log can be queried (subscribed) with a bookmark for replay. Now, putting these two features together yields _durable subscriptions_. ### Durable Subscriptions A _durable subscriber_ is one that receives all messages published to a topic (including a regular expression topic), even when the subscriber is offline. In AMPS this is accomplished through the use of the bookmark subscription on a client. Implementation of a _durable subscription_ in AMPS is accomplished on the client by persisting the last observed bookmark field received from a subscription. This enables a client to recover and resubscribe from the exact point in the transaction log where it left off. ### Durable Publishing A durable publisher maintains a persistent record of messages published until AMPS acknowledges that the message has been persisted. In the AMPS system, a durable publisher stores outgoing messages until AMPS sends a `persisted` acknowledgment that indicates the message has been persisted or cannot be persisted due to an error. Once the message is acknowledged, the publisher can remove the message. Should the publisher fail over, it can resend any messages in the store that have not been acknowledged by AMPS. The AMPS server uses the sequence number in the message to discard any duplicates. This helps ensure that no messages are lost, and provides fault-tolerance for publishers. The sequence number is also used in `persisted` acknowledgment messages to indicate which message, or messages, the acknowledgment applies to. To reduce network bandwidth, the AMPS server will conflate successful `persisted` acknowledgments when a transaction log is configured. That is, the server will acknowledge messages at a regular interval, as described in the section on [Acknowledgment Conflation and Publish Acknowledgments](../acks/publish-acks). The AMPS client libraries provide publish stores that manage durable publishing and maintaining and assigning sequence numbers to messages. ### Client Support The AMPS client libraries each provide different implementations of persistent subscriptions and persistent publication. Please refer to the _High Availability_ chapter of the _Developer Guide_ for the language of your choice to see how this feature is implemented. ## Heartbeat in High Availability Use of the heartbeat feature allows your application to quickly recover from detected connection failures. By default, connection failure detection occurs when AMPS receives an operating system error on the connection. This default method may result in unpredictable delays in detecting a connection failure on the client, particularly when failures in network routing hardware occur, and the client primarily acts as a subscriber. The heartbeat feature of the AMPS server and the AMPS clients allows connection failure to be detected quickly. Heartbeats ensure that regular messages are sent between the AMPS client and server on a predictable schedule. The AMPS server assumes disconnection has occurred if these regular heartbeats cease, ensuring disconnection is detected in a timely manner. Heartbeats are initialized by the AMPS client by sending a `heartbeat` message to the AMPS server. To enable heartbeats in your application, refer to the _High Availability_ chapter in the Developer Guide for your specific client language. --- # Message Ordering Considerations AMPS uses the name of the publisher and the sequence number assigned by the publisher to ensure that messages from each publisher are delivered to a subscription in order (see the [Message Ordering](../pub-sub/ordering) section for more details). However, AMPS does not enforce order across publishers. AMPS also guarantees that the transaction log on each instance is written in the same order in which that instance delivered messages to subscribers and that subsequent replays from the transaction log for the instance will reproduce the exact order in which messages were delivered originally. In a failover situation, messages from different publishers may be interleaved in a different order on different servers, even though the message stream from each publisher is preserved in order. Each instance preserves the order in which messages were processed by that instance and enforces that order. The AMPS client libraries include bookmark store implementations that are designed to ensure that when a bookmark subscription (that is, a replay from the transaction log) fails over across instances that replicate to each other, a subscription can be resumed without missing messages, even if multiple replicated instances receive publishes and, therefore, have different interleaved order in their transaction logs. --- # Example: Hub and Spoke / Expandable Mesh For more complex replication topologies, or in a situation where an installation may want to scale out to accommodate an ever-increasing number of subscribers, consider using a "hub and spoke" topology. This topology is particularly useful in cases where a large number of applications need to operate over the same data, but the applications themselves are largely independent of each other, where data is consumed in a different region or different organization than where the data originates, or in cases where given applications require intensive CPU or memory resources to work with the data, whereas other applications using the same data do not require these resources. For example, if two applications have different CPU-intensive views over the same data, isolating those applications into separate application instances can help to reduce the resources required for any one instance. In this topology, replication is handled by AMPS instances dedicated to managing replication. In this strategy, each instance has one of three distinct roles: * An _ingestion_ instance accepts messages from a publisher into the AMPS replication fabric. All ingestion instances replicate to each other (using `sync` acknowledgment) and replicate to the hub (also using `sync` acknowledgment). The ingestion instances do not define a state of the world. * One or more _hub_ instances that accept messages from the ingestion instances and replicate those messages to the application instances. The hub instances do not replicate back to the ingestion instances, and they do not define a state of the world. * The _application_ instances provide messages to applications that use the messages. These instances do not replicate back to the hub instances. If an application will use multiple instances, these instances replicate to each other using `sync` acknowledgment. The application instances define the state of the world as needed -- any `Topics`, `Views`, `ConflatedTopics`, `LocalQueues`, or `GroupLocalQueues` that the application will use. Different applications may use different application instances: each application instance only needs to define the state of the world that the applications that use that instance need. This architecture provides decoupling between publishers and applications, and decoupling between different applications that use the same message stream. This topology also reduces the risk and expense of adding more instances for application use. Only the "hub" instances need to be updated to add or remove application instances. Since the "hub" maintains only messages for replication (no state of the world is defined on the "hub" instances), adding or removing a destination at the hub instance is very efficient. Recovery times for the hub are very quick since the only state that needs to be recovered is the state of the transaction log itself. In the simplest configuration, the "hub" instance or instances simply pass through all messages and all topics to all downstream instances, leaving the application instances to determine what topics should be replicated. In more sophisticated configurations, the "hub" instances can direct topics for specific applications to a specific set of instances. The hub and spoke topology has the following advantages: * Easy to add and remove instances to a replication fabric. * Allows the ability to create autonomous groups of instances servicing a given application. * High resilience to failures within the application instances. * In many situations, reduces the bandwidth required to keep a large number of instances up to date (as compared with direct replication between the instances). The hub and spoke topology has the following limitations: * In some topologies, may have higher latency for active publishes. * Does _not_ support fully distributed queues (use local queues or group local queues on a subset of instances instead). * Requires an instance of AMPS (or two, for HA) that does not have client activity. * Requires exclusion of replication validation to the hub instance. --- # Slow Client Management and Capacity Limits AMPS provides the ability to manage memory consumption for clients to prevent slow clients, or clients that require large amounts of state, to disrupt service to the instance. Sometimes, AMPS can publish messages faster than an individual client can consume messages, particularly in applications where the pattern of messages includes "bursts" of messages. Clients that are unable to consume messages faster or equal to the rate messages are being sent to them are "slow clients". By default, AMPS queues messages for a slow client in memory to grant the slow client the opportunity to catch up. However, scenarios may arise where a client can be over-subscribed to the point that the client cannot consume messages as fast as messages are being sent to it. In particular, this can happen with the results of a large SOW query, where AMPS generates all of the messages for the query much faster than the network can transmit the messages. Some features, such as conflated subscriptions, aggregated subscriptions and pagination require AMPS to buffer messages in memory for extended periods of time. Without a way to set limits on memory consumption, subscribers using these features could cause AMPS to exceed available memory and reduce performance or exit. Memory capacity limits, typically called **slow client management**, are one of the ways that AMPS prevents slow clients, or clients that consume large amounts of memory, from disrupting service to other clients connected to the instance. 60East recommends enabling slow client management for instances that serve high message volume or are mission critical. There are two methods that AMPS uses for managing slow clients to minimize the effect of slow clients on the AMPS instance: 1. **Client Offlining** - When client offlining occurs, AMPS buffers the messages for that client to disk. This relieves pressure on memory, while allowing the client to continue processing messages. 2. **Disconnection** - When disconnection occurs, AMPS closes the client connection, which immediately ends any subscriptions, in-progress `sow` queries, or other commands from that client. AMPS also removes any offlined messages for that client. AMPS provides resource pool protection, to protect the capacity of the instance as a whole, and client-level protection, to identify unresponsive clients. ## Resource Pool Policies AMPS uses resource pools for memory and disk consumption for clients. When the memory limit is exceeded, AMPS chooses a client to be offlined. When the disk limit is exceeded, AMPS chooses a client to be disconnected. When choosing which client will be offlined or disconnected, AMPS identifies the client that uses the largest amount of resources (memory and/or disk). That client will be offlined or disconnected. The memory consumption calculated for a client includes both buffered messages and memory used to support features such as conflated subscriptions and aggregated subscriptions. AMPS allows you to use a global resource pool for the entire instance, a resource pool for each transport, or any combination of the two approaches. By default, AMPS configures a global resource pool that is shared across all transports. When an individual transport specifies a different setting for a resource pool, that transport receives an individual resource pool. For example, you might set high resource limits for a particular transport that serves a mission-critical application, allowing connections from that application to consume more resources than connections for less important applications. The following table shows resource pool options for slow client management: | Element | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `MessageMemoryLimit` |

    The total amount of memory to allocate to messages before offlining clients.

    Default: 10% of total host memory or 10% of the amount of host memory AMPS is allowed to consume (as reported by `ulimit -m` ), whichever is lowest.

    | | `MessageDiskLimit` |

    The total amount of disk space to allocate to messages before disconnecting clients.

    Default: 1GB or the amount specified in the `MessageMemoryLimit`, whichever is highest.

    | | `MessageDiskPath` |

    The path to use to write offline files.

    Default: `/var/tmp`

    | ## Individual Client Policies AMPS also allows you to set policies that apply to individual clients. These policies are applied to clients independently of the instance level policies. For example, a client that exceeds the capacity limit for an individual client will be disconnected, even if the instance overall has enough capacity to hold messages for the client. As with the Resource Pool Policies, Transports can either use instance-level settings or create settings specific to that transport. The following table shows the client level options for slow client management: | Element | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `ClientMessageAgeLimit` |

    The maximum amount of time for the client to lag behind. If a message for the client has been held longer than this time, the client will be disconnected. This parameter is an AMPS time interval (for example, `30s` for 30 seconds, or `1h` for 1 hour).

    Notice that this policy applies to all messages and all connections.

    If you have applications that will consume large result sets (SOW queries) over low-bandwidth network connections, consider creating a separate transport with the age limit set higher to allow those operations to complete.

    Default: No age limit

    | | `ClientMaxCapacity` |

    The amount of available capacity a single client can consume. Before a client is offlined, this limit applies to the `MessageMemoryLimit`. After a client is offlined, this limit applies to the `MessageDiskLimit`. This parameter is a percentage of the total.

    Default: `50%` (previous versions defaulted to `100%`)

    | Client offlining can require careful configuration, particularly in situations where applications retrieve large result sets from SOW queries when the application starts up. More information on tuning slow client offlining for AMPS is available in [Slow Client Offlining for Large Result Sets](../operation/operations-best-practices.md#slow-client-offlining-for-large-result-sets) section in the [Operations Best Practices](../operation/operations-best-practices) section. --- # High Availability AMPS is designed for high performance, mission-critical applications. Those systems typically need to meet availability guarantees. To reach those availability guarantees, systems need to be fault tolerant. It's not realistic to expect that networks will never fail, components will never need to be replaced, or that servers will never need maintenance. For high availability, you build applications that are fault tolerant: that keep working as designed even when part of the system fails or is taken offline for maintenance. AMPS is designed with this approach in mind. It assumes that components will occasionally fail or need maintenance and helps you to build systems that meet their guarantees even when part of the system is offline. When you plan for high availability, the first step is to ensure that each part of your system has the ability to continue running and delivering correct results if any other part of the system fails. You also ensure that each part of your system can be independently restarted without affecting the other parts of the system. The AMPS server includes the following features that help ensure high availability: * **Transaction logging** writes messages to persistent storage. In AMPS, the transaction log is not only the definitive record of what messages have been processed, it is also fully queryable by clients. Highly available systems make use of this capability to keep a consistent view of messages for all subscribers and publishers. The AMPS transaction log is described in detail in the chapter on [Record and Replay Messages](./txlog). * **Replication** allows AMPS instances to copy messages between instances. AMPS replication is peer-to-peer, and any number of AMPS instances can replicate to any number of AMPS instances. Replication can be filtered by topic. By default, AMPS instances only replicate messages published to that instance. An AMPS instance can also replicate messages received via replication using passthrough replication: the ability for instances to pass replication messages to other AMPS instances. * **Heartbeat monitoring** to actively detect when a connection is lost. Each client configures the heartbeat interval for that connection. :::info The only communication between instances of AMPS is through replication. AMPS instances do not share state through the filesystem or any out-of-band communication. AMPS high availability and replication do not rely on a quorum or a controller instance. Each instance of AMPS processes messages independently. Each instance of AMPS manages connections and subscriptions locally, for maximum availability. ::: The AMPS client libraries include the following features to help ensure high availability: * **Heartbeat monitoring** to actively detect when a connection is lost. As mentioned above, the interval for the heartbeat is configurable on a connection-by-connection basis. The interval for heartbeat can be set by the client, allowing you to configure a longer timeout on higher latency connections or less critical operations, and a lower timeout on fast connections or for clients that must detect failover quickly. * **Automatic reconnection and failover** allows clients to automatically reconnect when disconnection occurs, and to locate and connect to an active instance. * **Reliable publication** from clients, including an optional persistent message store. This allows message publication to survive client restarts as well as server failover. * **Subscription recovery and transaction log playback** allows clients to recover the state of their messaging after restarts. When used with a regular subscription or a sow and subscribe, the HAClient can restore the subscription at the point the client reconnects to AMPS. When used with a bookmark subscription, the HAClient can provide the ability to resume at the point the client lost the connection. These features guarantee that clients receive all messages published in the order published, including messages received while the clients were offline. Replay and resumable subscription features are provided by the transaction log, as described in [Record and Replay Messages](./txlog). For details on each client library, see the developer's guide for that library. Further samples can be found in the client distributions, available from the 60East website at [http://www.crankuptheamps.com/develop](http://www.crankuptheamps.com/develop). --- # Installing AMPS To install AMPS, unpack the distribution for your platform where you want the binaries and libraries to be stored. For the remainder of this guide, the installation directory will be referred to as `$AMPSDIR` as if an environment variable with that name was set to the correct path. Within `$AMPSDIR` are the following sub-directories: | Directory | Description | |-------------|-------------------------------------------| | bin | AMPS engine binaries and utilities | | docs | Documentation | | lib | Library dependencies | | sdk | Include files for the AMPS extension API | :::tip AMPS client libraries are available as a separate download from the AMPS website. See the AMPS developer page at [https://www.crankuptheamps.com/develop](http://www.crankuptheamps.com/develop) to download the latest libraries. ::: --- # Starting AMPS The AMPS engine binary is named `ampServer` and is found in `$AMPSDIR/bin`. Start the AMPS engine with a single command line argument that includes a valid path to an AMPS configuration file. You use the configuration file to enable and configure the AMPS features that your application will use. This guide discusses the full set of configuration options for each feature. The AMPS server generates a minimal sample configuration file with the `--sample-config` option. You can save the sample configuration file to `$AMPSDIR/amps_config.xml` with the following command line: ```bash $AMPSDIR/bin/ampServer --sample-config > $AMPSDIR/amps_config.xml ``` :::info The sample configuration file generated by AMPS includes a very minimal configuration. The client language distributions include a sample configuration file that sets up AMPS to work with the samples provided with that client, and this guide contains a full description of the configuration items with sample configuration snippets. The server sample configuration only provides configuration for subscribe/publish use of AMPS, and does not include any persistence for AMPS messages. The file enables the instance administrative interface (the "Galvanometer"), including the ability to subscribe to topics using a websocket connection from the instance administrative interface. A production configuration would likely provide persistent event and error logging to a file to allow an operations team to troubleshoot the instance and would typically persist monitoring statistics to a file. Such a configuration would likely enable additional message delivery features for certain topics and would also include configuration for high-availability and disaster recovery. The configuration would typically configure AMPS actions to perform routine maintenance. ::: :::info AMPS uses the current working directory for storing files (logs and persistence) for any relative paths specified in the configuration. While this is important for real deployments, the sample configuration used in this chapter does not persist anything, so you can safely start AMPS from any working directory using this configuration. ::: :::tip On older processor architectures, `ampServer` will start the `ampServer-compat` binary. The `ampServer-compat` binary avoids using hardware instructions that are not available on these systems. You can also set the `AMPS_PLATFORM_COMPAT` environment variable to force `ampServer` to start the `ampServer-compat` binary. 60East recommends using this option only on systems that do not support the hardware instructions used in the standard binary. The `ampServer-compat` binary will not perform as well as `ampServer`, since it uses fewer hardware optimizations. ::: Once you have a configuration file saved to `$AMPSDIR/amps_config.xml` you can start AMPS with that file as follows: ```bash $AMPSDIR/bin/ampServer $AMPSDIR/amps_config.xml ``` If your first start-up is successful, you should see AMPS display a simple message similar to the following to let you know that your instance has started correctly. ```bash AMPS A.B.C.D.973814.e1a57f7 - Copyright (c) 2006-202X 60East Technologies Inc. (Built: XXXX-YY-ZZT00:26:45Z) ``` The version numbers and dates will be appropriate for the version that you've started. If you see this, congratulations! You have successfully cranked up the AMPS! ## Command Line Options The AMPS server binary supports the following command line options: | Option | Effect | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `--verify-config` | Parse and verify the specified configuration file, then exit. | | `--sample-config` | Produce a minimal AMPS config.xml file to standard output, then exit. | | `--dump-config` | Process the specified configuration file, resolving any Include directives and expanding environment variables. Dump the resulting file to standard output. | | `--version` | Print the AMPS version string, then exit. | | `--help` | Print usage information for the command line options accepted by the `ampServer` program, then exit. | | `--daemon` | Run AMPS as a daemon process. | | `-D=` |

    Set the specified environment variable to the specified value when running the AMPS process. AMPS accepts any number of `-D` options.

    For example, to set the variable `AMPS_PATH` to `/mnt/fast/AMPS` use the command line option `-DAMPS_PATH=/mnt/fast/AMPS`

    | --- # Installing and Starting AMPS This section describes how to install and start AMPS. It describes the file structure of the AMPS distribution and how to configure a simple AMPS instance. The [Getting Started with AMPS](/docs/intro-guide/getting_started) section in the[ Introduction to AMPS](/docs/intro-guide/intro) covers setting up a basic development environment. This section includes a reference to the AMPS server [command line options](/docs/intro-guide/starting.md#command-line-options) and provides information to help create a production deployment of AMPS. --- # Documentation Conventions This manual is an introduction to the 60East Technologies AMPS product. It assumes that you have a working knowledge of Linux and uses the following conventions. | Construct | Usage | | ----------------------------------------------- | ------------------------------------------ | | Text | Standard document text | | `Code` | Inline code fragment | | _Variable_ | Variables within commands or configuration | |

    `Parameter`

    (required)

    | Required parameters in parameter tables | | `Optional` | Optional parameters in parameter tables | The AMPS documentation also includes the following types of notes: :::info Inside boxes with this icon, you will find usage tips or extra information. ::: :::tip Inside boxes with this icon, you will find information that's important to keep in mind when working with AMPS. These are typically recommendations that should generally be followed, but may not be applicable in special cases. ::: :::warning Inside boxes with this icon, you will find important information and guidelines that require special consideration or caution when using AMPS to ensure the proper functioning of the system and to avoid any potential issues or risks. ::: :::danger Inside boxes with this icon, you will find usage warnings or information that is critical for ensuring that AMPS functions correctly. ::: Additionally, here are the constructs used for displaying content filters, XML, code, command line, and script fragments. ```bash (expr1 = 1) OR (expr2 = 2) OR (expr3 = 3) OR (expr4 = 4) OR (expr5 = 5) OR (expr6 = 6) OR (expr7 = 7) OR (expr8 = 8) ``` Command lines will be formatted as in the following example: ```bash $ find . -name *.java ``` --- # Organization of this Guide This manual is divided into the following parts: * _Part One_ presents introductory material and a brief overview of AMPS * _Part Two_ explains the features of AMPS, including information on the following features: * [Subscribe and Publish](../pub-sub), the basic building blocks of AMPS applications * The expression language and functions used to take advantage of the content-aware features of AMPS are covered in [AMPS Expressions](../amps-expressions) and [AMPS Functions](../amps-functions) * [Record and Replay Messages](../txlog) using the AMPS transaction log * Competitive message consumption with [Message Queues](../queues) * The [Message Types](../message-types) that AMPS supports for content-aware processing * Current value caching and database functions using [State of the World (SOW) topics](../sow) State of the World topics enable many of the other advanced features in AMPS, such as: * [Aggregation and Analytics](../views) * [Querying the State of the World](../sow-queries) * [Out-of-Focus Messages](../oof) * [State of the World Message Enrichment](../enrichment) * [Incremental Message Updates](../delta-publish) * [Receiving Only Updated Fields](../delta-subscribe) This section also contains detailed chapters on specific topics, such as the AMPS filter language. Both application developers and administrators should become familiar with this section. * _Part Three_ discusses AMPS deployment and operations, including: * [Running AMPS as a Linux Service](../daemon) * [Logging](../logging) * [Event Topics](../events) * [Utilities](../utilities) * [Monitoring AMPS](../monitoring) * [Configuring AMPS for Automation with Actions](../actions) * [Replicating Messages Between Instances](../replication) * [Highly Available AMPS Installations](../ha) * [Operation and Deployment](../operation) * [Securing AMPS](../securing) * [Troubleshooting AMPS](../troubleshooting) This section is most useful for those with a focus on AMPS operations, although the information presented here is helpful for developers who want to design high-performance, high-availability applications that are easy to deploy and maintain. * Additional chapters provide reference information: * [Optionally-Loaded Modules](../optional-modules) describes special-purpose modules that are included in the AMPS distribution but are not loaded by default * [File Format Versions](../file-format-versions) lists the file formats used by each AMPS version --- # Product Overview AMPS, the Advanced Message Processing System, is built around an incredibly fast messaging engine that supports both publish-subscribe messaging and queuing. AMPS combines the capabilities necessary for scalable high-throughput, low-latency messaging in realtime deployments such as in financial services. AMPS goes beyond basic messaging to include advanced features such as high availability, historical replay, aggregation and analytics, content filtering and continuous query, last value caching, focus tracking, and more. Furthermore, AMPS is designed and engineered specifically for next generation computing environments. The architecture, design and implementation of AMPS allows the exploitation of parallelism inherent in emerging multi-socket, multi-core commodity systems and the low-latency, high-bandwidth of 10Gb Ethernet and faster networks. AMPS is designed to detect and take advantage of the capabilities of the hardware of the system on which it runs. AMPS does more than just route and deliver messages. AMPS was designed to lower the latency in real-world messaging deployments by focusing on the entire lifetime of a message from the message's origin to the time at which a subscriber takes action on the message. AMPS considers the full message lifetime, rather than just the "in flight" time, and allows you to optimize your applications to conserve network bandwidth and subscriber CPU utilization -- typically the first elements of a system to reach the saturation point in real messaging systems. AMPS offers both topic and content based subscription semantics, which makes it different than most other messaging platforms. Some of the highlights of AMPS include: * Topic and content based publish and subscribe * Message queuing, including content-based filtering and configurable strategies for delivery fairness * Client development kits for popular programming languages such as Java, C#, C++, C, Python, and JavaScript * Built-in support for FIX, NVFIX, JSON, BSON, MessagePack, BFlat, Google Protocol Buffer and XML messages. AMPS also supports uninterpreted binary messages, and allows you to create composite message types from existing message types. * State of the World queries * Historical State of the World queries * Easy to use command interface * Full Perl compatible regular expression matching * Content filters with SQL92 `WHERE` clause semantics * Built-in latency statistics and client status monitoring * Advanced subscription management, including delta publish and subscriptions and out-of-focus notifications * Basic CEP capabilities for real-time computation and analysis * Aggregation within topics and joins between topics, including joins between different message types * Replication for high availability * Fully queryable transaction log * Message replay functionality * Fully-integrated authentication and entitlement system, including content-based entitlement for fine-grained control * Optional encryption (SSL) between client and server * Extensibility API for adding message types, user-defined functions, user-specified actions, authentication, and entitlement functionality --- # Requirements ## Software Requirements The AMPS server is supported on the following platforms: * Linux 64-bit (2.6 kernel or later) on x86 compatible processors :::tip While 2.6 is the minimum kernel version supported, AMPS will select the most efficient mechanisms available to it and as a result, reaps greater benefit from more recent kernel and CPU versions. ::: The AMPS distribution contains all of the supporting libraries and dependencies needed to run on a typical Linux server installation: no further software is required. Some utilities provided with the AMPS server have additional dependencies. These utilities are not required to run the server, but can make it easier to troubleshoot and test on the system that hosts the AMPS instance: * `spark`, a basic command line client that supports a subset of AMPS functionality, requires Java 1.7 or later. * The utilities for inspecting AMPS files (`amps_sow_dump`, `amps_clients_ack_dump`, and so on) require a Python installation. * `amps-grep` requires a Python installation. --- # Technical Support For an outline of your specific support policies, please see your 60East Technologies License Agreement. Support contracts can be purchased through your 60East Technologies account representative. ## Support Steps You can save time if you complete the following steps before you contact 60East Technologies Support: 1. _**Check the documentation**_ The problem may already be solved and documented in the _User Guide_ for the product. 60East Technologies also provides answers to frequently asked support questions on the support website at: [http://crankuptheamps.com/support](/support). 2. _**Isolate the problem**_ If you require Support Services, please isolate the problem to the smallest test case possible. Capture erroneous output into a text file along with the commands used to generate the errors. 3. _**Collect your information**_ * Your product version number. * Your operating system and its kernel version number. * The expected behavior, observed behavior and all input used to reproduce the problem. * Submit your request. * If you have a minidump file, be sure to include that in your email to [crash@crankuptheamps.com](mailto:crash@crankuptheamps.com). The AMPS version number used when reporting your product version number follows a format listed below. The version number is composed of the following: ``` MAJOR.MINOR.FEATURE.HOTFIX.TIMESTAMP.TAG ``` ## AMPS Versioning and Certification Each AMPS version number component has the following breakdown: | Component | Description | Minimum Verification | | -------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------| | `MAJOR` |

    Increments when there are any backward-incompatible changes in functionality, file formats, client network formats or configuration; or when deprecated functionality is removed.

    May introduce major new functionality or include internal improvements that introduce major behavioral changes.

    | Megacert | | `MINOR` |

    Increments when functionality is added in a backwards-compatible way, or when functionality is deprecated.

    May include internal improvements, including internal improvements that introduce minor behavioral changes or changes to network formats used only by the AMPS server (such as replication).

    | Megacert | | `FEATURE` |

    Increments for previews of new features.

    May introduce behavioral changes to fix incorrect behavior, enable new functionality or to enhance performance.

    May include internal enhancements that do not introduce behavioral changes.

    Note: A feature level of `0` indicates a long-term stable release. A feature level above zero indicates the current feature level (a preview of the next long-term stable release).

    | Kilocert | | `HOTFIX` |

    A release for a critical defect impacting a customer. A hotfix release is designed to be 100% compatible with the release it fixes (that is, a release with same `MAJOR.MINOR.FEATURE` version).

    May introduce behavioral changes to fix incorrect behavior. May document previously undocumented features or extend surface area to improve usability for existing features.

    | Cert | | `TIMESTAMP` |

    Proprietary build timestamp.

    | (does not affect verification level) | | `TAG` |

    Identifier that corresponds to precise code used in the release.

    | (does not affect verification level) | The certification levels are defined in the following table. Notice that, in all cases, 60East will certify at a higher level if time permits or if a change involves a critical part of AMPS (such as replication or internal utility classes that are widely used). | Certification Level | Description | Time to Certify | | ---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------| | Megacert |

    Performance and long-haul testing.

    Full regression suite and stress-testing suite, including replication testing and application scenario tests.

    Full unit testing suite, including new unit tests to verify correct behavior of bugfixes in this release.

    | less than 2 weeks | | Kilocert |

    Full regression suite and stress-testing suite, including replication testing and application scenario tests.

    Full unit testing suite, including new unit tests to verify correct behavior of bugfixes in this release.

    | less than 1 week | | Cert |

    Full unit testing suite, including new unit tests to verify correct behavior of bugfixes in this release.

    Replication testing suite if release affects replication code.

    | 4 hours | ## Contacting 60East Technologies Support Please contact 60East Technologies Support Services according to the terms of your 60East Technologies License Agreement. Support is offered through the United States: | | | |-----------|-------------------------------------------------------------------------| | Web: | [http://www.crankuptheamps.com](http://www.crankuptheamps.com) | | E-mail: | [support@crankuptheamps.com](mailto:support@crankuptheamps.com) | | Support: | [http://crankuptheamps.com/support](/support) | Other support options (such as support via phone), may be available depending on the terms of your support agreement. --- # Introduction Thank you for choosing the Advanced Message Processing System (AMPS) from 60East Technologies. AMPS is a feature-rich message processing system that delivers previously unattainable low-latency and high-throughput performance to users. AMPS provides both publish-and-subscribe messaging and high-performance message queuing. AMPS is designed to help you quickly and easily develop and deploy data-intensive applications with demanding requirements for low latency and high performance. AMPS combines aspects of a traditional message bus, message queue, database, view server, analytics and event processing engine. The features that AMPS provides are designed to be easy to use, to work well together, and to provide high performance. ### Documentation Resource Overview The 60East documentation is intended to be used with a working (development) environment of AMPS available so that you can quickly explore the concepts discussed. 60East recommends starting with the [Introduction to AMPS](/docs/intro-guide/intro) guide to become familiar with AMPS, and then reading the sections of the AMPS User Guide for the features that your application will use. * The [Introduction to AMPS](/docs/intro-guide/intro) provides an overall introduction to AMPS, including information on setting up a development environment, the basic concepts and features of AMPS, and general advice on which features [combine effectively for specific scenarios](/docs/intro-guide/feature\_guide/). The 60East documentation is intended to be used with a working (development) environment of AMPS available so that you can quickly explore the concepts discussed. * The [AMPS Evaluation Guide](/docs/amps-eval-guide/eval) provides advice on evaluating AMPS, included a suggested evaluation process, tips on monitoring and measuring performance in an evaluation environment, and information on how to effectively partner with 60East on an evaluation of AMPS. * The _AMPS User Guide_ -- this guide -- provides a complete overview of AMPS features, covering instance deployment, administration, and configuration. It also explains the AMPS configuration file and the options for defining instance behavior. * The [Deployment Checklist](/docs/deployment-checklist/checklist) is a short document providing recommendations for deploying AMPS into a shared environment, whether that environment will be used for production, test, or development. These guides cover the general features of AMPS. This site provides additional guides, such as guides for developing applications with AMPS, a guide to the statistics available for monitoring, and so on. ### Resources for Developers For developers, becoming familiar with the Developer Guide for the AMPS Client library that you will be using is also recommended. The [developer page on the 60East web site](https://crankuptheamps.com/develop) contains reference material and links to download client libraries. Full source code (including example applications) is available for all client libraries. For many client libraries, 60East also includes pre-built binaries and makes binary distributions available through popular package management sites. Notice, however, that the pre-built distributions do not contain documentation, source code, or examples. For developers, 60East also provides an [AMPS Command Reference](/docs/amps-command-reference/) that describes the commands to the AMPS server and responses from the AMPS server. Once you are familiar with the features you will use, as described in the user guide, the Developer Guide for your client library of choice and the AMPS Command Reference provide details on how an application communicates with the AMPS server. --- # Message Categories In the AMPS log messages, the error identifier consists of an error category, followed by a hyphen, followed by an error identifier (eg. CC-NNNN). The error categories cover the different modules and features of AMPS, and can be helpful in diagnostics and troubleshooting by providing some context about where a message is being logged from. The error categories found in AMPS are listed in the table below: | AMPS Code | Component | | ------------- | ---------------------------------------------------------------- | | 00 | AMPS Startup | | 01 | General | | 02 | Message Processing | | 03 | Expiration | | 04 | Publish Engine | | 05 | Statistics | | 06 | Metadata | | 07 | Client | | 08 | Regex | | 09 | ID Generator | | 0A | Diff Merge | | 0B | Out of Focus Processing | | 0C | View | | 0D | Message Data Cache | | 0E | Conflated Topic | | 0F | Message Processor Manager | | 11 | Connectivity | | 12 | Trace In - for inbound messages | | 13 | Datasource | | 14 | Subscription Manager | | 15 | SOW | | 16 | Query | | 17 | Trace Out - for outbound messages | | 18 | Parser | | 19 | Administration Console | | 1A | Evaluation Engine | | 1B | SQLite | | 1C | Meta Data Manager | | 1D | Transaction Log Monitor | | 1E | Replication Bootstrap Initialization | | 1F | Client Session | | 20 | Global Heartbeat | | 21 | Transaction Replay | | 22 | TX Completion | | 23 | Bookmark Subscription | | 24 | Thread Monitor | | 25 | Authorization | | 26 | SOW Cache | | 28 | Memory Cache | | 29 | Plug-in Modules (including AMPS features implemented as modules) | | 2A | Message Pipeline | | 2B | Module Manager | | 2C | File Management | | 2D | NUMA Module | | 2F | SOW Update Broadcaster | | 30 | AMPS Internal Utilities | | 31 | AMPS Queues | | 70 | AMPS Networking | | FF | Shutdown | --- # Looking up Errors with ampserr In the `$AMPSDIR/bin` directory is the `ampserr` utility. Running this utility is useful for getting detailed information and messages about specific AMPS errors observed in the log files. The chapter on [Utilities](../utilities) in this guide, contains more information on using the `ampserr` utility and other debugging tools. --- # Using amps-grep to Find Information in Logs The AMPS logs contain a record of events in the instance. Log messages are intended to be read as a part of that sequence of events. While an individual message is useful for showing that a particular event happened, the other messages in the log will show what sequence of events led up to that event and what the results of that event were. At the same time, an active AMPS instance with dozens or hundreds of active clients generates a high volume of events, which can make it difficult to locate and correlate the events that are relevant to a specific problem. To help with this, the AMPS distribution contains the `amps-grep` tool to make it easy to find information in the logs. This section presents some of the most useful techniques for locating information in the logs using `amps-grep`. ## Finding Information for a Specific Client When troubleshooting problems with a specific client (or connection), it's often helpful to be able to see the full sequence of events for that client. For example, if AMPS is returning an `invalid options` message to a client, finding the command that the client is sending to AMPS and the detailed messages that AMPS logs in response, can be very helpful in understanding the details of the error. To find information about a specific client, use the following general pattern: ```bash $ amps-grep client_name log_files > out.txt ``` The `amps-grep` command extracts every event that contains the client name from the log files, and then the Linux shell writes those events to the `out.txt` file. Since the `amps-grep` tool is aware of the structure of multi-line AMPS event records, it captures the full event message, not just the individual lines that contain the name of the client. For example, if the client has a client name of `queue-processor-compute-host-39`, and the logs are stored in files with the suffix `.log`, the command to extract the events with that client name would be along the lines of: ``` $ amps-grep 'queue-processor-compute-host-39' *.log > out.txt ``` This command will write every event message that contains a reference to the client `queue-processor-compute-host-39` to the `out.txt` file. ## Finding Information for a Specific Thread Another situation where `amps-grep` can be useful is when it is important to get information about a specific thread in AMPS. For example, in a situation where AMPS has produced a minidump, it is typically more useful to see the activity on the thread that produced the dump, than it is to see the activity of other threads near the time that the minidump was produced. To track the sequence of events for a thread that produces a minidump, the first step is to find the AMPS identifier for the thread. This is the identifier used in the event log messages, which is different from the operating system assigned identifier for the thread. Then use that identifier to extract messages from the log. For example, given a minidump message like the following: ```bash 2020-04-25T07:27:59.1355850-07:00 [6] critical: 01-0022 AMPS has detected that it may not be running correctly and wrote a minidump to: /tmp/1516e4d1-8bca-b14b-17853753-45dba87b.dmp ``` AMPS has assigned thread ID `6` to this thread. For convenience in searching, AMPS always puts the thread ID in brackets at the beginning of the message (since `6`, by itself, is so common in the logs as to not be useful). Extracting every message recorded by thread ID `6` in the log can be done with a command like the following: ``` $ amps-grep ' [6] ' *.log > out.txt ``` With this command line, `amps-grep` locates all messages for that thread ID, and the Linux shell writes the results to the `out.txt` file. ## Tips for Using amps-grep When using `amps-grep`, there are a few things to be aware of: 1. Unlike regular `grep`, by default `amps-grep` uses *exact* matching rather than regular expression matching. To use a regular expression, provide the `-E` option to `amps-grep`. 2. If you are searching multiple files and piping the output of one `amps-grep` command to another `amps-grep` command, use the `-h` flag to the first `amps-grep` to suppress the file name on matching lines. If you do not provide `-h`, the presence of the file name can interfere with `amps-grep` correctly identifying the start and end of each AMPS log message. 3. If you want to find more than one search term, you can use the `-e` flag to specify multiple search terms, for example: ``` $ amps-grep -e 'error' -e 'warning' *.log ``` Although it is possible to use a regular expression for a search like this, it is not necessary to do so. 4. The `amps-grep` utility provides a usage message with more details on the available options and usage. --- # Message Levels AMPS has nine log levels of escalating severity. When configuring a logging target to capture messages for a specific log level, all log levels at or above that level are sent to the logging target. For example, if a logging target is configured to capture at the “error” level, then all messages at the “error”, “critical”, and “emergency” levels will be captured because “critical” and “emergency” are of a higher level. The following table contains a list of all the log levels within AMPS: | Level | Description | | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | developer |

    Information on the internal state of AMPS, typically used by the AMPS developers.

    Messages logged at this level do not indicate issues, but instead report the detailed state of the AMPS process.

    | | trace |

    All inbound/outbound data.

    Development and test instances are recommended to log at this level.

    Notice that logging at this level records information in and out of the instance, including pretty-printed message data and the full content of commands such as `logon`. Logs recorded at this level should be protected in production instances.

    | | stats | Statistics messages. | | info |

    General information messages.

    Production applications are recommended to log at this level or more verbose.

    | | warning | Problems that AMPS tries to correct that are often harmless. | | error |

    Events in which processing had to be aborted.

    Events at this level often indicate that a client will receive incomplete or incorrect results (for example, a subscription could not be entered because an unknown option was provided).

    | | critical | Events impacting major components of AMPS that if left uncorrected may cause a fatal event or message loss. | | emergency |

    A fatal event.

    AMPS will typically exit after logging an emergency-level event.

    | | none | No logging, even in the case of a critical or fatal event. | :::tip You can use the `IncludeErrors` tag to include specific messages in the logs, regardless of the level of the message. You can use the `ExcludeErrors` tag to exclude specific messages (for example `12-0010`, which records the full logon command), regardless of the level of the message. ::: Each logging target allows the specification of a `Level` attribute that will log all messages at the specified log level or with higher severity. The default `Level` is `none`, which would log nothing. Optionally, each target also allows the selection of specific log levels with the `Levels` attribute. Within `Levels`, a comma separated list of levels will be additionally included. For example, having a log of only `trace` messages may be useful for later playback, but since `trace` is at the lowest level in the severity hierarchy it would normally include all log messages. To only enable `trace` level, specify `trace` in the `Levels` setting as below: ```xml showLineNumbers ... gzip traces.log.gz trace ... ``` Logging only `trace` and `info` messages to a file is demonstrated below: ```xml showLineNumbers ... file traces-info.log trace,info ... ``` Logging `trace` and `info` messages in addition to levels of `error` and above (`error`, `critical` and `emergency`) is demonstrated below: ```xml showLineNumbers file traces-error-info.log error trace,info ``` :::info AMPS accepts the obsolete log level `debug` although no such log level is present in current versions of AMPS. If this level is specified, AMPS will treat this as a synonym for `info`. ::: ## Look Up Messages with `ampserr` The `ampserr` utility, included in the AMPS distribution, provides the ability to look up details on a log message or list messages. Usage and examples, including an example for capturing a full list of errors into a file, are available in the [Utilities](../utilities.md) section at [List/Explain Error Codes](../utilities/ampserr.md). --- # Log Message Format An AMPS log message is composed of the following: * Timestamp (an expanded date, such as `2021-11-23T14:49:38.3442510-08:00`) * AMPS Thread Identifier (the AMPS thread number, such as `1`) * Log Level (the severity of the message, such as `info`) * Error Identifier (the specific identifier for the message, such as `15-0008`) * Log Message (descriptive text providing a short message and additional details) An example of a log line (it will appear on a single line within the log): ```bash 2021-11-23T14:49:38.3442510-08:00 [1] info: 00-0015 AMPS initialization completed (0 seconds). ``` Each log message has a unique identifier of the form `CC-NNNN` where `CC` is the category or component within AMPS which is reporting the message and `NNNN` is the number that uniquely identifies that message within the module. Each logging target allows the direct exclusion and/or inclusion of error messages by identifier. For example, a log file which would include all messages from AMPS startup (category `00`) except for `00-0001` and `00-0004` would use the following configuration: ```xml showLineNumbers stdout 00-0002 00-0001,00-0004,12-1.* ``` In the above `Logging` configuration example, all log messages which are at or above the default log level of `info` will be emitted to the logging target of `stdout`. The configuration explicitly wants to see configuration messages where the error identifier matches `00-0002`. Additionally, the messages which match `00-0001`, `00-0004` will be excluded, along with any messages which match the regular expression of `12-1.*`. --- # Configuring Logging To enable logging, add a `Logging` section to the configuration and specify a `Target` with a `Protocol` value, along with any other relevant options. You can configure multiple targets by including multiple `Target` definitions within the `Logging` tag. For details on logging, including a cross reference to the types of messages and a breakdown of the format AMPS uses in log messages, see the other subsections of [Logging](/docs/amps-user-guide/logging). Described below are the configuration elements that apply to all logging protocols. Expand each item for more details.
    `Protocol` (required) Defines the logging target protocol. Valid values: `stdout`, `stderr`, `file`, `gzip`, `syslog`
    `Level` Defines a lower bound (inclusive) log level for logging. All log messages at the specified level and up are logged. A production server should be configured to log at `info` level or more verbose. Valid values: `developer`, `trace`, `stats`, `info`, `warning`, `error`, `critical`, `emergency`, `none` There is no default for this option.
    `Levels` A comma separated list of specific log levels. Only log messages at the specified levels will be logged. This element can be used with the `Level` element. In that case, AMPS will log all messages at `Level` and above, and in addition, will log errors at the levels specified by `Levels`. Valid values: `developer`, `trace`, `stats`, `info`, `warning`, `error`, `critical`, `emergency`, `none` There is no default for this option.
    `IncludeErrors` Additional errors that should be included when logging. If an error appears in this element, it will be logged regardless of the level of the error. This element accepts a comma-delimited list of error numbers. You can also provide a regular expression that matches a set of errors, such as `12-.*` There is no default for this option.
    `ExcludeErrors` Errors that should be excluded when logging. If an error appears in this element, it will not be logged regardless of the level of the error. If the same error appears in both `IncludeErrors` and `ExcludeErrors`, `ExcludeErrors` takes precedence, and the error will not be logged. This element accepts a comma-delimited list of error numbers. You can also provide a regular expression that matches a set of errors, such as `12-.*` There is no default for this option.
    :::info AMPS logging is always opt-in. That is, no messages are logged to a target by default. Logging must be explicitly requested using the configuration elements above. ::: ## Logging to Files Described below are the configuration items available for logging to a standard or compressed file. Expand each item for more details.
    `FileName`(required) The file to log to. If the `Protocol` is `file`, then `.log` is added to the file name. If the `Protocol` is `gzip`, then `.gz` is added to the file name. Required for `file` and `gzip` protocols. Default: `${PWD}/%Y-%m-%dT%H%M%S.log`
    `RotationThreshold` Log size at which log rotation will occur. See the information on [Byte Units](/shared/units-intervals-and-environment.md#using-units-in-the-configuration) for details on specifying file size.
    #### Standard File Examples The following logging target definition would place a log file with a name constructed from the timestamp and current log rotation number in the `./logs` subdirectory. The first log would have a name similar to `./logs/20121223125959-00000.log` and would store up to 2GB before creating the next log file named `./logs/201212240232-00001.log`. ```xml showLineNumbers ... file info ./logs/%Y%m%d%H%M%S-%n.log 2G ... ``` This example will create a single log named `amps.log` which will be appended to during each logging event. If `amps.log` contains data when AMPS starts, that data will be preserved and new log messages will be appended to the file. ```xml showLineNumbers ... file info amps.log ... ``` #### Compressed File Example The following logging target definition would place a log file with a name constructed from the timestamp and current log rotation number in the `./logs` subdirectory. The first log would have a name similar to `./logs/20121223125959-0.log.gz` and would store up to 2GB of uncompressed log messages before creating the next log file named `./logs/201212240232-1.log.gz`. ```xml showLineNumbers ... gzip info ./logs/%Y%m%d%H%M%S-%n.log.gz 2G ... ``` ## Logging to Syslog Described below are the configuration items available for logging to `syslog`. Expand each item for more details.
    `Ident` The `syslog` identifier for the AMPS instance. Default: AMPS Instance Name
    `Options` A comma separated list of syslog options. If using `syslog`, 60East recommends using `LOG_CONS`, `LOG_NDELAY`, and `LOG_PID`. AMPS uses the standard options to `syslog`, as described in the `syslog` man page.
    `Facility` The `syslog` facility to use.
    Below is an example of a `syslog` logging target that logs all messages at the `critical` severity level or higher, as well as log messages matching `30-0000` to the `syslog`. ```xml showLineNumbers ... syslog critical 30-0000 \amps dma LOG_CONS,LOG_NDELAY,LOG_PID LOG_USER ... ``` Below is an example that shows how to record messages to both `syslog` and `file` logging targets. ```xml showLineNumbers file /var/tmp/amps/logs/%Y%m%d%H%M%S-%n.log 2G trace critical syslog critical amps_dma LOG_CONS,LOG_NDELAY,LOG_PID LOG_USER file /var/tmp/amps/logs/initMessage 00-0015 ``` ## Logging to the Console The console logging target instructs AMPS to log certain messages to the console. Both the standard output and standard error streams are supported. Use a `Protocol` setting of `stdout` to select standard output, or `stderr` for standard error. Below is an example of a console logger that logs all messages at the `info` or `warning` level to standard out and all messages at the `error` level or higher to standard error (which includes `error`, `critical` and `emergency` levels). ```xml showLineNumbers ... stdout info,warning stderr error ... ``` ## Example: Development Instance Logging This logging configuration may be useful for development instances. This configuration is intended to meet the following considerations: * Warning, error, and critical messages are logged to standard output to make it easy for a developer to see if a command to AMPS produces an error. * Message traffic in and out of AMPS is logged to the `trace.log` file for debugging purposes. This file is rotated every 250MB. When the file hits the 250MB limit, it will be cleared and overwritten with new entries. * Static information about the instance -- including the configuration, detected hardware configuration, and so forth -- is logged to the `instance-info.log`. Logging this information to a separate file means it will still be available when the trace log is replaced. ```xml showLineNumbers ... file trace ./logs/trace.log 250MB stdout warning 00-0015 file ./logs/instance-info.log 00-0001,00-0002,00-0004,00-0015, 00-0030,00-0033,00-0032,00-0054, 01-0019,2D-0005,2D-0006,2D-0008,2D-0011 ... ``` --- # Logging to Files To log to a file, declare a logging target with a protocol value of `file`. Beyond the standard `Level`, `Levels`, `IncludeErrors`, and `ExcludeErrors` settings available on every logging target, file targets also permit the selection of a `FileName` mask and `RotationThreshold`. ## Compressed Files AMPS supports logging to compressed files as well. This is useful when trying to maintain a smaller logging footprint. Compressed file logging targets are the same as regular file targets except for the following: * The `Protocol` value is `gzip` instead of `file`. * The log file is written with gzip compression. * The `RotationThreshold` is metered off of the _uncompressed_ log messages. * Makes a trade off between a small increase in CPU utilization for a potentially large savings in logging footprint. ## Selecting a Filename The `FileName` attribute is a mask which is used to construct a directory and file name location for the log file. AMPS will resolve the file name mask using the symbols in the table below. For example, if a file name is masked as: ``` %Y-%m-%dT%H:%M:%S.log ``` AMPS would create a log file in the current working directory with a timestamp of the form: `2012-02-23T12:59:59.log`. If a `RotationThreshold` is specified in the configuration of the same log file, the next log file created will be named based on the current system time, not on the time that the previous log file was generated. Using the previous log file as an example, if the first rotation was to occur 10 minutes after the creation of the log file, then that file would be named `2012-02-23T13:09:59.log`. Log files which need a monotonically increasing counter when log rotation is enabled can use the `%n` mask to provide this functionality. If a file is masked as: ``` localhost-amps-%n.log ``` Then the first instance of that file would be created in the current working directory with a name of `localhost-amps-00000.log`. After the first log rotation, a log file would be created in the same directory named `localhost-amps-00001.log`. Log file rotation is discussed in greater detail in the [Log File Rotation](logging-to-a-file.md#log-file-rotation) section. | Mask | Definition | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `%Y` | Year | | `%m` | Month | | `%d` | Day | | `%H` | Hour | | `%M` | Minute | | `%S` | Second | | `%n` | Iterator which starts at `00000` when AMPS is first started and increments each time a `RotationThreshold` size is reached on the log file. | ## Log File Rotation Log files can be “rotated” by specifying a valid threshold in the `RotationThreshold` attribute. Values for this attribute have units of bytes unless another unit is specified as a suffix to the number. The valid unit suffixes are: | Unit Suffix | Base Unit | Examples | | --------------- | ------------------ | ---------------------------- | | no suffix | bytes | “1000000” is 1 million bytes | | k or K | thousands of bytes | “50k” is 50 thousand bytes | | m or M | millions of bytes | “10M” is 10 million bytes | | g or G | billions of bytes | “2G” is 2 billion bytes | | t or T | trillions of bytes | “0.5T” is 500 billion bytes | :::tip When using log rotation, if the next filename is the same as an existing file, the file will be truncated before logging continues. For example, if `amps.log` is used as the `FileName` mask and a `RotationThreshold` is specified, then the second rotation of the file will overwrite the first rotation. If it is desirable to keep all logging history, then it is recommended that either a timestamp or the `%n` rotation count be used within the `FileName` mask when enabling log rotation. ::: --- # Logging to a Compressed File AMPS supports logging to compressed files as well. This is useful when trying to maintain a smaller logging footprint. Compressed file logging targets are the same as regular file targets except for the following: * The `Protocol` value is `gzip` instead of `file`. * The log file is written with gzip compression. * The `RotationThreshold` is metered off of the _uncompressed_ log messages. * Makes a trade off between a small increase in CPU utilization for a potentially large savings in logging footprint. ### Example The following logging target definition would place a log file with a name constructed from the timestamp and current log rotation number in the `./logs` subdirectory. The first log would have a name similar to `./logs/20121223125959-0.log.gz` and would store up to 2GB of uncompressed log messages before creating the next log file named `./logs/201212240232-1.log.gz`. ```xml ... gzip info ./logs/%Y%m%d%H%M%S-%n.log.gz 2G ... ``` --- # Logging to the Console The console logging target instructs AMPS to log certain messages to the console. Both the standard output and standard error streams are supported. To select standard out use a `Protocol` setting of `stdout`. Likewise, for standard error use a `Protocol` of `stderr`. ### Example Below is an example of a console logger that logs all messages at the `info` or `warning` level to standard out and all messages at the `error` level or higher to standard error (which includes `error`, `critical` and `emergency` levels). ```xml ... stdout info,warning stderr error ... ``` --- # Logging to Syslog AMPS can also log messages to the host’s syslog mechanism. To use the syslog logging target, use a `Protocol` of `syslog` in the logging target definition. The host’s syslog mechanism allows a logger to specify an identifier on the message. This identifier is set through the `Ident` property and defaults to the AMPS instance name (see [Instance-Level Configuration](/docs/amps-user-guide/configuring-amps/instance-configuration) for details on configuring the AMPS instance name). The syslog logging target can be further configured by setting the `Options` parameter to a comma-delimited list of syslog flags. The recognized syslog flags are: | Level | Description | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `LOG_CONS` | Write directly to system console if there is an error while sending to system logger. | | `LOG_NDELAY` | Open the connection immediately (normally, the connection is opened when the first message is logged). | | `LOG_NOWAIT` | No effect on Linux platforms. | | `LOG_ODELAY` | The converse of LOG\_NDELAY; opening of the connection is delayed until `syslog()` is called. (This is the default, and need not be specified.) | | `LOG_PERROR` | Print to standard error as well. | | `LOG_PID` | Include PID with each message. | :::info AMPS already includes the process identifier (PID) with every message it logs, however, it is a good practice to set the `LOG_PID` flag so that downstream syslog analysis tools will find the PID where they expect it. ::: The `Facility` parameter can be used to set the syslog “facility”. Valid options are: `LOG_USER` (the default), `LOG_LOCAL0`, `LOG_LOCAL1`, `LOG_LOCAL2`, `LOG_LOCAL3`, `LOG_LOCAL4`, `LOG_LOCAL5`, `LOG_LOCAL6`, or `LOG_LOCAL7`. Finally, AMPS and the syslog do not have a perfect mapping between their respective log severity levels. AMPS uses the following table to convert the AMPS log level into one appropriate for the syslog: | AMPS Severity | Syslog Severity | | ----------------- | ------------------- | | none | LOG\_DEBUG | | developer | LOG\_DEBUG | | trace | LOG\_DEBUG | | stats | LOG\_INFO | | info | LOG\_INFO | | warning | LOG\_WARNING | | error | LOG\_ERR | | critical | LOG\_CRIT | | emergency | LOG\_EMERG | --- # Logging AMPS supports different types of log formats and logging to multiple targets including, files, syslog and the console. Every error message within AMPS is uniquely identified and can be filtered out or explicitly included in the logger output. This section of the *AMPS User Guide* describes the AMPS logger configuration, the unique settings for each logging target, and includes configuration examples to help you get started, which can be found in [Configuring Logging](logging/logging-configuration). --- # BFlat Messages The BFlat message format combines the simplicity and efficiency of simple, schemaless data formats such as FIX and NVFIX with the ability to manage binary data and preserve the full precision of numeric values. BFlat is especially useful for applications that deal with binary data or precise numeric values while demanding high levels of throughput. A BFlat message is composed of any number of tag/value pairs, similar to FIX and NVFIX messages. Tags and values can contain any value and can be of any length. Unlike formats such as FIX, there are no reserved characters. In practical terms, the name of a tag must be a valid XPath identifier to filter the message in AMPS. However, this is a limitation of XPath and not of the BFlat message format. The BFlat message type supports all AMPS features and there are no special considerations when using the BFlat message type. Open-source libraries for producing and parsing BFlat messages are available from the [BFlat project](http://bflat.io) site. # BFlat Data Types BFlat messages are strongly typed. For numeric values, BFlat can preserve the precise value of the following numeric types: |Type |Description| |--------|-----------------------------------------------------------------------------| |int8 |8-bit integer| |int16 |16-bit integer| |int32 |32-bit integer| |int64 |64-bit integer| |double |64-bit IEEE 754 floating point number| |datetime|UTC datetime containing milliseconds since Unix epoch (64-bit representation)| |leb128 |Signed LEB128 integer (variable length)| BFlat also includes the following non-numeric types: |Type |Description| |------|---------------------------------------------------------------------------------------------------------------------------| |null |Empty field| |string|

    String of bytes

    BFlat does not specify the encoding of a string; this is up to the application to determine.

    | |binary|Untyped sequence of bytes| BFlat includes the ability to represent arrays of values. ## Representing Data in BFlat By convention, BFlat serializers should choose the most compact representation of a given value. For example, if an integer value can fit into 8 bits, it should be serialized as a value of type `int8`. An application that parses BFlat should assume that a serializer may use different types for values in the same field, to optimize for a compact representation, and should not rely on a value being of a specific type. For example, a `TotalValue` field that can have any integer value could be serialized with a type of `int8` for values that fit in 8 bits, but could be serialized with a integer type that includes more bits if the value is larger, and could be serialized with the `leb128` type for integer values that cannot be represented in 64 bits. When the AMPS server must serialize a BFlat message (for example, when projecting a view or as a result of enrichment), AMPS tries to optimize for the most compact message size, which could change the representation if the serializer did not choose the most compact type for a value. --- # Composite Messages Sometimes, applications only need to filter on a small subset of the fields in a message. Sometimes applications need to send and receive messages that cannot be meaningfully parsed by AMPS, such as images or audio files. For these cases, AMPS provides a composite message type that lets you create a new message type by combining existing message types. For example, you might create a message type that includes three parts: the metadata for an image as a `json` document, a small JPG thumbnail as a `binary` message part, and a full size PNG image as another `binary` message part. Composite messages can also be useful when the message itself is large or resource-intensive to parse. In this case, you can create a message type that includes the information needed to filter messages in a JSON or NVFIX part, and include the full message in the unparsed payload of the composite message, as described below. AMPS provides two different types of composite messages. Messages created using the `composite-local` module preserve information about the individual parts for filtering, aggregation, and projection. Messages created using the `composite-global` module treat the individual parts as elements of a single document. Composite message types have the following restrictions: * Delta subscribe and delta publish are not supported for message types that use `composite-global`. * Views, joins, and aggregation cannot project message types that use `composite-global`. (However, composite message types that use `composite-global` _can_ be an `UnderlyingTopic` or one of the topics in a `Join`.) * Composite message types do not support features that automatically construct messages, such as subscriptions to the `AMPS/.*` topics and stats acks, regardless of the module the type uses. ### Unparsed Payload Section All composite message types, regardless of how they are defined, provide an _unparsed payload_ section. The unparsed payload section does not need to be declared in the `MessageType` declaration. As the name suggests, AMPS does not parse or interpret this section, so the unparsed payload can contain any content of any type. In an AMPS client, the unparsed payload section is represented as bytes that appear at the end of the message, after the last defined part. The unparsed payload is included to simplify the common technique where a message type contains a header that is used for filtering followed by an unparsed binary in cases where the only use of the message is subscribe / publish. If your composite message type contains a single binary part and your use of the message is simple subscribe/publish, consider using the unparsed payload section in your application rather than declaring a binary message part. Notice, however, that since AMPS does not parse or interpret this section, it may not appear in serialized representations of the message (for example, log messages, Galvanometer display, and so on). Any use of the message that considers the whole message (such as[ delta publish](../delta-publish) or [delta subscribe](../delta-subscribe)) will not consider the unparsed payload. ## Content Filtering with Composite Message Types Composite message types support filtering on the contents of the composite message. There are some simple conventions to remember when constructing expressions to filter on. For more details about content filtering, see [Filtering Subscriptions by Content](../pub-sub/content). These conventions are consistent anywhere that AMPS needs to find a value within the composite message type. That includes content filters for client subscriptions, identifying SOW keys, creating views and aggregates, creating conflated topics, and so on. ### composite-global When using the `composite-global` message type, AMPS combines all parts of the message into a unified set of XPath identifiers. AMPS creates the set of identifiers for each part of the message. If different parts of the message contain the same identifier, AMPS treats that identifier as though the identifier contained an array of values: AMPS creates an array that contains all of the values in the different parts of the message. Message types that do not support content filtering do not provide XPath identifiers. For example, consider the message below for a `composite-global` message type that includes two `json` parts and a `binary` part: ```json {"id":1,"data":"sample","message":"part one message"} {"message":"another part","customer":"Awesome Amalgamated, Ltd."} 0xDEEA0934DF23A37780934... ``` AMPS constructs the following set of XPath identifiers and values: | Identifier | Value | | ----------- | -------------------------------------- | | `/id` | `1` | | `/data` | `"sample"` | | `/message` | `["part one message", "another part"]` | | `/customer` | `"Awesome Amalgamated, Ltd."` | In short, when using `composite-global`, AMPS combines the parsable parts of the message into a single global set of XPath values, and ignores any part of the message that cannot be parsed. ### composite-local When using the `composite-local` message type, AMPS creates a distinct set of XPath identifiers for each part of the message. AMPS adds an XPath step with the position of the message part at the beginning of the identifier. Message types that do not support content filtering do not provide XPath identifiers, and AMPS skips over them. For example, consider the message below for a `composite-local` message type that includes two `json` parts and a `binary` part: ```json {"id":1,"data":"sample","message":"part one message"} {"message":"another part","customer":"Awesome Amalgamated, Ltd."} 0xDEEA0934DF23A37780934... ``` AMPS constructs the following set of XPath identifiers and values: | Identifier | Value | | ------------- | ----------------------------- | | `/0/id` | `1` | | `/0/data` | `"sample"` | | `/0/message` | `"part one message"` | | `/1/message` | `"another part"` | | `/1/customer` | `"Awesome Amalgamated, Ltd."` | In short, when using `composite-local`, AMPS creates XPath identifiers for each part of the message, using the position of the message part within the composite as the first part of the identifier. AMPS skips over any part of the message that cannot be parsed, and simply produces no values for that part of the message. ## Choosing a Composite Type To choose which composite type best fits your application, consider the following factors: * If you need to use delta messaging with this message type, use `composite-local`. * If there may be redundant field names in the parts of the message, and it is important to be able to filter based on which part contains the field, use `composite-local`. * If you need to be able to create views of this type, use `composite-local`. Otherwise, `composite-global` may be easier and more straightforward for client filtering, since clients do not need to know the detailed structure of the message type to be able to filter on the message. Notice, though, that subscribers will need to know how the message type is defined, since that will determine whether subscribers need to include part specifiers (`/0`, `/1`, and so on) to their filters. --- # Configuring Message Types The `MessageTypes` tag defines the message types supported by the AMPS instance. A single AMPS instance can support multiple message types. As mentioned in the [Default Message Types](default-message-types) section, `MessageType` definitions for `fix`, `nvfix`, `xml`, `json`, `bflat`, `msgpack`, `bson`, and `binary` are automatically loaded by AMPS. You only need to define a new `MessageType` for these types if the settings for the message type need to be changed (for example, to create a custom FIX-based type that changes the `FieldSeparator` of the message). The `MessageTypes` tag can contain multiple `MessageType` definitions. To add more than one message type to the message types that are loaded by default, include multiple `MessageType` tags. AMPS loads the capability to use Google protocol buffer (`protobuf`) messages by default. To use protocol buffer messages, you configure one or more message types that use the `protobuf` module and load the `.proto` files that define the format of the messages you will be processing with AMPS. AMPS also supports the ability to create a composite message type by combining a number of existing message types. Composite message types are defined using the `MessageType` configuration element. Described below are the configuration items available for a `MessageType`. Expand each item for more details.
    `Name` (required) This element defines the name for the message type. The name is used to specify `MessageType` in other sections such as `Transport`, `TransactionLog` and the elements of the `SOW` section. By default, AMPS loads message types for `fix`, `nvfix`, `json`, `bflat`, `msgpack`, `bson`, `xml` and `binary`. It is typically not necessary to configure these types for use. Other message types, such as Google protocol buffers and `C` structs, are available by default, but require configuration to be used.
    `Module` This element specifies the name of the module that will be loaded for this message type. By default, AMPS loads the modules that implement the following message types: `fix`, `nvfix`, `json`, `bflat`, `msgpack`, `bson`, `xml`, `protobuf`, and `binary`. AMPS supports creating composite message types out of existing message types using the `composite-global` and `composite-local` modules, which are loaded by default.
    `AMPSVersionCompliance` Sets the version compatibility for FIX messages that AMPS sends to the `/AMPS/SOWStats` topic. AMPS accepts three values for this option:
    • `2` creates messages that use the FIX field tags used by AMPS 2.X.
    • `4` creates messages that use the default FIX field tags (the values used in AMPS 4.X). With this version, FIX messages use different field numbering for `/AMPS/SOWStats` and `/AMPS/ClientStatus` messages.
    • `5` creates messages that use a unified set of FIX tags. When this option is set to `5`, AMPS uses consistent field numbering between `/AMPS/SOWStats` and `/AMPS/ClientStatus` messages (which is only available on versions 5.X and later).
    For message types other than FIX, there is no difference between `4` and `5`. These message types were not supported in AMPS 2.X. AMPS provides reasonable values for these message types when this value is set to `2`, but there is no backward compatibility to enforce. For most cases, you can leave this option set to the default. If you are using a system that requires consistent FIX tags across messages, set this parameter to `5`. If you are using an existing system that expects AMPS 2.X tags, set this parameter to `2`. Default: `4` (for compatibility with the largest number of existing installations)
    `Options` Options to pass to a custom message type module. AMPS does not specify the format or type of the items within an `Options` element. AMPS simply parses the XML and then sends the XML to the module. If you are configuring a custom message type, see the documentation for that message type module for details.
    ## Message Type Specific Options Below are the configuration items that apply to specific message types. ### FIX/NVFIX Options Described below are the options that apply to `fix` and `nvfix` message types. Expand each item for more details.
    `FieldSeparator` Applies to `fix` and `nvfix` message types. Sequence of characters used to separate field items in a FIX message. Note: This field is the ASCII value of the char sequence.
    `HeaderSeparator` Applies to `fix` and `nvfix` message types. Sequence of characters used to separate the header from the body in a FIX message. Note: This field is the ASCII value of the char sequence.
    `MessageSeparator` Applies to `fix` and `nvfix` message types. Sequence of characters used to separate message items in the body in a FIX message. Note: This field is the ASCII value of the char sequence.
    The example below defines a FIX-based message type with custom separators. ```xml showLineNumbers fix-custom fix 1 2 5 ``` ### JSON Option Described below is the option that applies to the `json` message type. Expand the item for more details.
    `EarlyTerminationOptimization` Applies to the `json` message type. By default, AMPS includes an optimization to allow the server to only partially parse JSON messages. This may result in unexpected behavior for some messages. For example, given a message such as `{ "code" : 1, "data" : "some data", "code" : 2 }`, AMPS will report the value of `code` as `1` when this optimization is active. To ensure consistent results, in this mode AMPS always reports the first value for a field even when AMPS fully parses the message. When set to `false`, the optimization is disabled. AMPS will fully parse all JSON messages and report the last value for a field. For the message above, AMPS would report the value of `code` as `2`. Default: `true`
    The example below disables the optimization: ```xml showLineNumbers json-custom json false ``` ### Composite-Local and Composite-Global Option The `MessageType` entries for the composite message can be any AMPS message type, including both the built-in types and any previously defined message type. Once the new composite message type is created, you can use the new type in the configuration file. Described below is the option that applies to `composite-local` or `composite-global` message types. Expand the item for more details.
    `MessageType` (required) Applies to message types that use the `composite-local` or `composite-global` modules. For composite message types, the `MessageType` definition must contain one or more message type declarations that specify the types that the composite message type contains.
    For example, the `MessageType` element below declares a new composite message type named `images`. The new type contains a `json` document at the beginning of the message, followed by two uninterpreted binary message parts. AMPS will combine the XPath identifiers for all message parts into a single set of identifiers. Notice that, because only one part of the message type is parsable, using `composite-global` simplifies the identifiers for the message. ```xml showLineNumbers images composite-global json binary binary ``` The example below defines a composite message type that combines a json message and a custom-payload message: ```xml showLineNumbers custom-composite composite-local json custom-payload ``` ### Google Protocol Buffer Options To use a protobuf message, you must first edit the configuration file to include a new `MessageType`. Then, specify the path to the protobuf file and the name of the protobuf file itself inside the `MessageType`. Each message type references a `ProtoFile`, and specifies a single top-level type from the file. The `ProtoFile` may include other files through the standard protocol buffer include mechanism. Likewise, the top-level type may be any valid protocol buffer definition, including definitions that contain other types. Once the protocol buffer `MessageType` is created as described above, you must either create a `Transport` that specifies that message type exactly, or you must create a `Transport` that can accept any known message type and ensure that the client specifies the new message type (in the example case, `my-protobuf-message`) in the connect string. Described below are the options that apply to `protobuf` message types. Expand each item for more details.
    `Type` (required) Applies to message types that use the `protobuf` module. The name of the type within the `.proto` file to use for this message type. The name must be package-qualified (for example, `my.package.Message` would load the type `Message` within the package `my.package`). Obsolete usage - A previous meaning of this element was made obsolete in AMPS 4.0 and later versions. That usage was replaced by the `Module` directive.
    `ProtoPath` (required) Applies to message types that use the `protobuf` module. The path in which to search for `.proto` files. The content of this element has the following syntax:
    `alias ; full-path`
    The `alias` provides a short identifier to use when searching for `.proto` files. The `full-path` is the path that is substituted for that identifier. A configuration may omit the alias, and simply provide the path. For example, to use the path `/mnt/repository/protodefs` when no alias is provided, you would declare a `ProtoPath` of:
    `/mnt/repository/protodefs`
    or 
    `;/mnt/repository/protodefs`
    The following `ProtoPath` declaration sets `proto-archive` as an alias for `/mnt/shared/protofiles`:
    `proto-archive;/mnt/shared/protofiles`
    AMPS uses the aliases provided in this configuration item when processing `import` statements within the loaded `.proto` files, with the empty alias used for import statements that do not specify a path alias. For example, given the definitions above, this import statement:
    `import "proto-archive/AType.proto";`
    will load the file at `/mnt/shared/protofiles/AType.proto`, while the import statement:
    `import "MyFavoriteType.proto";`
    will load the file at `/mnt/repository/protodefs/MyFavoriteType.proto`. If no `ProtoPath` declaration sets an empty alias, all imports processed by AMPS must use one of the aliases set for the instance, or AMPS will fail to find the specified file. Unless your existing definitions use an aliasing scheme, it is most convenient to set the empty alias. You may specify any number of `ProtoPath` declarations.
    `ProtoFile` (required) Applies to message types that use the `protobuf` module. The name of the `.proto` file to use for this message type. To use an alias, prefix the name of the file with the alias. For example, if your `ProtoPath` declarations have created the `proto-archive` alias for the directory where your `.proto` files are stored, you could use the following to access the `my-messages.proto` file in that directory.
    `proto-archive/my-messages.proto`
    Below is a sample configuration of a protobuf message type: ```xml showLineNumbers my-protobuf-messages protobuf proto-archive;/mnt/shared/protofiles proto-archive/person.proto MyNamespace.Message ``` ### Struct Message Type Option The `MessageType` definition for the `struct` message type contains a description of each field of the `struct` and the AMPS identifier that will be used for each field.
    `Field` (required) Applies to message types that use the `struct` module. Defines the binary format of the field and the AMPS identifier to use for the field. The message type interprets the fields in the order in which they are defined. For example, to specify that the `struct` contains a 4-byte integer in native byte format (little-endian on x64 Linux) and that AMPS should use `/id` as the identifier for the field, you would use the declaration of `/id = i`.
    See the [Struct Message Types](struct-message-types) section for the full table of the data type options available. ### Custom Message Types The example below defines a custom message type. The `Module` in this case - `type-module`, must be the `Name` of a `Module` specified in the `Modules` section of the configuration file. ```xml showLineNumbers custom-payload type-module ``` --- # Default Message Types AMPS automatically loads modules for the following message types: | Message Type | Description | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `bson` |

    Binary JSON (BSON) messages.

    See [http://www.bsonspec.org](http://www.bsonspec.org) for information on this format.

    | | `bflat` |

    BFlat, a schemaless message format based on key-value pairs that includes support for binary representations of numeric data.

    See [http://bflat.io](http://bflat.io) for information on this format. The section on [BFlat Messages](bflat-messages) describes how AMPS interprets BFlat messages.

    | | `fix` |

    FIX messages using numeric tags. FIX is a standard format widely used in the financial industry.

    See [https://www.fixtrading.org/what-is-fix/](https://www.fixtrading.org/what-is-fix/) for more information on this format.

    | | `json` |

    JSON (JavaScript Object Notation) messages.

    See [http://www.json.org](http://www.json.org) for information on this format.

    | | `msgpack` |

    MessagePack messages.

    MessagePack is a schemaless serialization format designed to efficiently encode data.

    See [http://msgpack.org/index.html](http://msgpack.org/index.html) for more information on MessagePack. The section on [MessagePack Messages](messagepack-messages) describes how AMPS interprets messagepack messages.

    | | `nvfix` |

    NVFIX (name/value FIX) messages.

    NVFIX uses the same basic format as FIX, but allows tags to contain any byte that is not `=` or the configured field separator character (by default, the ASCII `SOH` character).

    By contrast, FIX requires that tags are numeric.

    | | `xml` |

    XML messages (of any schema).

    AMPS preserves element names and attributes when parsing XML data. For performance, the AMPS `xml` message parser limits nested element depth to 64 levels of nesting.

    | | `binary` |

    Uninterpreted binary payload.

    Since this module does not attempt to parse the payload, it does not support content filtering, views and aggregates.

    Likewise, because there is no set format for the payload, this message type cannot support features that construct messages (such as delta messaging, `/AMPS/.*` topic subscriptions and `stats` acks).

    | | `protobuf` |

    Google protocol buffer messages.

    To use this message type, you must configure a `MessageType` with the format of the messages (the `.proto` files). The section on [Protobuf Message Types](protobuf-message-types) describes this configuration.

    | | `struct` |

    Binary data in the format of a `C` language `struct`.

    To use this message type, you must configure a `MessageType` that specifies the format of the message. The section on [Struct Message Types](struct-message-types) describes this configuration.

    | With these message types, AMPS automatically loads the module that provides the message type. AMPS declares message types for all of the above message types except for `protobuf` and `struct`. For those types, AMPS requires additional configuration information to correctly interpret messages. For efficiency, AMPS only parses the content of a message if required, and only to the extent required. For example, if AMPS only needs to find the `id` tag in an NVFIX message, AMPS will not fully parse the message, but will stop parsing the message after finding the `id` tag. This provides significant performance improvements, and also means that AMPS does not verify the format or validity of messages unless it needs to parse the messages. When AMPS parses a message, it may only partially parse a message, and may not detect corruption or invalid format in a message if that corruption occurs after the point at which AMPS has all of the required information from the message. The FIX and NVFIX message types support configuration of the field and message delimiters. AMPS also allows you to create new message types by assembling existing message types into a composite message. Composite message types are described in [Composite Messages](composite-messages), and require additional configuration: | Message Type Name | Description | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `composite-global` |

    Composite message type that combines message parts for content filtering. This message type combines one or more existing message types into a message.

    This type of composite message does not distinguish between sections of the message when parsing the message.

    This type is described in more detail in the section on `composite-global` in [Composite Messages](composite-messages#composite-global).

    | | `composite-local` |

    Composite message type, filterable by individual parts. This message type combines one or more existing message types into a message.

    This type of composite message preserves information about the sections of the message when parsing the message.

    This type is described in more detail in the section on `composite-local` in [Composite Messages](composite-messages#composite-local).

    | --- # MessagePack Messages AMPS fully supports MessagePack messages, with the following implementation decisions to represent MessagePack messages in the AMPS type system. See [AMPS Data Types](../amps-expressions/amps-data-types) for information on the AMPS data types. Notice, in particular, that the AMPS expression language supports automatic type conversion, so while this table shows the default AMPS representation for a given MessagePack type, AMPS will convert a value as needed once the message has been parsed. | MessagePack Type | AMPS Representation | | ------------------ | -------------------- | | nil | NULL | | bool | Boolean | | int (all widths) | Integer | | float (all widths) | Float | | str (all widths) | String | | bin (all widths) | String | | array (all widths) | Array of AMPS values | | map (all widths) | Nested AMPS values | | ext (all widths) | String | Notice that AMPS does not attempt to interpret extension types, and instead represents them as arrays of bytes (a String in the AMPS type system). --- # Protobuf Message Types Protocol buffers, or protobufs for short, is an efficient, automated mechanism for serializing structured data. AMPS supports Google protobuf messages (version 2 and version 3) as a message format. Since Google protocol buffers use a fixed format for messages, to use protobuf, you must configure AMPS with the definition of the messages AMPS will process. This involves defining a `MessageType`. You must define a `MessageType` for AMPS to be able to parse protobuf messages. 60East recommends that the `.proto` files used with AMPS explicitly declare the protocol buffer syntax version used. If there is no explicit declaration, AMPS assumes the file uses protocol buffer 2 syntax. The AMPS engine is message-type agnostic. Except for the limitations described in this section, there is no difference to the AMPS engine between message types that use protocol buffers and other message types such as JSON or XML or FIX. ## Filtering with Protobuf Messages To filter protobuf messages, there are a couple of conventions you must remember. AMPS XPath identifiers begin at the outermost message, so you can simply use member names for that message. If you have nested messages, you use the name of the nested message and the member name when creating an XPath identifier. For example, suppose you have the following definition in a `.proto` file: ```protobuf showLineNumbers message person { required string name = 1; required int32 personID = 2; } ``` To access the `personID` data member, you simply use the name of the data member as the XPath identifier. An example filter that verifies that a `personID` is greater than 1000 would be: ```protobuf /personID > 1000 ``` If you have nested messages, you simply provide the path to the nested message you want to access. Let's assume that the `person` message from the above example was nested inside another message with the name of `record`. The example filter below shows how to access the nested `person` message, and then filter to the `personID`: ```protobuf /person/personID > 1000 ``` In this case, the first part of the identifier (`/person`) specifies the sub-message. The second part of the identifier (`/personID`) specifies the field within that sub-message. Notice that, as always, there is no need to specify the name of the message for the outermost message. ## Working with Multiple Protocol Buffer Types Some applications require messages of different types: for example, an inventory management system may work with customer records, inventory records, and shipping order records. When using protocol buffers, each of these messages would use a different `.proto` file, and therefore would be a different message type. Unlike a self-describing format such as JSON or XML, the serialized form of a protocol buffer message type does not automatically contain any information about the type of message or the fields that the message contains. Therefore, each protocol buffer message type is best considered as a completely distinct type. For example, the parser created for an order record and the parser created for a customer record are different. Unlike self-describing formats, it is not possible to use a single parser for these types, or for a parser to correctly handle a previously-unknown message structure. There are two approaches to working with multiple protocol buffer types in an AMPS application: 1. Keep the message types distinct. Each message type requires a separate connection to AMPS. The advantage of this approach is that the `.proto` files can be maintained and updated separately. Each connection has a distinct type and only needs to handle messages of that type. The disadvantage of this approach is that the application must make a connection to AMPS for each type of message received. 2. Create a "container" type that can _optionally_ contain any of the needed message types. The advantage of this approach is that this requires only a single connection to AMPS. Since there is a single "container" type, a topic can hold this "container" type and have heterogeneous actual contents. The disadvantage to this approach is that it requires a consumer to understand the "container" type and changes to the contained types may need to be carefully managed across the consumers that use the container. A "container" type is typically a `oneof` of the contained types. For example, you might define a container as follows: ```protobuf showLineNumbers message Container { oneof { Order order_type = 1; Payment payment_type = 2; } } message Order { required string customer_id = 1; ... } message Payment { required string customer_id = 1; ... } ``` In this case, the container type will include **either** an `Order` or a `Payment`. ## Union Types When using a protocol buffer message type that contains a union, you can navigate the union using the names defined in the top-level element. For example, given the union defined below: ```protobuf showLineNumbers message MyUnion { optional Order order_type = 1; optional Payment payment_type = 2; } message Order { required string customer_id = 1; ... } message Payment { required string customer_id = 1; ... } ``` Providing a filter of `/order_type IS NOT NULL` will return all of the `MyUnion` messages that contain an `Order`, while providing a filter of `/payment_type/customer_id = '42'` will return only the `MyUnion` messages that contain a `Payment` message with a `customer_id` of `42`. ## Protobuf Message Type Limitations Since the `protobuf` message type requires a specific, fixed definition for messages, AMPS does not support operations that construct messages that may contain arbitrary values. In particular, protobuf does not support: * Creating a View with a `protobuf` type as the `MessageType`. AMPS allows you to aggregate protobuf messages and project the results as another type, but the destination `MessageType` for a View cannot be a `protobuf` message type. * Creating an aggregated subscription for a topic that contains messages of a `protobuf` message type. * Subscriptions to AMPS internal topics. Protobuf message types do not support creating messages for AMPS internal topics, such as `/AMPS/ClientStatus`. * Enriching or preprocessing `protobuf` message types. AMPS does not support enrichment or preprocessing of `protobuf` messages. Protocol buffer version 3 messages, prior to version 3.15, provided fixed default values for omitted fields. This meant that there was no reliable way for AMPS to determine if a missing field has been intentionally left out of the message, or simply contains the fixed default value. As of version 3.15, Protocol Buffers implemented explicit field presence tracking utilizing the `optional` keyword on a field. The result is some additional caveats for protocol buffer version 3 message types: * Protocol buffer version 3 message types generated with version < 3.15, do not support delta publish or delta subscribe. * Protocol buffer version 3 message types generated with version 3.15 and greater, *do* support delta publish and delta subscribe, with the use of the `optional` keyword. Protocol buffer version 2 message types can require that specific fields are provided in a message (that is, fields can be marked required). The result is an additional limitation for protocol buffer version 2 message types: * Protocol buffer version 2 message types do not support providing a subset of fields in a message by specifying a select list. There are no other limitations in working with protocol buffer message types. ## Working with Optional Default Values Google protocol buffers provide the ability for a message to have fields that are both _optional_, so they need not be provided in the serialized message, and _defaulted_, so that there is a specific value interpreted when there is no value provided. When no value is provided in the serialized message for an optional default value, AMPS interprets the message differently depending on the context: * For most uses, AMPS interprets the message as though the value is _present and set to the default value_. This means that you can filter on optional default values, use them as SOW keys, and aggregate optional default values regardless of whether a value is present in the serialized message. * For _delta messaging_ with protocol buffer version 2, AMPS treats an optional default value as though there is _no value present_. AMPS does not provide the default value. This means that a delta update must provide the default value _explicitly_ in the serialized message to set the field to the default value. This also means that, if the value present in the message is not the default value, but was not changed on the current update, AMPS will not emit that value in messages to delta subscribers. * Protocol buffer version 3 fields intended to be omitted for delta publish or delta subscribe must be explicitly labeled `optional`, including nested message fields. * Client applications *must* use the generated `getter`, `setter`, `clear`, and `hazzer` methods as described in the official Protocol Buffer 3 documentation to utilize the explicit presence tracking mechanisms. * Unlabeled Protocol Buffer 3 scalar fields are treated as required for AMPS delta processing, so they appear in delta messages even when unchanged. * Protocol buffer version 3 does not support presence tracking for repeated fields and maps. These are considered to be always present for the purpose of delta publish or delta subscribe. --- # Struct Message Types AMPS includes a message type that allows the server to parse and interpret binary data in a fixed format. When configuring the message type, you must include a definition of the format. This format is designed to allow AMPS to process messages serialized from raw memory, such as would be specified in a C-language `struct`. ## Configuring a Struct Message Type To configure a `struct` message type, you must define each field that AMPS will use. This does not necessarily have to match the original definition of the data. It is possible to "skip over" parts of the binary data that AMPS should ignore by declaring that data to be padding. A `struct` message type definition must include one or more `Field` elements, specified in the order in which the data appears in the message. Each `Field` specifies the name of the field, and the type and length of the data to be used for that `Field`. The specifier for a field is composed of: _field name_ `=` _data format specifier_ where the field name is the XPath identifier that AMPS will use for this field and the data format specifier is a specifier for the type of data and number of bytes for this field. The data format specifier is modeled after the specifiers for the Python `struct` module. The format requires a data type specifier. The data type specifier may be preceded by an optional byte order specifier. For variable-width data types (for example, strings), include an optional byte order specifier and an optional count specifier. The following data type specifiers are recognized by the module: | Specifier | C Type | Size (bytes) | AMPS Type | | ------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------ | | `x` | _n/a_ | (number of bytes must be specified) | padding bytes, ignored by AMPS | | `c` | `char` | 1 (number of bytes may be specified) | string | | `b` | `signed char` | 1 | integer | | `B` | `unsigned char` | 1 | integer | | `?` |

    `bool`

    (as though `stdbool.h` were included)

    | 1 | boolean | | `h` | `short` | 2 | integer | | `H` | `unsigned short` | 2 | integer | | `i` | `int` | 4 | integer | | `I` | `unsigned int` | 4 | integer | | `l` | `long` | 4 | integer | | `L` | `unsigned long` | 4 | integer | | `q` | `long long` | 8 | integer | | `Q` | `unsigned long long` | 8 | integer | | `n` | `ssize_t` | | integer | | `N` | `size_t` | | integer | | `f` | `float` | 4 | double | | `d` | `double` | 8 | double | | `s` | `char[]` |

    Number of bytes must be specified.

    AMPS will interpret the specified number of bytes as the string.

    | string | | `S` | `char[]` |

    Number of bytes must be specified.

    AMPS will interpret the string up to the first NULL character or the number of bytes specified.

    | string | | `p` | `uint8_t` followed by `char[]` |

    Number of bytes must be specified.

    AMPS will consume the number of bytes specified in the `Field` configuration. The first byte of data specifies the length of the string. Only that number of bytes, starting with the second byte of the data, will be interpreted as the value.

    | string | The byte order specifiers are as follows: | Specifier | Byte Order | | ------------- | -------------------------------------- | | `@` | Native (little-endian for AMPS server) | | `=` | Native (little-endian for AMPS server) | | `<` | Little-endian | | `>` | Big-endian | If no byte order is specified, AMPS assumes a little-endian byte order to match the native byte order of the AMPS server. For example, given the following C struct: ```c showLineNumbers struct data_type { int32_t id; int32_t internal_id; float price; char label[32]; char code[16]; char routing_instructions[32]; }; ``` A message type declaration that would make the `id`, `price`, and `code` members available to AMPS could be constructed as follows: ```xml showLineNumbers sample_struct_type struct /id = i /ignored = 4x /price = f /ignored = 32x /code = 16s ``` This configuration declares that the message will be interpreted as follows: * The first four bytes of the message will be interpreted as a (little-endian) integer, and that value will be considered to be the `/id` field of the message. * The next four bytes are skipped as padding -- notice that, although a field name is required in the specifier syntax, padding bytes are ignored by AMPS, so there is no field named `/ignored` generated. However, if the message is pretty-printed (or displayed in Galvanometer), AMPS will indicate that there are padding bytes present for this field. * The next four bytes are interpreted as a (little-endian) float, and that value will be considered the value of the `/price` field. * The next 32 bytes (the `label` in the C struct) are skipped as padding. Again, if the message is pretty-printed (or displayed in Galvanometer), AMPS will indicate that there are padding bytes present for this field. * The next 16 bytes of the message are an AMPS string that will be used for the value of the `/code` field. * Any remaining bytes (the `routing_instructions` from the C struct, in this case) are ignored. If the message is pretty-printed (or displayed in Galvanometer), AMPS will indicate that there are extra bytes present. ## Limitations of Struct Message Types Since the `struct` message type requires a specific, fixed definition for messages, AMPS does not support operations that construct messages that may contain arbitrary values. In particular, message types defined using the `struct` message type do not support: * Creating a View with a `struct` message type as the `MessageType`. AMPS allows you to aggregate `struct` message types and project the results as another message type, but the destination `MessageType` for a View cannot be a `struct` message type. * Creating an aggregated subscription for a topic that contains messages of a `struct` message type. * Subscriptions to AMPS internal topics (for example, `/AMPS/ClientStatus`). * Enriching or preprocessing messages of a `struct` message type. * Delta publish or delta subscribe * Select lists --- # Message Types Message communication between the publisher and subscriber in AMPS is managed through the use of message types. Message types define the data contained within an AMPS message. Each topic has a specific message type. Transports used for publishers and subscribers can also define specific message types. For a given transport, AMPS will only process messages of the type or types that the transport accepts. When AMPS needs to use the data within a message, AMPS uses the message type to parse the message into an internal representation. AMPS uses the same internal representation for all message types. Likewise, if AMPS needs to create a new message from a set of values (for example, for a view), AMPS uses the message type to serialize that set of values into the correct format. AMPS filters, commands, processing flow, and so forth are the same for every message type. Message types do not change how AMPS processes messages. A message type simply allows AMPS to work with data of a particular format. In some cases, a given message type cannot support all of the capabilities in AMPS. For example, the unparsed `binary` message type allows arbitrary payloads. This can be extremely useful, but because there is no set format for that message type, none of the capabilities that rely on parsing data are supported by the `binary` message type. Where a message type cannot provide a specific capability to AMPS, those limitations are described below. Except where limitations are described in this section, all message types provided with the AMPS server support all AMPS features. The AMPS engine itself is message-type agnostic. There is no difference in configuring a SOW that uses a composite type than there is configuring a SOW that uses JSON, or BFlat, or Google Protocol buffers. Message types in AMPS are implemented as plug-in modules. For more information on plug-in modules, contact 60East support for access to the AMPS Server SDK. The communication transports used by AMPS accept message sizes of up to approximately 200MB in a single command to AMPS. Messages larger than 200MB may be rejected by the transport as invalid. Should your use of AMPS require larger message sizes, contact 60East support. :::tip AMPS limits messages to approximately 200MB in total size. ::: --- # Statistics Collection The basic monitoring interface is useful for examining many important aspects about an AMPS instance. This includes health and monitoring information for the AMPS engine as well as the host AMPS is running on. All of this information is designed to be easily accessible to make gathering performance and availability information from AMPS easy. The monitoring interface also provides easy access to perform administrative actions. The root of the AMPS Monitoring interface URI contains the following child resources: * The `host` resource provides information about the current operating system state. * The `instance` resource provides information about the instance of AMPS. * The `administration` resource provides access to functions that modify the state of the instance (such as disconnecting a client). The information in the monitoring database is taken from the statistics database for the AMPS instance. AMPS provides actions for managing the statistics database, as described in the section on the action to [Truncate Statistics](/docs/amps-user-guide/actions/do-elements/do-manage-stats). The fields provided through the basic monitoring interface (and the statistics database) are described in the [AMPS Monitoring Guide](../../amps-monitoring-guide/). --- # Galvanometer The AMPS Galvanometer provides an extensive set of visualizations of the state of the instance. Galvanometer also provides the ability to query the instance and display the results. ## Understanding Galvanometer Galvanometer is a JavaScript application that uses the administrative monitoring interface to provide information about an AMPS instance. Galvanometer also includes a lightweight, read-only AMPS client application (using the AMPS Javascript client) that can be optionally enabled to inspect data in the instance. ## Using TLS/SSL with Galvanometer When the `Admin` interface is configured to use TLS/SSL, Galvanometer will also use TLS/SSL with the certificate and key file specified. For the replication graph to be correctly displayed, the instances that replicate to each other must either _all_ use TLS/SSL for the `Admin` interface or _none_ of the instances can use TLS/SSL for the `Admin` interface. If some of the instances in the replication graph use TLS/SSL for the `Admin` interface and some do not, the information shown in the replication graph will be incomplete. ## Authorization and Entitlement in Galvanometer In order to enable Galvanometer to provide credentials to the AMPS instance (in case it is required to access AMPS monitoring information), the special `WWWAuthenticate` option is supported. This option specifies how credentials will be provided to AMPS. The option can have the following values: * Negotiate (Kerberos) * NTLM * Basic realm="\" (Basic Auth) When using `Negotiate` or `NTLM`, Galvanometer will automatically supply corresponding authorization tokens to AMPS. If `Basic Auth` is used for authorization, the Login/Password dialog will require a user to enter credentials. ```xml showLineNumbers ... Basic realm="AMPS Admin" ... ``` ### Statistics Entitlement Galvanometer uses queries of the HTTP admin interface to provide the state of the instance. These queries are handled as any other query of the admin interface. If the user does not have permission to view a particular path in the Admin interface, the AMPS Admin interface will not provide that data and Galvanometer will not show meaningful results for those statistics. As described in the [Entitlement](../securing/entitlement) section of this guide, the AMPS entitlement system treats statistics retrieval as a `read` request to an `admin` resource type. A user that is not entitled to retrieve a specific `admin` resource cannot view those statistics, and so Galvanometer will not show that information. ### Entitlement to Administrator Actions As with requests for statistics, the AMPS entitlement system treats a request for an administrative action as a `read` request to an `admin` resource type. A user that is not entitled to access that resource will not be able to run the action. For actions that can alter server state (such as disconnecting a client), Galvanometer will run an entitlement check to see if the current user has permission to perform the action. The results of this entitlement check are used to determine how Galvanometer will display the control for that action. The control may be hidden or shown as disabled if the current user does not have permission to that action. These entitlement checks do *not* indicate that the action has been performed. See the [Entitlement](../securing/entitlement) section of this guide for more information. ### Using Anonymous Paths The `AnonymousPaths` option allows Galvanometer to bypass authentication and/or entitlement for `Admin` paths that match a regular expression. For resources that match the `AnonymousPaths` option, the `Admin` interface does not require authentication and does not check entitlements. The most common use of `AnonymousPaths` is to allow Galvanometer to correctly display the replication graph when the instance is configured to use `Negotiate` or `NTLM` for authorization. Galvanometer determines the replication graph by polling the instances that participate in replication. Since most browsers disallow sending cross-domain authorization tokens, it is necessary to provide access to replication paths without requiring authorization for Galvanometer to be able to display the replication graph. For installations that use `Negotiate` or `NTLM`, Galvanometer may not be allowed to construct a replication graph if this option is not set. `AnonymousPaths` can also be used to provide access to a specific resource, without allowing access to any other information in the `Admin` interface. For example, an instance might specify `^/amps$` for unauthenticated users to be able to verify that the instance is running and processing `Admin` requests, but without allowing those users to obtain any other data about the instance. The following example shows how to add an `AnonymousPaths` directive that allows any connection to access replication information about the instance. ```xml showLineNumbers ^/amps/instance/replication ``` The `AnonymousPaths` option is **disabled** by default. ### Make Replication Page Work with NTLM / Negotiate Authentication When using `Negotiate` or `NTLM` for authorization and/or entitlement, it prevents Galvanometer from correctly displaying replication graphs by forbidding access to destination instances of AMPS since most browsers disallow sending cross-domain authorization tokens that are required in order to authorize AJAX data requests from a browser. ## Enabling Queries and Subscriptions in Galvanometer Much of the functionality available in Galvanometer uses the basic monitoring interface. Galvanometer submits queries and subscriptions to AMPS using the `websocket` protocol. To use these functions in Galvanometer, you must provide the name of a `Transport` of type `websocket` for Galvanometer to use. For example, the following directive specifies that Galvanometer will use the `Transport` with the `Name` of `websocket-any` to submit commands to AMPS. ```xml showLineNumbers websocket-any ``` The configuration block above requires that the AMPSConfig file contains a `Transport` with the `Name` of `websocket-any` of `Type` `websocket`. When this configuration item is specified, Galvanometer will enable the query and subscription capabilities, and submit commands to AMPS over the specified `Transport`. The queries and subscriptions use the AMPS JavaScript client to connect to AMPS. For example, the `websocket-any` transport referenced in the snippet above might be defined as follows: ```xml showLineNumbers websocket-any websocket tcp 9008 ``` Notice that Galvanometer connects as a client using this `Transport`. There is no special transport or protocol for Galvanometer, and the security configured for the instance (or the `Transport`) applies to Galvanometer. :::warning If the `Transport` is configured to use TLS/SSL, it must use certificates signed by a certificate authority (CA) known to the browser that will be used to access AMPS. For security reasons, browsers disallow self-signed certificates by default. This means that, although a client application may be able to connect, a browser will not allow a websocket connection to a transport that uses a self-signed certificate. ::: ### Make Galvanometer Queries and Subscriptions Work Through a Proxy When an AMPS instance exists underneath a proxy, you may want to include the `SQLTransportInetAddr` element to directly provide the URI that the Galvanometer will use for submitting queries and subscriptions to AMPS. For example, a proxy that hosts access to a secure websocket transport where `proxy_host_address/amps_four/wss` is the path the proxy uses to indicate the `SQLTransport` port: ```xml showLineNumbers proxy_host_address/amps_four/wss ``` ## Queries and Subscriptions with Basic Auth in Galvanometer When Basic Auth is used for authorization and entitlement, an additional option `TrustedAdmin` allows Galvanometer to use a valid session cookie created after successful authorization to the Admin API for queries and subscriptions. This option forces AMPS to reuse credentials supplied by Galvanometer for websocket connections created by Galvanometer. ```xml showLineNumbers ... websocket-portal websocket enabled ... ``` `TrustedAdmin` is only supported by the websocket-based protocols and is **disabled** by default. ## Disabling Galvanometer Galvanometer is enabled in the monitoring interface by default. To disable Galvanometer, add the following directive to the `Admin` configuration block: ```xml showLineNumbers disabled ``` Disabling Galvanometer with this configuration item has no effect on the basic monitoring interface. --- # Configuring Monitoring The AMPS monitoring interface is defined in the configuration file used on AMPS start up. The `Admin` tag is used to control the behavior of the administration server and statistics collection for the instance. Described below are the configuration items available for `Admin`. Expand each item for more details.
    `InetAddr` Defines a port for the embedded HTTP admin server, which can then be accessed via a browser. This element can also specify an IP address, in which case the HTTP server listens only on that address. If no IP address is specified, the HTTP server listens on all available addresses. Starting with version 5.3.3, both IPv4 and IPv6 address formats are fully supported for specifying the network address of the embedded HTTP server. If no address is specified AMPS will listen for incoming connections on both IPv4 and IPv6 protocols. If you wish to limit AMPS to listen for addresses of only a specific IP protocol you may specify the `ANY` address for that protocol. For example: `0.0.0.0:8445` will cause AMPS to listen on port 8445 for only IPv4 addresses. `[::]:8445` will cause AMPS to listen on port 8445 for only IPv6 addresses. There is no default for this parameter. If this parameter is not provided, AMPS does not provide an HTTP admin server, but will continue to collect statistics.
    `FileName` Location for storing the statistics information reported by the Admin Server. When a filename is provided, 60East recommends configuring an `Action` to periodically truncate the statistics in the file. See [Truncate Statistics](/docs/amps-user-guide/actions/do-elements/do-manage-stats) for details. Default: `:memory:` When the `FileName` is set to the default, the statistics database is maintained in memory.
    `Interval` The refresh interval for the Admin Server to update gathered statistics. Default: `10s` Minimum: `1s`
    `WWWAuthenticate` The HTTP authentication type used for the Admin Server when `Authentication` is configured. This specifies how the Admin Server will retrieve credentials from HTTP requests. This option accepts one of:
    • `Basic realm=""`: Basic authentication
    • `NTLM`: Microsoft security protocol
    • `Negotiate`: Negotiated authentication
    Default: `Negotiate`
    `Authentication` The authentication to use for the Administrative interface. This is an `Authentication` element, as described in the [Configuring Authentication](/docs/amps-user-guide/securing/configuring-authentication) section.
    `Entitlement` The entitlement to use for the Administrative interface. This is an `Entitlement` element, as described in the [Configuring Entitlement](/docs/amps-user-guide/securing/configuring-entitlement) section of this guide.
    `AnonymousPaths` The regular expression that defines paths in the Admin Server that can be accessed anonymously without going through authentication and entitlement. Default: There is no default for this option.
    `Header` Adds the specified HTTP header to responses from the Admin console. The contents of this element are added as an HTTP header verbatim. To add more than one header, include this element multiple times. For example, the following elements add the specified headers to HTTP responses from the admin console: ```xml showLineNumbers
    X-Special-Information: "AMPS Admin"
    X-Other-Information: "abc;123"
    ``` Default: There is no default for this option.
    `ExternalInetAddr` This parameter allows the instance to explicitly report a value that should be used for connections to the admin interface. When an upstream instance replicates to this instance, this is the address that will be recorded as the admin address in the statistics database for the upstream instance. This is the address that will be used by Galvanometer to collect replication information for this instance when Galvanometer displays a replication view. Notice that this parameter does not affect the addresses that are used by the admin server. Instead, it provides information on the address to use to reach this server in cases where this should be a different address than that used for replication between instances. This parameter is useful for allowing Galvanometer to build a replication view in cases where the admin interface must be reached through a specific address; for example, when the instance must be accessed through a proxy. Default: There is no default for this option.
    `AccessControlAllowOrigin` This option is included to allow the replication mesh functionality of Galvanometer to function in cases where organizational policies require an explicit domain list in the `Access-Control-Allow-Origin` header of HTTP responses, particularly in cases where requests to the admin console might be received from multiple subdomains (such as `nyc.us.my.com`, `lon.uk.my.com`, `tok.jp.my.com`, etc.). If this option is used, the value should be set to a regular expression that matches the set of domains from which requests might be expected. When this option is set, and the `Origin` header in the incoming HTTP request matches the value provided, the admin web server will return the value of the `Origin` header in the request as the value of the `Access-Control-Allow-Origin` header of the response. Otherwise, the AMPS server will respond with the detected host address of the AMPS server. If this option is set, to comply with the CORS standard, the configuration should also generally include a header that indicates the origin response can vary on each request, as follows: `
    Vary: Origin
    ` When this option is not set, the Admin server will default to providing `*` for the header value.
    `SessionOptions` This option allows you to set options for the cookie provided to authenticated admin conditions. The value of the option is appended to the admin cookie. This option should be a valid set of options for an HTTP cookie. By default, the value of this option is `max-age=86400; path=/; HttpOnly; secure` when the Admin interface is configured to use https. Otherwise, `max-age=86400; path=/; HttpOnly`.
    `SQLTransport` This option allows you to define the `Transport` (must be of type `websocket`) that the Galvanometer will use to submit queries and subscriptions to AMPS. The value of this option should point to a configured `Transport` of type `websocket` as defined in the `Transports` section. Default: There is no default for this option. If this parameter is not provided, this indicates that no `Transport` may be used by Galvanometer and the SQL page in the Galvanometer will be greyed out.
    `SQLTransportInetAddr` This option allows you to define the URI that the Galvanometer will use to submit queries and subscriptions to AMPS. The value of this option should point to the `SQLTransport` port. If this option is used in an proxied environment, the value of this option should point to the path the proxy uses to indicate the `SQLTransport` port. Default: There is no default for this option. When not provided, the Galvanometer will use the host that Galvanometer was loaded from when opening a connection for queries and subscriptions.
    :::info By default, AMPS will store the monitoring interface database information in system memory. If the AMPS instance is going to be up for a long time, or the monitoring interface statistics interval will be updated frequently, or if this is a production system where it is important to be able to troubleshoot problems, it is strongly recommended that the `FileName` setting be specified to allow persistence of the data to a local file. ::: AMPS supports the ability to connect to the Admin interface over HTTPS. To enable HTTPS, provide a `Certificate` and a `PrivateKey` in the `Admin` configuration block. Described below are the configuration items used to enable HTTPS. Expand each item for more details.
    `Certificate` The certificate file to use for the Admin Server. Default: There is no default for this option.
    `PrivateKey` The private key to use for the Admin Server. Default: There is no default for this option.
    `Ciphers` The cipher list to use for the Admin Server. The cipher list is passed to the OpenSSL implementation without being interpreted by the AMPS server. For OpenSSL, details on the format of the cipher list are available at: [https://www.openssl.org/docs/man1.1.1/man1/ciphers.html](https://www.openssl.org/docs/man1.1.1/man1/ciphers.html) Default: There is no default for this option.
    Below is an example of how to configure the monitoring interface. This will start an http server in the AMPS process. ```xml showLineNumbers stats.db localhost:8085 10s ``` In this example `localhost` is the hostname and `8085` is the port assigned to the monitoring interface. With this configuration: | | | | -------------------------- | ----------------------------------------- | | http://localhost:8085/ | Root URI for Galvanometer. | | http://localhost:8085/amps | Root URI for simple monitoring interface. | The `Interval` tag is used to set the update interval for the AMPS monitoring interface. In this example, statistics will be updated every 10 seconds. The basic monitoring interface is accessible through a web browser, but also follows a Representational State Transfer (RESTful) URI style for programmatic traversal of the directory structure of the monitoring interface. Below is an example of how to configure monitoring to allow the Galvanometer to collect replication information when a specific address needs to be used by the admin interface. ```xml showLineNumbers 9090 stats.db 20s proxy.example.com:8185 ``` --- # Output Formatting The AMPS monitoring interface offers several possible output formats to ease the consumption of monitoring reporting data. The possible options are XML, CSV and RNC output formats, each of which is discussed in more detail below. ## JSON Document Output All monitoring interface resources can have the current node, along with all child nodes, list its output as a JSON document by appending the `.json` file extension to the end of the resource name. For example, if an administrator would like to have a JSON document of all of the CPUs on the server, including all the relevant statistics about those CPUs, then the following URI will generate that information: `http://localhost:8085/amps/host/cpus.json` The document that is returned will be similar to the following: ```javascript showLineNumbers { "amps": { "host": { "cpus": [ { "id":"all", "idle_percent":"62.452316076294", "iowait_percent":"0.490463215259", "system_percent":"10.681198910082", "user_percent":"26.376021798365" }, { "id":"cpu0", "idle_percent":"75.417130144605", "iowait_percent":"0.333704115684", "system_percent":"7.563959955506", "user_percent":"16.685205784205" }, { "id":"cpu1", "idle_percent":"50.000000000000", "iowait_percent":"0.642398286938", "system_percent":"13.597430406852", "user_percent":"35.760171306210" } ] } } } ``` Appending the `.json` file extension to any AMPS monitoring interface resource will generate the corresponding JSON document. ## XML Document Output All monitoring interface resources can have the current node, along with all child nodes, list its output as an XML document by appending the `.xml` file extension to the end of the resource name. For example, if an administrator would like to have an XML document of all of the currently running processors, including all the relevant statistics about those processors, then the following URI will generate that information: `http://localhost:8085/amps/instance/processors/all.xml` The document that is returned will be similar to the following: ```xml showLineNumbers 0 0 AMPS Aggregate Processor Stats 1855 0 0 0 0 0 ``` Appending the `.xml` file extension to any AMPS monitoring interface resource will generate the corresponding XML document. ## CSV Document Output The `.csv` file extension can be appended to any **leaf node** resource to have a CSV file generated to examine those values. This can also be coupled with the time range selection to generate reports. See [Time Range Selection](time-range-selection) above for more details on time range selection. Below is a sample of the `.csv` output from the monitoring interface from the following URL: `http://localhost:8085/amps/instance/processors/all/matches_found_per_sec.csv?t0=20230830T0` This resource will create a file with the following contents: ```bash showLineNumbers 20230830T000000.000000Z,94244 20230830T000010.000000Z,304661 20230830T000020.000000Z,301078 20230830T000030.000000Z,304661 20230830T000040.000000Z,0 20230830T000050.000000Z,0 20230830T000100.000000Z,0 20230830T000110.000000Z,0 20230830T000120.000000Z,302390 20230830T000130.000000Z,307637 20230830T000140.000000Z,0 20230830T000150.000000Z,0 20230830T000200.000000Z,0 ``` ## Leaf Nodes A leaf node of the monitoring interface represents a single recorded statistic. Leaf nodes **do support** CSV document output. Examples of leaf node endpoints include: ```bash http://localhost:8085/amps/instance/processors/all/messages_received_per_sec ``` or ```bash http://localhost:8085/amps/instance/transaction_log/write_latency ``` ## Non-Leaf Nodes A non-leaf node represents an aggregate of related statistics. Non-leaf nodes **do not support** CSV document output. Due to the limitations of the tabular format of CSV, there is no clear way to translate the hierarchical structure of a non-leaf node into a CSV document. Examples of non-leaf node endpoints include: ```bash http://localhost:8085/amps/instance/processors/all ``` or ```bash http://localhost:8085/amps/instance/transaction_log ``` ## RNC Document Output AMPS supports generation of an XML schema via the Relax NG Compact (RNC) specification language. To generate an RNC file, enter the following URL in a browser: ```bash http://localhost:port/amps.rnc ``` AMPS will display the RNC schema. To convert the RNC schema into an XML schema, first save the RNC output to a file: ```bash %> wget http://localhost:9090/amps.rnc ``` The output can then be converted to an xml schema using Trang (available at [http://code.google.com/p/jing-trang/](http://code.google.com/p/jing-trang/)) with: ```bash trang -I rnc -O xsd amps.rnc amps.xsd ``` --- Most data provided in the AMPS monitoring interface is collected at the interval specified in the statistics configuration. This means that the statistics presented are not a continuous sample, or a sample when "all operations are complete". Instead, the statistics reflect the point in time when the statistics are collected. When monitoring statistics, consider not only the values reported, but the change in values over time and how those values compare to other values. For example, the `transport_rx_queue` for a client connecting over the network indicates the number of bytes currently in the TCP buffer for a given connection. However, a single sample that shows a nonzero count for this metric simply means that, at the time statistics were collected, there were bytes in that buffer. Whether this indicates a problem or not depends on the context. If the next sample for that client shows that AMPS is consuming a large number of bytes for the client, then the fact that there are bytes in the queue simply means that the connection is active. On the other hand, if the `bytes_in` (total bytes consumed for this connection) and `bytes_in_per_second` (bytes per second consumed for this connection, averaged over the last statistics interval) indicate a slowdown, that might be cause for concern. Likewise, an empty `transport_rx_queue` does not mean that no messages are being received for the client if `bytes_in` and `bytes_in_per_second` show the expected traffic. Likewise, a client that connects, runs a query, consumes the results and disconnects in a period of time less than the statistics interval, may not be captured in the statistics database at all. If the client connection does not exist at any of the times that statistics are recorded, AMPS will not capture statistics for that client connection, so that client will not appear in the statistics database. --- # Time Range Selection AMPS keeps a history of the monitoring interface statistics, and allows that data to be queried. By selecting a leaf node of the monitoring interface resources, a time-based query can be constructed to view a historical report of the information. A time-based query is created by appending either one or both of the `t0` or `t1` query parameters to a url of the admin REST interface. For example, if an administrator wanted to see the number of messages per second consumed by all processors from midnight UTC on November 30, 2011 until 23:25:00 UTC on November 30, 2011, then pointing a browser to: ```bash http://localhost:8085/amps/instance/processors/all/messages_received_per_sec?t0=20111130T0&t1=20111130T232500 ``` will generate the report and output it in the following plain text format (note: entire dataset is not presented, but is truncated). ```csv 20111130T000000.000000Z,0 20111130T000010.000000Z,0 20111130T000020.000000Z,0 20111130T000030.000000Z,94244 20111130T000040.000000Z,304661 20111130T000050.000000Z,301078 20111130T000100.000000Z,308922 20111130T000110.000000Z,306177 20111130T000120.000000Z,302140 20111130T000130.000000Z,302390 20111130T000140.000000Z,307637 20111130T000150.000000Z,310109 20111130T000200.000000Z,309888 20111130T000210.000000Z,299993 20111130T000220.000000Z,310002 20111130T000230.000000Z,300612 20111130T000240.000000Z,299387 ``` All times used for the report generation and presentation are ISO-8601 formatted. ISO-8601 formatting is of the following form: `YYYYMMDDThhmmss`, where `YYYY` is the year, `MM` is the month, `DD` is the year, `T` is a separator between the date and time, `hh` is the hours, `mm` is the minutes and `ss` is the seconds. Decimals are permitted after the `ss` units. All times used for the report generation and presentation are stored and returned in UTC time. :::info As discussed in the following sections, the date-time range can be used with plain text (html), comma-separated values (csv), json, and XML formats. ::: ## Time Based Query Behavior All times used for the `t0` and `t1` parameters **must be** ISO-8601 formatted. ISO-8601 formatting is of the following form: `YYYYMMDDThhmmss`, where `YYYY` is the year, `MM` is the month, `DD` is the year, `T` is a separator between the date and time, `hh` is the hours, `mm` is the minutes and `ss` is the seconds. Decimals are permitted after the `ss` units. All times used for the `t0` and `t1` parameters **must be** in UTC time. All times in the admin interface are stored and compared in UTC time. The behavior of time based queries is affected by the combination of the `t0` and `t1` query parameters. These behaviors are described in the table below: | Query Parameter Values | Behavior | | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Only `t0` is set | The result set is the list of values recorded from time `t0` until the latest recorded admin interval. The range is inclusive. | | Only `t1` is set | The result set is the list of values recorded from the first admin interval recorded in the stats.db of the AMPS instance until time `t1`. The range is inclusive. | | Both `t0` and `t1` are set to _different_ timestamp values | The result set is the list of values recorded starting from time `t0` until time `t1`. This range is inclusive. | | Both `t0` and `t1` are set to _the same_ timestamp value |

    Returns a single value that represents the recorded value of that statistic at the specific admin interval set by `t0` and `t1`.

    Important:

    Unlike the other time query behaviors, when selecting a single data point by setting `t0` and `t1` to the same value, that value must be a timestamp that is present in the statistics. Valid timestamps can be obtained via the instance/timestamp admin API endpoint.

    | ## Leaf Nodes A leaf node of the monitoring interface represents a single recorded statistic. Leaf nodes **fully support** time-range selections using the `t0` and `t1` parameters as described above. Examples of leaf node endpoints include: ```bash http://localhost:8085/amps/instance/processors/all/messages_received_per_sec ``` or ```bash http://localhost:8085/amps/instance/transaction_log/write_latency ``` ## Non-Leaf Nodes A non-leaf node represents an aggregate of related statistics. Non-leaf nodes **do not support** time-range selections. Non-leaf nodes support historical queries for a specific valid historical admin interval timestamp. A query for a specific timestamp is achieved by setting the `t0` and `t1` parameters to the same timestamp value as described above. Examples of non-leaf node endpoints include: ```bash http://localhost:8085/amps/instance/processors/all ``` or ```bash http://localhost:8085/amps/instance/transaction_log ``` --- # Monitoring AMPS The AMPS monitoring interface has two distinct components: 1. A basic monitoring interface that provides statistics for the AMPS instance in common machine-readable formats. This interface also provides administrative functions, such as enabling and disabling transports, disconnecting clients, and upgrading and downgrading replication links. 2. The AMPS _Galvanometer_, a browser-based monitoring tool that shows a graphical representation of the statistics for AMPS. The Galvanometer includes the ability to automatically request information about replication message flow from a set of replicated instances and provide information about that message flow across that set of instances. It includes the ability to enter subscriptions and queries and display the results in a grid. The AMPS _Galvanometer_ uses the basic monitoring interface for access to AMPS statistics. This section describes the basics of the monitoring interface. The [AMPS Monitoring Guide](../amps-monitoring-guide/) describes the statistics that AMPS collects in detail. --- # Out-of-Focus Messages (OOF) One of the more difficult problems in messaging is knowing when a record that previously matched a subscription has been updated so that the record no longer matches the subscription. AMPS solves this problem by providing an out-of-focus, or _OOF_, message to let subscribers know that a record that previously matched a subscription no longer matches the subscription. The OOF messages help subscribers easily maintain state and remove records that are no longer relevant. OOF notification is optional. A subscriber must explicitly request that AMPS provide out-of-focus messages for a subscription. When OOF notification has been requested, AMPS produces an `oof` message for any record that previously matched the subscription at the point at which: * The record is deleted * The record expires * The record no longer matches the filter criteria * The record is no longer within the pagination window (for paginated subscriptions), _or_ * The subscriber is no longer entitled to view the new state of the record. AMPS produces an `oof` message for each record that no longer matches the subscription. The `oof` message is sent as part of processing the update that caused the record to no longer match. Each `oof` message contains information the subscriber can use to identify the record that has gone out of focus and the reason that the record is now out of focus. Since AMPS must maintain the current state of a record to know when to produce an `oof` message, these messages are only supported for SOW topics, conflated topics, and views. The `oof` option is not supported for subscriptions that do not refer to a SOW topic, conflated topic, or view. When AMPS returns an OOF message, the data contained in the body of the message represents the updated state of the message (except as described below). This will allow the client to make a determination as to how to handle the data, be it to remove the data from the client view, change the subscription to broaden the filter, remove data from a calculation, and so on. Because the notification contains the reason and, where applicable, the new data, an application can take a different action depending on why the message no longer matches. For example, an application may present a different icon for an order that moves to a status of `completed` than it would present for an order that moves to a status of `cancelled` or an order that has expired. When a `delta_publish` command causes the SOW record to go out of focus, AMPS returns the fully merged record. When there is no updated message to send, AMPS sends the state of the record before the change that produced the `oof`. This can occur when the message had been deleted, when the message has expired, or when an update causes the client to no longer have permission to receive the record. For a conflated view or a subscription that uses conflation, the data included in the `oof` message will be the last data that the subscriber received. When both an update to a message and a change that would cause the message to go out of focus happen in the same conflation interval, the subscriber receives an `oof` notification with the previously-received state of the message. Likewise, if a change that causes a message to go out of focus and a change that causes the message to come back into focus occur within the same conflation interval, the subscriber receives the state of the message at the end of the conflation interval. The subscriber does not receive an indication that the message had gone out of focus during the conflation interval and then come back into focus. ## Out-of-Focus Reason Codes An out of focus message returns the reason that the message was produced in the `reason` field of the message header.
    Reason FieldDescriptionMessage Contents
    `deleted`The message was deleted from the topic.previous message
    `expired`The message was removed from the topic due to expiration.previous message
    `match`The message no longer matches the content filter or has moved outside of the record set requested.updated message that no longer matches subscription criteria
    `entitlement`The message has changed such that the user is not entitled to see the updated message.previous message
    ## Usage Consider the following scenario where AMPS is configured with the following SOW key for the buyer topic: ```XML showLineNumbers buyer xml /buyer/id ``` When the following message is published, it is persisted in the SOW topic: ```XML showLineNumbers 100 NY ``` A client issues a `sow_and_subscribe` request for the topic `buyer` with the filter `/buyer/loc="NY"` and the `oof` option set on the request. The client will be sent the message as part of the SOW query result. Subsequently, the following message is published to update the `loc` tag to LN: ```XML showLineNumbers 100 LN ``` The original message in the SOW cache is updated. The client does not receive the second publish message, because that message does not match the filter (`/buyer/loc="NY"`). This is problematic. The client has a message that is no longer in the SOW cache and that no longer matches the current state of the record. Since the `oof` option was set on the subscription, however, the AMPS engine sends an `oof` message to let these clients know that the message that they hold is no longer in the SOW cache. The following is an example of what's returned. The header of the message will contain the following fields to help the application identify the reason for the `oof` message and which message no longer matches: | Field | Value | | ------- | --------------------- | | Command | `oof` | | Topic | `buyer` | | Reason | `match` | | SowKey | `6387219447538349146` | The message data will contain the updated message, as shown following. ```XML showLineNumbers 100 LN ``` Had the message been deleted, the message data for the OOF notification would contain deleted message, and the reason would be `deleted`. An easy way to think about the situations where AMPS sends an OOF notification is to consider what would happen if the client re-issued the original `sow` request after the above message was published. The `/client/loc="NY"` expression no longer matches the message in the SOW cache and as a result, this message would not be returned. ## Example To help reinforce the concept of OOF messages, and how OOF messaging can be used in AMPS, consider a scenario where there is a GUI application whose requirement is to display all open orders of a client. There are several possible solutions to ensure that the GUI client data is constantly updated as information changes, some of which are examined below; however, the goal of this section is to build up a `sow_and_subscribe` message to demonstrate the power that OOF notifications add to AMPS. ### Client-Side Filtering in a SOW and Subscribe Command First, consider an approach that sends a `sow_and_subscribe` message on the topic `orders` using the filter `/Client="Adam".` AMPS completes the `sow` portion of this call by sending all matching messages from the `orders` SOW topic. AMPS then places a subscription whereby all future messages that match the filter get sent to the subscribing GUI client. As the messages come in, the GUI client will be responsible for determining the state of the order. It does this by examining the `State` field and determining if the state is equal to “Open” or not, and then updating the GUI based on the information returned. This approach puts the burden of work on the GUI and, in a high volume environment, has the potential to make the client GUI unresponsive due to the potential load that this filtering can place on a CPU. If a client GUI becomes unresponsive, AMPS will queue the messages to ensure that the client is given the opportunity to catch up. The specifics of how AMPS handles slow clients is covered in the section discussing [Slow Client Management](ha/slow-client-management-and-capacity-limits). ### AMPS Filtering in a SOW and Subscribe Command The next step is to add an additional ’AND’ clause to the filter. In this scenario we can let AMPS do the filtering work that was previously handled on the client. This is accomplished by modifying our original `sow_and_subscribe` to use the following filter: ```sql /Client = "Adam" AND /State = "Open" ``` Similar to the above case, this `sow_and_subscribe` will first send all messages from the `orders` SOW topic that have a `Client` field matching "Adam" and a `State` field matching "Open". Once all of the SOW topic messages have been sent to the client, the subscription will ensure that all future messages matching the filter will be sent to the client. There is a less obvious issue with this approach to maintaining the client state. The problem with this solution is that, while it initially will yield all open orders for client "Adam", this scenario is unable to stay in sync with the server. For example, when the order for Adam is filled, the `State` changes to `State=Filled`. This means that, inside AMPS, the order on the client will no longer match the initial filter criteria. The client will continue to display and maintain these out-of-sync records. Since the client is not subscribed to messages with a `State` of “Filled,” the GUI client would never be updated to reflect this change. ### OOF Processing in a SOW and Subscribe Command The final solution is to implement the same `sow_and_subscribe` query which was used in the first scenario. This time, we use the filter requests only for the `State` that we're interested in, but we add the `oof` option to the command so the subscriber receives OOF messages. ```sql /Client = "Adam" AND /State = "Open" options: oof ``` AMPS will respond immediately with the query results, exactly as it does with a `sow_and_subscribe` command that does not use the `oof` option. This approach provides the following advantage: for all future messages in which the same `Open` order is updated, such that its status is no longer `Open`, AMPS will send the client an `OOF` message specifying that the record which previously matched the filter criteria has fallen out of focus. AMPS will not send any further information about the message unless another incoming AMPS message causes that message to come back into focus. In the following diagram, the Publisher publishes a message stating that Adam’s order for MSFT has been fulfilled. When AMPS processes this message, it will notify the GUI client with an `oof` message that the original record no longer matches the filter criteria. The `oof` message will include a `Reason` field with it in the message header, defining the reason for the message to lose focus. In this case the `Reason` field will state `match` since the record no longer matches the filter. AMPS will also send `oof` messages when a message is deleted or has expired from the SOW topic. We see the power of the `oof` message when a client application wants to have a local cache that is a subset of the SOW. This is best managed by first issuing a query filter `sow_and_subscribe` which populates the GUI, and enabling the `oof` option. AMPS informs our application when those records which originally matched no longer do, at which time the program can remove them. --- # Capacity Planning Sizing an AMPS deployment can be a complicated process that includes many factors, such as: configuration parameters used for AMPS, the data used within the deployment and how the deployment will be used. This section presents guidelines that you can use in sizing your host environment for an AMPS deployment given the following components that need to be taken into account: Memory, Storage, CPU and Network. Capacity planning is one of the most important aspects of ensuring that an AMPS deployment can meet the needs of the application. :::tip The capacity planning formulas in this section are intended to help you size a system to run an instance of AMPS. The actual resource consumption will vary based on usage and configuration. ::: ## System Goals and Requirements When planning the capacity for a system, the most important questions to understand are: the purpose of the instance and the Service Level Agreement (SLA) offered by the instance. For example, is this a server for use by a development team for early exploration of ideas, or will this instance be core infrastructure for a major application? Is it important that the instance has the absolute minimum latency possible, or is the most important aspect of the system query response time for a 1TB topic in the SOW? Since AMPS efficiently uses the system hardware, the limits of an AMPS instance are typically a result of the limitations of the underlying host system. Proper capacity planning (and [Linux OS tuning](linux-configuration)) can mean the difference between an instance that performs well and handles increased traffic without incident and an instance that constantly pushes the hardware to the limit and becomes less responsive when traffic increases. For guidance on choosing between physical hardware, virtual machines, and containers, see [Host Guidance](host-guidance). ### Single-Tenant or Multi-Tenant AMPS performs well in both single-tenant and multi-tenant installations. When choosing whether to host multiple applications on a given system (more than one AMPS instance, or a system that hosts both an AMPS instance and other applications), it is important to plan for the _highest_ level of traffic expected on all applications simultaneously. In a business setting, it is common for a sudden increase in traffic to affect a number of systems in the business, rather than being isolated to just one system. When planning capacity for a multi-tenant system, provision a host that exceeds the total maximum capacity required for **all** applications, including AMPS instances, on the system at peak load. For multi-tenant installations, disable AMPS-level NUMA tuning in the configuration file. Likewise, if multiple applications will be hosted on the system, disable AMPS-level NUMA tuning in the configuration file. For container deployments, follow the guidance in [Host Guidance](host-guidance#containers) before leaving AMPS-level NUMA tuning enabled. ```xml showLineNumbers ... disabled ... ``` ## Memory AMPS is designed for high performance. It is designed to use memory, as needed, to improve performance and reduce latency. One of the most important aspects of managing an AMPS instance is, being sure that the instance has enough physical memory available to perform well. This section contains general guidelines for creating an approximate sizing estimate for the AMPS process itself. An estimate on total memory capacity for a server would include the estimate for the AMPS process itself and estimates for any other processes running on the system (including monitoring software, security software, other applications, update and maintenance tasks, and so on). Notice that it is possible for AMPS to maintain quantities of data much larger than physical memory (for example, terabytes of SOW data). For instances that have this requirement, contact 60East support for tuning and sizing guidance. ### Estimating AMPS Instance Memory Usage The best way to estimate the memory usage for an AMPS instance is to simulate, as closely as possible, the traffic and usage pattern for the instance and collect statistics that show the amount of memory that the instance uses. If actual numbers aren't available, you can use the formulas in this section to come up with a working approximation of the amount of memory to make available to AMPS for a given amount of data, number of clients, and so on. The AMPS server will use memory as necessary for performance, so the formulas here offer general estimates for system sizing purposes rather than precise predictions. AMPS needs less than 1GB for its own binary image and initial start up state for most configurations. For production instances, we estimate 5GB as a typical working memory footprint for an active installation. As a general estimate, because of indexing for queries, AMPS may need up to twice the size of messages stored in a topic in the SOW to fully index that topic (the same sizing applies to messages in views and conflated topics). AMPS maintains a copy of the latest journal file in memory for quick access, and maintains a small amount of metadata for each message in an AMPS queue. The `MessageMemoryLimit` configured for the instance (or the total of all `MessageMemoryLimit` settings for each `Transport` in the instance) specifies the total amount of memory devoted to buffering messages for clients, including conflated subscriptions, aggregated subscriptions, and paginated subscriptions. This puts a general estimate of the amount of memory to be available for the AMPS server itself at: $$ \begin{aligned} 5\text{GB} &+ \text{SowSizeEstimate} \\ &+ ( C \times 4096 \text{ bytes}) \\ &+ \text{TMemLimit} \\ &+ (J \times 2) \\ &+ (Q \times 250 \text{ bytes}) [ + (\text{QA} \times 20 \text{ bytes}) ] \end{aligned} $$ where: * $$SowSizeEstimate$$ = Estimates for SOW topic size, as described below (in bytes) * $$C$$ = Number of Clients * $$TMemLimit$$ = Total of all MessageMemoryLimit settings in the instance * $$J$$ = JournalSize setting * $$Q$$ = Total number of active unacknowledged messages in the queues for the instance * $$QA$$ = Total number of acknowledgments received for messages that are not yet in the queue By default, all unacknowledged messages in the instance will be active in the queue. When a queue specifies a `TargetQueueDepth`, the total number of active unacknowledged messages for the queue will, in most cases, be limited to the `TargetQueueDepth`. When acknowledgment messages are received for messages that are not currently active in the queue, AMPS must track those acknowledgments to be able to efficiently prevent those messages from entering the queue. Not every application consumption pattern can produce this situation; however, if it arises, this calculation can help estimate the amount of memory required to maintain information about these acknowledgments until the message enters the queue. To calculate the `SowSizeEstimate`, the memory footprint required for topics, views and conflated topics in the SOW, use the following formula to calculate for each (`Topic`, `View` and `ConflatedTopic`): $$ ( 2 * (S + 128\; bytes) * M ) + ((16\; bytes * M) * H) $$ where: * $$S$$ = Average message size for the Topic, View or ConflatedTopic in the SOW (in bytes) * $$M$$ = Maximum expected number of messages for the Topic, View or ConflatedTopic * $$H$$ = Number of hash indexes for the Topic, View or ConflatedTopic When a memo index is created for a field, the index contains the data for that field. This estimate assumes that all of the fields in the topic may potentially have a memo index created. Notice that queries for a field that is not present in any message in the topic will still produce an index for that field: see the [Indexing SOW Topics](../sow/sow_indexing) section for details. Notice also that applications with a pattern of querying fields that do not exist may produce a larger set of indices. Estimating topic-by-topic generally gives a more precise estimate. However, if that data is not available, you can also use overall message sizes and message count for the instance. If more configuration detail is available, it may be possible to create a more precise estimate. For example, if the `SlabSize` configured for a SOW topic is not an exact fit for the message size + header, it is possible to estimate the amount of free space remaining in each slab. As a simple example, a general estimate of the amount of memory that should be left available to run an instance of AMPS might be: $$ \begin{aligned} 5\text{GB} &+ [ ( 2 \times (1024+128) \times 1,000,000 ) + (16 \times (1,000,000 \times 2) ) \\ &+ ( 2 \times ( 512+128) \times 1,000,000 ) + ( 0 ) \\ &+ ( 2 \times (1024+128) \times 8,000,000 ) + (16 \times (8,000,000 \times 4)) ] \\ &+ ( 200 \times 4096) + ( 10,000,000,000) \\ &+ ( 1,000,000,000 \times 2 ) + ( 750,000 \times 250) \end{aligned} $$ where: For the`SowSizeEstimate`, the instance will have two Topics and a View. For the first topic: * $$S = 1024$$ * $$M = 4,750,000$$ * $$H = 2$$ For a view over the first topic (the view uses no HashIndexes): * $$S = 512$$ * $$M = 3,000,000$$ * $$H = 0$$ For the second topic: * $$S = 1024$$ * $$M = 8,000,000$$ * $$H = 4$$ For the overall AMPS estimate: * $$C = 200$$ * $$TMemLimit = 10,000,000,000 (10GB)$$ * $$J = 1,000,000,000 (1GB)$$ * $$Q = 750,000$$ This shows sizing for an AMPS deployment with the following characteristics: * Three topics in the SOW (including one view): * One topic has a message size of 1024 bytes, will hold 5 million messages and configure 2 hash indexes. * One view has a message size of 512 bytes, will hold 3 million messages and does not configure a hash index. * One topic has a message size of 1024 bytes, will hold 8 million messages and will configure 4 hash indexes. * A maximum of 10GB of memory for in-flight messages and working state (for aggregated subscriptions, pagination sets and so on) * A journal size setting configured to 1GB * A maximum of 750,000 total unacknowledged messages at a time across all message queues * No more than 200 clients connected simultaneously * No external modules loaded This estimate suggests that _no less than_ 52GB of physical memory on the server should be available for the AMPS instance itself while AMPS is processing the expected volume of messages. When AMPS first starts, or if traffic is light, AMPS may consume less than the estimated amount. AMPS may also consume more than this amount of memory during memory-intensive operations in some cases. **Try it yourself:** Experiment with the formula and watch memory needs change in real time. :::tip The formula in this section is a general estimate designed to produce a recommended minimum amount of physical memory to have available for AMPS. It is intended as a guideline when actual measurements are not available. For more accurate estimates, use measurements of the expected workload. A given instance of AMPS may not match these estimates at any particular time, based on usage, precise configuration, traffic, client activity, and so forth. ::: ### Estimating Overall System Capacity The AMPS instance memory usage is one component of estimating the needs of the overall system. In addition to this, there is also: operating system tasks, management and maintenance (including monitoring, security and management software), and any other applications running on the system must also be taken into consideration. Further, Linux memory management is most efficient when the operating system has 10-20% headroom. For best performance and a lower risk of problems related to an unexpected spike in message volume, 60East recommends factoring in all of the components that will consume memory on the system, and then sizing the overall physical memory to handle 200% of the capacity estimated while still retaining 10-20% physical RAM. Note that these are rough guidelines. An especially critical system, or a system that has in the past seen larger volumes might size memory to 350% or more, while a less critical system might allocate less than 200% of the estimated capacity. A VM on a developer desktop might be sized at or below the capacity estimate, since the system is completely under the control of a single user and is not intended to handle production loads. For example, in the estimate above, the system should reserve a minimum of 52GB of free RAM for the AMPS process itself. Suppose that the monitoring, access control and server management software are very lightweight and only consume 3GB of memory under production load. The following estimates would be reasonable: * **Production server with strict SLA and tolerance for usage variation** - _128GB_ This estimate accounts for 200% of the estimated required capacity while still allowing 16GB (between 10 and 20% of the physical memory) free for efficient memory management. (Calculation: 52GB for AMPS, 3GB for the monitoring software = 55GB. Multiply by 2 = 110GB.) For a server that needs to be available during periods of heavy activity, this sizing could be a good option. * **Production server with stable usage or variable SLA** - _96GB_ This estimate covers the expected capacity of the AMPS server and monitoring software and leaves enough headroom for the operating system, but does not allow for large growth in volume or changes in usage patterns. For a server with predictable usage and volumes, or a server where some performance impact is acceptable if volume or usage increases and where the general estimates for AMPS capacity are known to be very precise, this server size could be a good option. * **Shared development server with minimal performance SLA** - _64GB_ This minimal estimate covers the expected capacity of the AMPS server and monitoring system, but does not leave enough excess capacity for the server to absorb unexpected traffic. This server would be expected to have periodic performance degradation and to possibly exit due to out of memory conditions if the volume of traffic increases or the usage pattern changes. This could be a good sizing for an instance that is used only for development, where the instance is not guaranteed to provide any particular performance guarantees and it is acceptable for the server to be temporarily unavailable if there was an unexpected increase in load. ## Storage AMPS needs enough space to store its own binary images, configuration files, SOW persistence files, log files, transaction log journals, and slow client offline storage, if any. Not every deployment configures a SOW or transaction log, so the storage requirements are largely driven by the configuration. ### AMPS Error and Event Log Files Log file sizes vary depending on the log level and how the engine is used. For example, in the worst-case, `trace` level logging. AMPS will need at least enough storage for every message published into AMPS and every message sent out of AMPS plus 20%. For `info` level logging, a good estimate of AMPS log file sizes would be 2MB per 10 million messages published. Logging space overhead can be capped by implementing a log rotation strategy which uses the same file name for each rotation. This strategy effectively truncates the file when it reaches the log rotation threshold to prevent it from growing larger. ### SOW Topics When calculating the amount of storage to reserve for topics in the SOW, there are a couple of factors to keep in mind. The first is the average size of messages stored in the SOW, the number of messages stored in the SOW and the `SlabSize` defined in the configuration file for each `Topic`. Using these values, it is possible to estimate the minimum and maximum storage requirements for the SOW. A rough estimate of the minimum size for a SOW topic is as follows: $$Min = ( MsgSize * MsgCount ) + ( Cores * SlabSize )$$ where: * $$Min$$ = Minimum SOW size * $$MsgSize$$ = Average SOW message size * $$MsgCount$$ = Number of SOW messages * $$Cores$$ = Number of processor cores in the system * $$SlabSize$$ = Slab Size for the SOW A rough estimate of the maximum size for a SOW topic is as follows: $$ MsgSlabMin = \frac{SlabSize}{MsgSize} / 2 $$ $$ Max = ( \frac{MaxMsgCount}{MsgSlabMin} * SlabSize ) + ( Cores * SlabSize ) $$ where: * $$Max$$ = Maximum SOW size * $$SlabSize$$ = Slab size for the SOW * $$MsgSize$$ = Estimated average SOW message size * $$MaxMsgCount$$ = Number of SOW messages * $$Cores$$ = Number of CPU cores in the system The storage requirements should typically be between the two values above, however it is still possible for the SOW to consume additional storage based on the unused capacity configured for each SOW topic. Notice that, as suggested in this calculation, AMPS reserves the configured `SlabSize` for each processor core in the system the first time a thread running on that core writes to the SOW. For example, in an AMPS configuration file with the `SlabSize` set to 1MB, the SOW for this topic will consume 1MB per processor core with no messages stored in the SOW. Pre-allocating SOW capacity in chunks, as a chunk is needed, is more efficient for the operating system and storage devices, and helps amortize the SOW extension costs over more messages. It is also important to be aware of the maximum message size that AMPS guarantees the SOW can hold. The maximum message size is calculated in the following manner: $$Max = SlabSize - 64$$ where: * $$Max$$ = Maximum message size that can be stored in the SOW (in bytes) * $$SlabSize$$ = The configured SlabSize for the SOW (in bytes) This calculation says that the maximum message size that can be stored in the SOW in a single message is the `SlabSize` minus 64 bytes for the record header information. **Try it yourself:** Experiment with the formula to see SOW storage needs. ### Transaction Logs Transaction logs are used for message replay, replication and to ensure consistency in environments where each message is critical. Transaction logs are optional in AMPS (though some features require them), and transaction logs can be configured to record specific topics. When planning for transaction logs, there are three main considerations: 1. The total size needed for the transaction log, including in disaster recovery scenarios 2. The size to allow for each file that makes up the transaction log 3. How many files to preallocate You can calculate the approximate total size of the transaction log once the system reaches steady state as follows: $$TxLog = ( S + 512 ) * N + ( Jsize ) + ( Tindex )$$ where: * $$TxLog$$ = Estimated storage capacity required for transaction log * $$S$$ = Average message size * $$N$$ = Number of messages to retain * $$Jsize$$ = Journal file size * $$Tindex$$ = Topic Index (if configured), larger of 200MB or $$(64 * NumberOfMessagesIndexed)$$ Size your files to match the aging policy for the transaction log data. To remove data from the transaction log, use AMPS actions to remove the journal files that are no longer needed. You can size your files to make this easier. For example, if your application typically generates 100GB a day of transaction log, you could size your files in 25GB units to make it easier to remove 100GB increments. AMPS allows you to preallocate files for the transaction log. For applications that are very latency-sensitive, preallocation can help provide consistent latency. We recommend that those applications preallocate files, if storage capacity and retention policy permit. For example, an application that sees heavy throughput during a working day might preallocate enough files so that there is no need for additional allocation within the working day. Notice that, if your application uses replication, the AMPS transaction log maintenance actions will not delete unreplicated messages that this instance is responsible for replicating. This means that, when calculating the maximum storage space required, the recovery window for a failure is also important. For example, many systems have a policy of not restarting a failed system until a scheduled maintenance window: if one server in a replicated set of servers could, potentially, be offline for up to 8 hours, then the other servers must be able to store a minimum of 8 hours of journals, even in cases where the normal retention period would be shorter. **Try it yourself:** Experiment with the formula to see Transaction Log storage needs. ### File-Backed Queue Metadata AMPS can, optionally, persist queue metadata to the filesystem to allow metadata to be paged out of memory and potentially improve recovery time. For any queue that uses the `FileBackedMetadata` option, the following formula can be used to estimate the storage space that AMPS may use for each queue: $$MAX(\; MaxMsgCount * 250\; \text{bytes}, 4\; \text{MB}\; )$$ where: * $$MaxMsgCount$$ = Maximum number of messages active in the queue The maximum number of messages active is the largest number of unacknowledged messages in the queue since the instance was started, as measured by the maximum value of the `queue_depth` metric for the queue. Notice that if the queue also uses the `TargetQueueDepth` option, the active message count will typically be the `TargetQueueDepth` unless AMPS has temporarily expanded this depth to avoid halting queue delivery. AMPS preallocates files of approximately 4MB for the metadata cache, then grows the file if needed to maintain metadata. The size of the file does not shrink while the instance is running. AMPS may reduce the size of the file during recovery. As with all capacity estimates, this formula is intended to provide a working approximation of the amount of disk space needed, and does not mean that the file will be precisely the size the formula indicates. ### Choosing Storage Devices The previous sections discuss the scope of sizing the storage, however scenarios exist where the performance of the storage devices must also be taken into consideration. In cases where messages are persisted (to the transaction log, to a topic in the SOW, or both), overall throughput of the instance can be limited by the performance of the storage device. It is important that the storage device be able to keep up with the peak rate at which the instance will receive messages. Different aspects of the AMPS server have different patterns of access to storage, as shown below: | Feature | Storage Usage | | ------------------------ | ----------------------------------------------------------------------------------------------------------------- | | State of the World (SOW) | Random access read/write/update. | | Transaction Log | Sequential write for recording messages / sequential read for replay, replication and queue message distribution. | | Error and Event Log | Sequential write only. | | Statistics Database | Random access read/write/update. | | Persisted Queue Metadata | Random access read/write/update. | For each of these uses, ensure that the underlying system has enough I/O bandwidth to meet the needs of an instance. For example, publishing to a topic that is in the SOW and also recorded in the transaction log will write the message to both the SOW and the transaction log, and may (depending on logging settings) also generate a write to the error and event log. Consider a case where an instance is recording messages in the transaction log at a high incoming message rate. If performance greater than 50MB/second is required for the AMPS transaction log, experience has demonstrated that flash storage (or better) would be recommended. Magnetic hard disks lack the performance to produce results greater than this with a consistent latency profile. :::tip For applications that require high performance and persist state, 60East recommends separating message data and information about AMPS itself. That is, it is often helpful to have the statistics database and error and event log on one partition or device, and the SOW, transaction log and persistent queue metadata (if configured) on other partitions or devices. ::: ## CPU SOW queries with content filtering make heavy use of CPU-based operations and, as such, CPU performance directly impacts the content filtering performance and rates at which AMPS processes messages. The number of cores within a CPU largely determines how quickly SOW queries execute. AMPS contains optimizations which are only enabled on recent 64-bit x86 CPUs. To achieve the highest level performance, consider deploying on a CPU which includes support for the SSE 4.2 instruction set. To give an idea of AMPS performance, repeated testing has demonstrated that a moderate query filter with 5 predicates can be executed against 1KB messages at more than 1,000,000 messages per second, per core on an Intel i7 3GHz CPU. This applies to both subscription based content filtering and SOW queries. Actual messaging rates will vary based on matching ratios and network utilization. ## Network When capacity planning a network for AMPS, the requirements for messaging traffic are largely dependent on the following factors: * Average message size * The rate at which publishers will publish messages to AMPS * The number of publishers and the number of subscribers AMPS requires sufficient network capacity to service inbound publishing as well as outbound messaging requirements. In most deployments, outbound messaging to subscribers and query clients has the highest bandwidth requirements due to the increased likeliness for a “one to many” relationship of a single published message matching subscriptions/queries for many clients. Estimating network capacity requires knowledge about several factors, including but not limited to: the average message size published to the AMPS instance, the number of messages published per second, the average expected match ratio per subscription, the number of subscriptions, and the background query load. Once these key metrics are known, then the necessary network capacity can be calculated: $$R * ( Sz + 128 ) ( 1 + M * Sb ) + Q$$ where: * $$R$$ = Rate * $$Sz$$ = Average Message Size * `128` = Estimate of metadata size (including subscription IDs, bookmark and timestamp information if provided, and so on) * $$M$$ = Match Ratio * $$Sb$$ = Number of Subscribers * $$Q$$ = Query Load where “Query Load” is defined as: $$Mq * S * Qs$$ where: * $$Mq$$ = Messages per Query * $$S$$ = Average Message Size * $$Qs$$ = Queries per Second In a deployment required to process published messages at a rate of 5000 messages per second, with each message having an average message size of 600 bytes, the expected match rate per subscription is 2% (or 0.02) with 100 subscriptions. The deployment is also expected to process 5 queries per 1 minute (or 12 queries per second), with each query expected to return 1000 messages. $$5000 * 600 B * ( 1 + 0.02 * 100 ) + ( 1000 * 600 B * 1/12 ) = \; approx \; 9 MB / s = \; approx \; 72 Mb / s$$ Based on these requirements, this deployment would need at least 72Mb/s of network capacity to achieve the desired goals. This analysis demonstrates AMPS by itself would fall into a 100Mb/s class network. It is important to note, this analysis does not examine any other network based activity which may exist on the host and as such, a larger capacity networking infrastructure than 100Mb/s would likely be required. **Try it yourself:** Experiment with the formula to see network capacity requirements. ### Replication Network Bandwidth For replication connections, the general recommendation is to estimate bandwidth needs as though each outgoing replication destination is a subscriber that subscribes to all of the replicated topics, and each incoming replication stream is a publisher that fully publishes the replicated topics. Although AMPS replication connections support compression, the general recommendation is to provision enough network capacity to support the full replication stream, and then to use compression to save capacity. ### Additional Network Considerations When calculating the available bandwidth for an instance, it is also important to take into account any other use of the network. For example, if the system uses network attached storage, and the traffic to that storage is not isolated from messaging traffic, bandwidth to the storage device should also be taken into account when planning the network capacity available to the instance. Likewise, any other process that consumes bandwidth, such as monitoring applications or log collection processes, should be considered when planning the overall bandwidth capacity available. ## NUMA Considerations AMPS is designed to take advantage of non-uniform memory access (NUMA). For the lowest latency in networking, we recommend that you install your NIC in the slot closest to NUMA node 0. When AMPS NUMA tuning is enabled, AMPS runs critical threads on node 0, so positioning the NIC closest to that node provides the shortest path from processor to NIC. When a single instance of AMPS is deployed on the system (physical host), as is the case with most critical production systems, 60East recommends leaving AMPS NUMA tuning enabled (this is the default). If more than one instance of AMPS is running on the same physical host, or if other CPU-intensive processes are running on the same physical host, 60East recommends disabling AMPS NUMA tuning in the AMPS configuration file and relying on the operating system NUMA management. Likewise, if a mechanism is used to restrict AMPS to specific processors, AMPS NUMA tuning should be disabled. :::tip When AMPS is deployed on a virtual machine, 60East recommends disabling the AMPS level NUMA tuning in the configuration file. When AMPS is deployed in a container on a NUMA host, follow the guidance in [Host Guidance](host-guidance#containers). AMPS-level NUMA tuning should remain enabled in a container only when every condition in that section is true, including workload isolation, visible host NUMA topology, compatible runtime CPU and memory constraints, and performance testing of the complete container and host environment. ::: --- # Host Guidance AMPS can run on physical hardware, virtual machines, or containers. The right hosting model depends on the performance, latency, and deployment flexibility needed by the application. Use the [Capacity Planning](capacity-planning) section to size memory, storage, CPU, and network resources for an AMPS deployment. Use this section to plan how those resources are provided by the host environment. ## Physical Servers, Virtual Machines, Containers Although AMPS is designed to be highly adaptive to hardware on which it runs, AMPS does not require a dedicated physical server. AMPS can be successfully deployed on physical hardware, virtual machines, or containers. In any deployment model, 60East recommends [tuning Linux for best performance](linux-configuration) rather than accepting the distribution defaults, which are typically tuned for interactive use rather than for a high performance server. Typically, installations that require the highest level of performance and lowest levels of latency deploy on physical hardware, with a single AMPS instance per server. Installations that are willing to trade predictable performance for ease and flexibility of deployment often use virtual machines or containers. Virtual machines and containers both add a layer between AMPS and the physical host resources. Plan capacity, monitoring, and performance testing for the underlying host as well as for the virtual machine or container that runs AMPS. ### Physical Servers For deployments that require the highest throughput, most consistent latency, or the greatest isolation from other workloads, 60East recommends running one AMPS instance on a dedicated physical server. Choose the server configuration for the expected workload. A processor with the largest available core count is not necessarily the best processor for every AMPS deployment. | Primary workload | Hardware characteristics to prioritize | | --- | --- | | SOW queries, views, joins, aggregations, and content filtering | Strong sustained per-core performance, high memory bandwidth, and enough physical cores for the expected concurrent workload | | High-rate publishing, replication, queue delivery, and transaction log replay | More physical cores, sufficient network bandwidth, and high sustained sequential storage performance | | Large SOW topics, views, indexes, or large query result sets | Memory capacity, memory bandwidth, and fast random-access storage | | Low and predictable latency | Dedicated resources, strong sustained CPU frequency, fully populated memory channels, local low-latency storage, network interfaces capable of [kernel bypass](./open-onload.md), and performance-oriented [Linux tuning](./linux-configuration.md) | #### CPU Selection Consider both physical core count and sustained clock frequency when selecting a CPU. Higher core counts help AMPS process independent work concurrently, including simultaneous queries, publishers, subscribers, replication connections, and transaction log readers. However, increasing core count often reduces the processor's base or sustained all-core frequency. A high-core-count processor with a relatively low sustained frequency may perform well as a publishing, replication, or replay server while providing less desirable response times for CPU-intensive queries, views, and aggregations. Do not make sizing decisions using maximum turbo frequency alone. Compare the sustained frequency the processor can maintain while the expected number of cores is active. Validate the processor using the actual server power, cooling, firmware, and BIOS configuration that will be used in production. When comparing different CPU SKUs of the same generation, the model with the highest CPU core count to frequency product ($\text{CPUCount} \times \text{Frequency}$) typically offers a good initial comparison metric, subject to workload validation. 60East recommends enabling simultaneous multithreading, such as Intel Hyper-Threading or AMD SMT, when available. AMPS benefits from SMT because many workloads alternate between CPU execution and waiting for memory, storage, or networking. Logical CPUs can provide additional throughput, but they do not provide the same capacity as an equal number of physical cores. Base capacity planning on physical cores and treat SMT capacity as additional headroom. For multi-socket systems, install CPUs and memory symmetrically. Leave AMPS NUMA tuning enabled when the server is dedicated to one AMPS instance and performance testing confirms the expected topology and performance. #### Memory Selection First calculate the required memory capacity using the [Capacity Planning](./capacity-planning) guidance. Include SOW topics, views, indexes, journal files, queue state, client message buffers, the operating system, monitoring and security software, and expected growth. When selecting the physical configuration: - Populate all available memory channels (in a vendor-approved configuration) before increasing DIMM capacity within only a subset of channels. - Install matching DIMMs symmetrically across sockets and memory channels. - Compare memory speed and aggregate memory bandwidth, not only total capacity. - As a practical minimum for a balanced physical server, provide at least 4GB of RAM per physical core. This is only a hardware configuration floor. Use the workload-derived estimate whenever it requires more memory. - Preserve the production headroom recommended in the Capacity Planning section. AMPS should not depend on swapping to support its normal workload. Memory bandwidth can become limiting before either CPU utilization or total memory capacity is exhausted. Avoid pairing a large number of CPU cores with a memory configuration that does not populate the processor's available memory channels. :::note With current server-memory pricing, reducing installed memory capacity can materially lower the cost of a physical server. When reducing capacity, continue to populate every memory channel supported by each installed CPU, following the server manufacturer's population guidelines. Prefer smaller DIMMs distributed across all channels over fewer large DIMMs that leave channels unused. A server may have sufficient memory capacity but substantially reduced memory bandwidth when channels are not fully populated. Do not reduce installed memory below the workload-derived capacity and headroom requirements described in the Capacity Planning section. ::: #### Storage Selection Storage should be sized for both capacity and performance. Capacity depends on SOW size, transaction log retention, queue state, logging, statistics, and the amount of data that must be retained while another replicated instance is unavailable. Performance requirements depend on how AMPS uses each device: - Transaction logs primarily require predictable sequential write performance and low write latency. Replay, replication, and queue delivery also require sequential read performance. - SOW topics and persisted queue metadata perform random reads, writes, and updates. - Error logs and the statistics database can create additional I/O that competes with message persistence (depending on configuration). For high-performance deployments that persist messages, direct-attached enterprise NVMe storage is generally the best starting point. Where storage activity is substantial, place the transaction log and SOW data on separate devices or storage resources so that random SOW activity cannot interfere with transaction log writes. Place error logs and the statistics database away from latency-sensitive message storage when practical. Do not select storage based only on an advertised maximum throughput or IOPS rating. Long or inconsistent write latencies can stall the persisted message path even when average throughput appears sufficient. Shared, network-attached, or remote storage should be tested under the competing load and failure conditions expected in production. Run [`amps_bio_perf_test`](/docs/amps-user-guide/utilities/amps_bio_perf) against the actual filesystem and device configuration planned for the transaction log on a server with no other usage during the test. Evaluate sustained throughput and the distribution of write latency, including high-percentile and maximum latency. Since this utility models transaction log access, also run a representative AMPS workload to validate SOW and mixed-I/O performance. Choose device redundancy and RAID configuration according to the durability and recovery requirements of the deployment. Striping can increase throughput but does not replace AMPS replication, backups, or an appropriate recovery plan. #### Validate the Complete Server Before selecting a production configuration, test the complete workload on representative hardware. Include expected peak and burst rates, concurrent queries, subscriptions, SOW updates, transaction logging, replication, queue backlog recovery, and the monitoring and security software that will run in production. The most useful comparison is not between processor or storage specifications. It is between complete server configurations running the expected AMPS workload with sufficient headroom to meet the deployment's service-level requirements. ### Virtual Machines When deploying in a virtual machine, it is important to consider the capacity of _both_ the virtual machine itself and the underlying host hardware. In other words, the total memory needed by all virtual machines -- with all applications hosted by those machines running at peak traffic simultaneously -- should not exceed the physical memory of the hardware. Likewise, the total number of CPUs specified in all of the virtual machines on the host should not exceed the number of CPUs on the host hardware, the network bandwidth needed should not exceed the bandwidth allocated to the host, the traffic to the storage device should not exceed the throughput that the storage device is capable of, and so on. In an enterprise environment, it is not unusual for a wide variety of applications to all see peak loads at the same time, so the system should be provisioned to provide enough capacity that every hosted application can meet peak throughput requirements at the same time. For x86-64 cloud deployments, the best starting point is a balanced general-purpose instance. | Cloud provider | Recommended starting point | Alternatives and notes | | --- | --- | --- | | [AWS EC2](https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html) | M-family x86-64 instances, such as `M8i` or `M8a` where available. | Use `M7i` or `M7a` where M8 instances are not available. | | [Google Cloud](https://cloud.google.com/compute/docs/general-purpose-machines) | General-purpose `C4 standard` machines. | Use `N4 standard` or `N2 standard` based on regional availability and storage or network requirements. | | [Azure](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/d-family) | D-family x86-64 instances, such as `Dsv7` or `Ddsv7` where available. | Use `Dsv5` or `Dv5` where Dsv7 or Ddsv7 instances are not available. | Choose the initial VM size from the CPU, memory, network, and storage requirements identified during capacity planning. Start with a balanced instance type, then use workload testing to decide whether the AMPS use case benefits from moving to a more specialized instance family. :::tip Use workload testing to choose the next direction: expand toward compute-optimized instances when CPU is the limiting resource, memory-optimized instances when memory capacity or memory bandwidth is limiting, network-optimized instances when message throughput or replication traffic is limited by networking, and storage-optimized instances when persistence, transaction log, or SOW activity is limited by storage throughput or latency. ::: On virtual machines, disable AMPS-level NUMA tuning in the AMPS configuration file. ```xml showLineNumbers ... disabled ... ``` For applications requiring low latency and predictable response times, 60East does not recommend using virtualization systems that dynamically move running virtual machines for load-balancing purposes. Although these systems work well for their intended purpose, a machine migration will cause a period where AMPS isn't delivering messages or responding to queries, while also having a gap in observability (Galvanometer, Admin statistics, or logging). ### Containers When deploying in a container, ensure that the host system has the capacity to support all of the containers running on the host at peak capacity. This capacity includes CPU, memory, networking, storage capacity, and storage throughput. 60East tests and certifies AMPS within Podman containers. Containers can provide density and compute-efficiency advantages by allowing multiple AMPS instances or related services to share host resources, provided that the host is sized and managed for the combined peak load of all containers. AMPS can perform NUMA tuning within a container when the physical host is a NUMA system and the container has access to the relevant host topology. This section provides guidance for AMPS-level NUMA tuning in containers. Leave AMPS-level NUMA tuning enabled in a container only when all of the following are true: * The AMPS container is the only significant CPU-intensive workload on the host. This is typically a dedicated host, but a host that also runs lightweight supporting containers or processes can qualify when they do not materially compete for CPU or memory locality. * The host NUMA topology is visible inside the container. * The container runtime or orchestrator is not using CPU pinning, CPU quotas, memory limits, or memory placement policies that conflict with the topology AMPS detects. * Performance testing of the complete container and host environment shows that AMPS NUMA tuning improves or preserves the required performance for the workload. Disable AMPS-level NUMA tuning when any of these conditions is not true. This includes hosts that run multiple significant AMPS instances, containers, tenants, or CPU-intensive applications, and containers where the runtime restricts CPU or memory placement. In these deployments, the host and container runtime are responsible for allocating CPU and memory resources across workloads. Before leaving AMPS-level NUMA tuning enabled in a container, verify the topology visible from inside the container, review the runtime CPU and memory constraints, and compare workload performance with AMPS NUMA tuning enabled and disabled. AMPS supports a wide variety of use cases with different CPU, networking, storage, and memory profiles. For high-performance computing (HPC) use cases, test the complete container and host environment for acceptable performance for each AMPS workload, rather than assuming that results from one workload or host configuration apply to another. For best networking performance in a container, 60East recommends using host networking to remove network address translation (NAT) and virtual bridge interfaces from the message path. :::warning Host networking makes AMPS listeners use the host network namespace. Use host networking only with intended bind addresses configured for [Transports](../transports) and the [Admin interface](../monitoring/monitor-configuration), host firewalls or cloud security groups limiting access to the required peers, and [Securing AMPS](../securing) controls such as TLS, authentication, and entitlement configured before the container is exposed. ::: ### Monitoring and Virtualization Considerations For virtualized hosting or deployment in a container, develop a plan for monitoring the physical hardware as well as the environment that runs AMPS. If possible, the monitoring plan should include a method for correlating the activity in a virtualized or containerized environment with the activity on the physical host. For example, it is important to be able to correlate CPU saturation on a virtual machine to CPU saturation on the physical host. --- # Linux OS Settings ## Linux Operating System Configuration This section covers some settings which are specific to running AMPS on a Linux Operating System. ## ulimit --- The `ulimit` command is used by a Linux administrator to get and set user limits on various system resources. `ulimit -c` It is common for an AMPS instance to be configured to consume gigabytes of memory for large SOW caches. If a failure were to occur in a large deployment it could take seconds (maybe even hours, depending on storage performance and process size) to dump the core file. AMPS has a minidump reporting mechanism built in that collects information important to debugging an instance before exiting. This minidump is much faster than dumping a core file to disk. For this reason, it is recommended that the per user core file size limit is set to 0 to prevent a large process image from being dumped to storage. `ulimit -n` The number of file descriptors allowed for a user running AMPS needs to be at least double the sum of counts for the following: connected clients, SOW topics and pre-allocated journal files. * _Minimum:_ 4096 * _Recommended:_ 32768, or the value recommended by AMPS in any diagnostic messages, whichever is greater ## Transparent Huge Pages --- Transparent huge pages is enabled by default for most linux distributions, which can add significant overhead to memory management. For this reason 60East recommends changing the setting to `madvise`, which requires applications to explicitly request use of transparent huge pages. Previously the recommendation was to use `never`, which is still acceptable, but is less flexible as it disables transparent huge pages for all applications running on the system. To change the setting until the operating system is rebooted, the following command can be used: ```bash echo madvise > /sys/kernel/mm/transparent_hugepage/enabled ``` To make a permanent change to this setting, add the above command to the startup scripts, or add the `transparent_hugepage=madvise` option to the kernel startup flags (see the documentation for your Linux distribution for details). * _Recommended: madvise_ ## /proc/sys/fs/aio-max-nr --- Each AMPS instance requires AIO in the kernel to support at least 16384 plus 8192 for each SOW topic in simultaneous I/O operations. The setting `aio-max-nr` is global to the host and impacts all applications. As such this value needs to be set high enough to service all applications using AIO on the host. * _Minimum:_ 65536 * _Recommended:_ 1048576 To view the value of this setting, as root you can enter the following command: ```bash cat /proc/sys/fs/aio-max-nr ``` To edit this value, as root you can enter the following command: ```bash sysctl -w fs.aio-max-nr=1048576 ``` This command will update the value for `/proc/sys/fs/aio-max-nr` and allow 1,048,576 simultaneous I/O operations, but will only do so until the next time the machine is rebooted. To make a permanent change to this setting, as a root user, edit the `/etc/sysctl.conf` file and either edit or append the following setting: ```bash fs.aio-max-nr = 1048576 ``` ## /proc/sys/fs/file-max --- Each AMPS instance needs file descriptors to service connections and maintain file handles for open files. This number needs to be at least double the sum of counts for the following: connected clients, SOW topics and pre-allocated journal files. This file-max setting is global to the host and impacts all applications, so this needs to be set high enough to service all applications on the host. * _Minimum:_ 262144 * _Recommended:_ 6815744 To view the value of this setting, as root you can enter the following command: ```bash cat /proc/sys/fs/file-max ``` To edit this value, as root you can enter the following command: ```bash sysctl -w fs.file-max=6815744 ``` This command will update the value for `/proc/sys/fs/file-max` and allow 6,815,744 concurrent files to be opened, but will only do so until the next time the machine is rebooted. To make a permanent change to this setting, as a root user, edit the `/etc/sysctl.conf` file and either edit or append the following setting: ```bash fs.file-max = 6815744 ``` ## /proc/sys/vm/min\_free\_kbytes --- This parameter sets the minimum amount of memory to keep free in the system. Setting this value properly can help the operating system function more effectively in low-memory situations. If this value is set too low, the operating system can have difficulty reclaiming memory, which can lead to unnecessary out-of-memory events. If this value is set too high, overall system efficiency decreases as the operating system can spend more time than necessary reclaiming memory. 60East recommends setting this parameter to 1% of the physical memory on the system, rounding up to the nearest GB. Notice that the units of this parameter are in kilobytes. For example, to set this value for a system that has 128GB of memory, you would calculate 1% of the physical memory (1.28 GB), round up to the nearest GB (2 GB) and then allocate 2000000 KB as the min\_free\_kbytes. * _Minimum:_ 1000000 (1GB) * _Recommended:_ 1% of physical memory, rounded up to the nearest GB To edit this value, as root you can enter the following command: ```bash sysctl -w vm.min_free_kbytes=2000000 ``` Notice that this tuning recommendation is designed for a server-class machine with a reasonable amount of memory. For a small development machine or blade (for example, a system with less than 32GB of memory), leaving this parameter at the operating system default may be more appropriate. ## /proc/sys/vm/max\_map\_count --- AMPS makes extensive use of memory mapped files, and frequently modifies the maps. The `/proc/sys/vm/max_map_count` parameter sets the maximum number of maps that the Linux kernel will allow for a process. If the number of requested maps exceeds the number of maps in this parameter, memory allocation operations can fail even when there is sufficient memory available. This setting is global to the host and applies to all applications, so this needs to be set high enough for the most map-intensive application on the host. * _Minimum:_ 65530 * _Recommended:_ 500000 To edit this value, as root you can enter the following command: ```bash sysctl -w vm.max_map_count=500000 ``` This command will update the value for `/proc/sys/vm/max_map_count` and allow 500,000 maps to be created, but will only do so until the next time the machine is rebooted. To make a permanent change to this setting, as a root user, edit the `/etc/sysctl.conf` file and either edit or append the following setting: ```bash vm.max_map_count=500000 ``` ## /proc/sys/net/core/somaxconn --- AMPS can quickly accept new inbound connections, but during a failover event in which hundreds or thousands of clients are trying to connect to an instance this can cause the backlog of unaccepted connections to exceed the host's configured limit and return an `ECONNREFUSED` (Connection Refused) error back to the client. Since this limit is in the host environments networking layer, this means the error occurs before AMPS has a chance to accept the client connection and no logging of the connection attempt will be in the AMPS logging. 60East recommends that, to better absorb large inbound connection spikes, setting the `somaxconn` system parameter to 4096 (in some cases higher if you have many thousands of connections). This setting is global to the host and applies to all applications. * _Recommended:_ 4096 To edit this value, as root you can enter the following command: ```bash sysctl -w net.core.somaxconn=4096 ``` This command will update the value for `/proc/sys/net/core/somaxconn` and increase the backlog of unaccepted connections. Using the command above will change the `somaxconn` setting until the operating system is rebooted. To make a permanent change to this setting, as a root user, edit the `/etc/sysctl.conf` file and either edit or append the following setting: ```bash net.core.somaxconn=4096 ``` ## /proc/sys/vm/swappiness --- AMPS performs best when the data that it needs to retain resident is in memory. If the operating system needs to use swap because the system requires more memory than is available, performance degrades substantially. 60East recommends that, for systems that host performance-critical instances of AMPS, the `vm.swappiness` setting is set to `0`. This will minimize swapping on this system, which will improve performance with the tradeoff of making it more likely for processes to be killed by the operating system in low-memory situations. This does not disable swapping entirely, it only delays it until it's absolutely necessary. This setting is global to the host and applies to all applications. * _Recommended:_ 0 To edit this value, as root you can enter the following command: ```bash sysctl -w vm.swappiness=0 ``` This command will update the value for `/proc/sys/vm/swappiness` and direct the operating system to avoid using swap space until the system is under severe memory pressure. Using the command above will change the swappiness setting until the operating system is rebooted. To make a permanent change to this setting, as a root user, edit the `/etc/sysctl.conf` file and either edit or append the following setting: ```bash vm.swappiness=0 ``` ## /proc/sys/vm/force_cgroup_v2_swappiness --- :::warning 60East recommends this setting **only** for Red Hat Enterprise Linux 8. ::: Only for installations on Red Hat Enterprise Linux 8 (RHEL8), this option helps to enforce `vm.swappiness` system-wide. If on cgroup v1, by default, the system will use a per-cgroup swappiness value. The default cgroup v1 swappiness value is `60`, which is not recommended. When this is set to `1` the system value `vm.swappiness` is used and behaves like RHEL7 or RHEL9. Without this setting, a RHEL8 system using cgroup v1 can prematurely swap and degrade performance substantially. This setting is global to the host and applies to all applications. * _Recommended:_ 1 To edit this value, as root you can enter the following command: ```bash sysctl -w vm.force_cgroup_v2_swappiness=1 ``` Using the command above will change the `vm.force_cgroup_v2_swappiness` setting until the operating system is rebooted. To make a permanent change to this setting, as a root user, edit the `/etc/sysctl.conf` file and either edit or append the following setting: ```bash vm.force_cgroup_v2_swappiness=1 ``` ## /proc/sys/net/ipv4/tcp\_frto --- This option controls whether Forward RTO-Recovery (FRTO) is enabled for the TCP network. Enabling FRTO can be beneficial for overall network performance if a system is _sending_ packets over wireless networks with substantial interference (for example, public WiFi in an urban area). However, this recovery algorithm can reduce performance in wired networks. While this option is enabled on most current Linux distributions by default, disabling the option can improve network performance. 60East recommends disabling this option unless the server is directly delivering traffic over a congested WiFi network. This setting is global to the host and applies to all applications. * _Recommended:_ 0 To edit this value, as root you can enter the following command: ```bash sysctl -w net.ipv4.tcp_frto=0 ``` This command will update the value for `/proc/sys/net/ipv4/tcp_frto` and direct the operating system to disable FRTO. Using the command above will change the setting until the operating system is rebooted. To make a permanent change to this setting, as a root user, edit the `/etc/sysctl.conf` file and either edit or append the following setting: ```bash net.ipv4.tcp_frto=0 ``` --- # OpenOnload OpenOnload can reduce network latency for AMPS deployments that use supported Solarflare or Xilinx network adapters. OpenOnload provides a user-space network stack for selected sockets, which can reduce operating system overhead in latency-sensitive deployments. Use OpenOnload when the application requires the lowest practical TCP latency and the host network hardware, driver, operating system, and operational model support OpenOnload. For most deployments, start with the general [Host Guidance](host-guidance) and [Linux OS Settings](linux-configuration) recommendations, then test OpenOnload with the complete AMPS workload before deploying it in production. ## Runtime and Driver Compatibility Keep the OpenOnload user-space libraries compatible with the OpenOnload driver on the host. Avoid mixing a container image library with a different host driver version unless that combination is explicitly tested and supported by the operations team. Visit the [Onload user guide](https://docs.amd.com/r/en-US/ug1586-onload-user) to learn more about Onload installation and configuration. ## AMPS Configuration AMPS uses OpenOnload when the AMPS process is started with the OpenOnload runtime active. In that environment, AMPS detects OpenOnload and uses OpenOnload stacks for eligible TCP sockets. The AMPS configuration file can include the `OpenOnload` transport setting when the configuration should explicitly show that a transport is intended to run with OpenOnload. This setting is optional and does not load OpenOnload by itself. ```xml showLineNumbers ... amps-tcp tcp nic-ip:9007 amps enabled ... ``` :::note The setting above does not load OpenOnload by itself. The AMPS process must still be started with the OpenOnload runtime active, as shown in the next section. ::: ## Starting AMPS with OpenOnload On a physical host, the `onload` command is the most direct way to start AMPS with OpenOnload. The command configures the process environment and preloads the OpenOnload library before starting AMPS. ```bash showLineNumbers onload ampServer config.xml ``` Setting `LD_PRELOAD` directly also loads the OpenOnload runtime when the deployment does not use the `onload` wrapper. ```bash showLineNumbers LD_PRELOAD=libonload.so ./ampServer config.xml ``` Optionally, an OpenOnload profile can be provided to tune behavior. The following settings can be applied by supplying `--profile=` with the `onload` command. ```bash showLineNumbers onload --profile=/path/to/amps.opf ampServer config.xml ``` The following is the AMPS low-latency Onload profile that can be used as a starting point. Save the contents to `amps.opf`, test the settings with the complete workload, and pass the profile to `onload`. ```bash showLineNumbers # SPDX-License-Identifier: BSD-2-Clause # X-SPDX-Copyright-Text: (c) Solarflare Communications Inc # 60East AMPS Low Latency Profile # Enable CTPIO even if using full-featured firmware. N.B. This is # incompatible with multicast loopback. onload_set EF_CTPIO_SWITCH_BYPASS 1 # Assume that application does not invoke calls on a single epoll object # concurrently. onload_set EF_EPOLL_MT_SAFE 1 # Enable polling / spinning. When the application makes a blocking call # such as recv() or poll(), this causes Onload to busy wait for up to 100ms # before blocking. onload_set EF_POLL_USEC 100000 # Disable FASTSTART when connection is new or has been idle for a while. # The additional acks it causes add latency on the receive path. onload_set EF_TCP_FASTSTART_INIT 0 onload_set EF_TCP_FASTSTART_IDLE 0 # Use a large initial congestion window so that the slow-start algorithm # doesn't cause delays. We don't enable this by default because it breaks # the TCP specs, and could cause congestion in your network. Uncomment if # you think you need this. # onload_set EF_TCP_INITIAL_CWND 1048576 # When TCP_NODELAY is used, always kick packets out immediately. This is # not enabled by default because most apps benefit from the default # behaviour. onload_set EF_NONAGLE_INFLIGHT_MAX 65535 ``` Alternatively, set the same OpenOnload options as environment variables in the AMPS process environment before starting `ampServer`: ```bash showLineNumbers export LD_PRELOAD=libonload.so export EF_CTPIO_SWITCH_BYPASS=1 export EF_EPOLL_MT_SAFE=1 export EF_POLL_USEC=100000 export EF_TCP_FASTSTART_INIT=0 export EF_TCP_FASTSTART_IDLE=0 # export EF_TCP_INITIAL_CWND=1048576 # optional export EF_NONAGLE_INFLIGHT_MAX=65535 ``` After setting the variables, start AMPS normally: ```bash showLineNumbers ./ampServer config.xml ``` ## Monitoring At AMPS startup, confirm that the runtime environment is active. AMPS logs environment variables such as `LD_PRELOAD`, and this can be used to verify that the process started with `libonload.so` loaded. ```text The environment variable 'LD_PRELOAD' is set to 'libonload.so' ``` Use OpenOnload tools such as `onload_stackdump` to confirm that AMPS has OpenOnload stacks and to inspect the OpenOnload environment used by the process. ```bash showLineNumbers onload_stackdump ``` Example output with AMPS running under Onload: ```text #stack-id stack-name pids 1 AMPS:DEFAULT-p1 1 ``` The absence of AMPS stacks could indicate could indicate that AMPS did not start under Onload. As AMPS clients connect, run `onload_stackdump` again and confirm that new stack names beginning with `AMPS:` appear. ```text #stack-id stack-name pids 19 AMPS:0-t230 1 1 AMPS:DEFAULT-p1 1 ``` For OpenOnload deployments, include the following in the production monitoring plan: * OpenOnload runtime verification at process startup. * NIC counters, packet drops, retransmits, and link status. * End-to-end publish and subscribe latency from the application perspective. For general AMPS, host, container, and VM monitoring guidance, see [Monitoring and Virtualization Considerations](host-guidance#monitoring-and-virtualization-considerations). ## Virtual Machines OpenOnload is primarily a physical-host acceleration technology. In a virtual machine, OpenOnload is only appropriate when the VM has the required access to supported network hardware, such as through a hardware passthrough configuration that exposes the NIC and required driver behavior to the guest. Do not assume that OpenOnload improves an AMPS deployment just because the physical host has supported hardware. Test the exact VM type, host configuration, NIC exposure model, CPU allocation, and AMPS workload. Validate Onload using `onload_stackdump` to confirm AMPS is running with the accelerated path. If the VM cannot use the accelerated path, follow the standard [virtual machine guidance](host-guidance#virtual-machines) and run AMPS with the normal TCP stack. ## Containers OpenOnload can be used from a container when the container has access to the OpenOnload runtime, the host network path, and the required device nodes. The [Onload container guidance](https://docs.amd.com/r/en-US/ug1586-onload-user/Onload-in-a-Docker-Container) uses Docker; the example below shows the same general requirements using Podman. Adjust the image, paths, and runtime options for the deployment. ```bash showLineNumbers podman run -it \ --name amps \ --network host \ --device=/dev/onload \ --device=/dev/onload_epoll \ -v /path/to/amps:/amps \ -v /usr/lib64/libonload.so:/usr/lib64/libonload.so \ -w /amps/bin \ -e LD_PRELOAD=libonload.so \ -e EF_CTPIO_SWITCH_BYPASS=1 \ -e EF_EPOLL_MT_SAFE=1 \ -e EF_POLL_USEC=100000 \ -e EF_TCP_FASTSTART_INIT=0 \ -e EF_TCP_FASTSTART_IDLE=0 \ -e EF_NONAGLE_INFLIGHT_MAX=65535 \ fedora:latest \ ./ampServer config.xml ``` The example mounts the host OpenOnload shared library at `/usr/lib64/libonload.so` into the same path in the container. Another valid approach is to build the matching OpenOnload user-space components into the container image, provided that the image contents are kept compatible with the host driver. Prefer explicit device mappings when the container runtime and host policy allow them. Some environments require `--privileged` for OpenOnload because device, capability, or namespace access is otherwise incomplete. Use `--privileged` only when the security model for the host and container has been reviewed. :::warning Host networking and OpenOnload device access give the AMPS container direct access to host networking resources. Review the host networking security guidance in [Host Guidance](host-guidance#containers) before exposing the container to production networks. ::: --- # Operations Best Practices This section covers a selection of best practices for deploying AMPS. ### Monitoring AMPS exposes the statistics available for monitoring via a RESTful interface, described in the [Monitoring AMPS](/docs/amps-user-guide/monitoring) section and the [AMPS Monitoring Guide](/docs/amps-monitoring-guide/). The interface is available through the address specified in the `Admin` section of the configuration. This interface allows developers, administrators, and monitoring tools to easily inspect various aspects of AMPS performance and resource consumption using standard monitoring tools. At times, AMPS will emit log messages notifying that a thread has encountered a deadlock or stressful operation. These messages will repeat with the word “stuck” in them. AMPS will attempt to resolve these issues, however after 60 seconds of a single thread being stuck, AMPS will automatically emit a minidump to the previously configured minidump directory. This minidump can be used by 60East support to assist in troubleshooting the location of the stuck thread or the stressful process. Monitor the contents of `dmesg` on the instance for errors that affect the AMPS process. For example, if the operating system runs low on memory and begins shutting down processes, this information will be recorded in `dmesg`. Likewise, system events such as hardware failures that can affect AMPS are most likely to be recorded in the `dmesg` output. Another area to examine when monitoring AMPS is the `last_active` monitor for the processors. This can be found in the `/amps/instance/processors/all/last_active` url in the monitoring interface. If the `last_active` value continually increases for more than one minute and there is a noticeable decline in the quality of service, then it may be best to fail-over and restart the AMPS instance. ### Logging 60East recommends that an instance of AMPS used for production log at `info` level (at a minimum). This provides a basic record of the operations requested in AMPS, and is the minimum level of logging needed to troubleshoot most issues. Further, a production instance should have the capacity available to log at a more verbose level if necessary, for troubleshooting and diagnostic purposes. An instance used for development or UAT purposes should typically log at `trace` level so that the interaction between an application and AMPS is captured. 60East also recommends capturing stdout and stderr for the AMPS process. This can provide information about operating system or runtime errors in the event that a problem occurs outside of the control of AMPS (and, therefore, cannot be recorded in the AMPS event log). ### Stopping AMPS To stop AMPS, ensure that AMPS runs the `amps-action-do-shutdown` action. By default, this action is run when AMPS receives `SIGHUP`, `SIGINT`, or `SIGTERM`. However, you can also configure an action to shut down AMPS in response to other conditions. For example, if your company policy is to reboot servers every Saturday night, and AMPS is not running as a system service (or daemon), you could schedule an AMPS shutdown every Saturday before the system reboot. When AMPS is installed to run as a system service (or daemon), AMPS installs shutdown scripts that will cleanly stop AMPS during a system shutdown or reboot. ### SOW Parameters Choosing the ideal `SlabSize` for your SOW topic is a balance between the frequency of SOW expansion and storage space efficiency. A large `SlabSize` will preallocate space for records when AMPS begins writing to the SOW. If detailed tuning is not necessary, 60East recommends leaving the `SlabSize` at the default size if your messages are smaller than the default `SlabSize`. If your messages are larger than the default SlabSize, a good starting point for the `SlabSize` is to set it to several times the maximum message size you expect to store in the SOW. There are three considerations when setting the optimum `SlabSize`: 1. Frequency of allocations 2. Overall size of the SOW 3. Efficient use of space A `SlabSize` that is small results in frequent extensions of your SOW topic to occur. These frequent extensions can reduce throughput in a heavily loaded system, and in extreme cases can exhaust the kernel limit on the number of regions that a process can map. Increasing the `SlabSize` will reduce the number of allocations. When the `SlabSize` is large, then the risk of the SOW resize affecting performance is reduced. Since each slab is larger, however, there will be more space consumed if you are only storing a small number of messages: this cost will amortize as the number of messages in the SOW exceeds the _number of cores in the system_ \* _the number of messages that fit into a slab_. To most efficiently use space, set a `SlabSize` that minimizes the amount of unused space in a slab. For example, if your message sizes are average 512 bytes but can reach a maximum of 1.2 MB, one approach would be to set a `SlabSize` of 2.5MB to hold approximately 5 average-sized messages and two of the larger-sized messages. Looking at the actual distribution of message sizes in the SOW (which can be done with the `amps_sow_dump` utility) can help you determine how best to size slabs for maximum space efficiency. For optimizing the `SlabSize`, determine how important each aspect of SOW tuning is for your application, and adjust the configuration to balance allocation frequency, overall SOW size, and space to meet the needs of your application. Given AMPS is highly-parallelized, AMPS operates more efficiently when it is able to run tasks in parallel. When considering options for SlabSize, be sure that the value you choose will result in a number of slabs that is at least equal to the number of cores in the system. A SlabSize setting that results in only a few slabs could cause reduced query performance. For example, a system with a single publisher and a SlabSize large enough to hold all of the records produced by that publisher, doesn’t allow a query to be parallelized since all of the records will be in a single slab. ### Slow Clients As described in [Slow Client Management](../ha/slow-client-management-and-capacity-limits), AMPS provides capacity limits for slow clients to reduce the memory resources consumed by slow clients. This section discusses tuning slow client handling to achieve your availability goals. #### Slow Client Offlining for Large Result Sets The default settings for AMPS work well in a wide variety of applications with minimal tuning. If you have particularly large SOW topics and your application is disconnecting clients due to exceeding the offlining threshold when the clients are retrieving large SOW query result sets, 60East recommends the following settings as a baseline for further tuning: | Parameter | Recommendation | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `MessageMemoryLimit` |

    This controls the maximum memory consumed by AMPS for client messages. You can increase this parameter to allow AMPS to use more memory for records. Notice, however, that memory devoted to client messages is unavailable for other purposes.

    Recommended starting point for tuning large result sets: 10% of the system memory (for example, on a server with 128GB of memory, start with a 13GB limit). 60East recommends tuning the `MessageDiskLimit` first. If necessary, increase this parameter by 1-2% at a time.

    Use caution with settings over 20%: devoting large amounts of memory to client messages may cause swapping and reduce, rather than increase, overall performance.

    | | `MessageDiskLimit` |

    The maximum amount of space to consume for offline messages.

    Recommended starting point for tuning large result sets: average record size * number of expected records * number of simultaneous clients, or `MessageMemoryLimit`, whichever is greater.

    | | `MessageDiskPath` |

    The path in which to store offline message files.

    60East recommends that the message disk path be hosted on fast, high-capacity storage such as a PCIe-attached flash drive. The available storage capacity of the disk must be greater than the configured `MessageDiskLimit`.

    Pay attention to the performance characteristics of the device: for example, some devices suffer reduced performance when they run low on free space, so for those devices you would want to make sure that there is space available on the device even when AMPS is close to the `MessageDiskLimit`.

    | 60East recommends that you use these settings as a baseline for further tuning, bearing in mind the needs and expected messaging patterns of your application. #### WAN Traffic and Slow Client Settings In some installations, a single AMPS instance will serve both applications that are local to the instance and applications that retrieve data over a higher-latency network. For example, applications in a small regional office may use a server in another region over a WAN. In these situations, consider either adjusting the slow client settings so that those clients can complete operations such as large SOW queries successfully, or consider creating a separate transport with higher capacity settings that will be used _only_ by the small number of clients that require these settings due to network limitations. In particular, if you set a `ClientMessageAgeLimit` for an instance or transport, ensure that this limit is large enough that the network can consume the results of the SOW queries that clients are expected to make within the allotted time. ### Minidump AMPS includes the ability to generate a minidump file, which can be used by 60East support, to help troubleshoot a problematic instance. The minidump captures thread state information: a snapshot of where in the source code each thread is, the call stack for each thread, and the register information for each frame of the call stack. A minidump also contains basic information about the system that AMPS was running on, such as the processor type and number of sockets. Minidumps _do not_ contain other internal state of AMPS or the contents of application memory. Minidumps _do not_ contain detailed information about the host system, and have no information about the state of the host or operating system. Instead, minidumps identify the point of failure to help 60East quickly narrow down the issue without generating large files or potentially compromising sensitive data. Minidumps can be produced much faster than a standard core dump, and use significantly less space since the minidump contains only a small subset of the information a core dump would contain (see the [ulimit](linux-configuration.md#ulimit) section in [Linux OS Settings](linux-configuration) for more configuration options). Because minidumps are relatively inexpensive, the AMPS server may produce minidumps for temporary conditions that the server subsequently recovers from. AMPS also allows creation of a minidump on demand. Generation of a minidump file occurs in the following ways: 1. When AMPS detects a crash internally, a minidump file will automatically be generated. This includes cases where an AMPS thread or critical internal component has not reported progress for an extended period of time (typically 300 seconds). 2. When a user clicks on the `minidump` link in the `amps/instance/administrator` link from the administrator console (see the _AMPS Monitoring Reference_ for more information). 3. By sending the running AMPS process the `SIGQUIT` signal. 4. In response to a configured action. 5. If a thread fails to report progress with the AMPS thread monitor for approximately 60 seconds, a minidump will automatically be generated. This should be sent to AMPS support for evaluation along with a description of the operations taking place at the time (typically, info level or more verbose logging). By default the minidump is configured to write to `/tmp`, but this can be changed in the AMPS configuration by modifying the `MiniDumpDirectory`. 60East recommends monitoring the minidump directory. If minidumps occur, contact 60East support for diagnosis and troubleshooting. Bear in mind that minidumps are often a symptom of a slowdown in the server due to resource constraints rather than an indication that the server has exited. Once a minidump is submitted to 60East (and acknowledged as received), there is no further need to retain that minidump. 60East recommends removing minidumps when they are no longer needed. ### Deployment and Upgrade Plan 60East offers a [deployment checklist](/docs/deployment-checklist/checklist) for use when planning or upgrading an installation of AMPS. The checklist covers recommendations for operations considerations such as: * Capacity Planning * Operating System Configuration * AMPS Configuration * Developing and Configuring Maintenance Plans * Creating a Monitoring Strategy * Creating a Patching and Upgrade Plan * Creating a Support Plan and Verifying the Support Process The checklist may not cover all aspects of deployment in a particular environment but can be used to create a checklist and deployment plan for your environment. --- # Using AMPS with a Proxy ## Accessing AMPS Through a Proxy For some installations, it is important to be able to configure access to an AMPS instance through a proxy server. AMPS client connections and AMPS replication connections are TCP connections that use a custom protocol to communicate. Any proxy that does not alter the content of packets will work as expected for both client connections and replication connections. Using a proxy with AMPS is most straightforward when there is a one-to-one relationship between a port on the proxy and a port on the AMPS instance. In many cases, though, it is useful to have a single port on the proxy and then to route to a different backend instance of AMPS. For guidance on configuring a proxy for use with AMPS, refer to the [NGINX Proxy Configuration](/docs/amps-integrations/nginx-proxy/intro) or [Apache Proxy Configuration](/docs/amps-integrations/apache-proxy/intro) guides. ### Websocket Connections Current versions of the AMPS JavaScript client support the ability to insert arbitrary paths between the hostname and port and the protocol and message type components of the AMPS connection URI. For example, a URI of `ws://proxyhost:8080/amps-prod-system/amps/json` could be used by the proxy to route to the system designated `amps-prod-system`. ### Galvanometer Connections The Galvanometer relies on the path provided to determine which resource to return. If it's necessary to proxy multiple systems behind a single port, it is typically most useful to have the proxy rewrite the URI when forwarding the request. For example, the proxy could rewrite a request for `http://proxyhost:8085/prod-system/` to `http://amps-prod-system:8085/`. For example, an NGINX proxy configuration could be implemented along the lines of: ```bash showLineNumbers server { listen 8085 default; listen [::]:8085; server_name AMPS; # server context location /amps-prod-system/ { proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://amps-prod-system:8085/; #hostname and port of AMPS host for first AMPS instance } location /amps-dev-system/ { proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://amps-dev-system:8085/; #hostname and port of AMPS host for second AMPS instance } } ``` The `Admin` configuration item allows an instance to advertise a publicly-accessible address for the admin server. This is useful when the internet address used for replication between servers is a private address, for example, in a hosting situation where replication connections between two instances use a private network, while Galvanometer connections use a proxy or public network. In this case, use the `ExternalInetAddr`option in the configuration to specify an address where the admin server can be reached. :::info The `ExternalInetAddr` option does not override the `InetAddr` parameter or change the network addresses that the admin server uses. Instead, it's intended to provide an externally visible address that will reach the `InetAddr`. ::: Galvanometer relies on the hostnames provided by the AMPS configuration to locate replicated instances and retrieve information from their admin interfaces. If Galvanometer does not have access to the replicated instances at the hostname or IP provided in the AMPS configuration file, it will not be able to display information about replicated instances. This does not indicate an issue with replication on those instances, it just means that the Galvanometer view (which is constructed by Galvanometer in the browser) cannot provide the information. The `ExternalInetAddr` parameter is used to provide that information to Galvanometer. ### TCP/SSL Connections AMPS Clients that use `TCP/SSL` support connection through an HTTP proxy when the HTTP Preflight option is provided in the connection string. For example, a URI of `tcp://proxyhost:80/amps-prod-system/amps/json?http_preflight=true` could be used by the proxy to route an HTTP Upgrade Request to the system designated `amps-prod-system`. If `amps-prod-system` responds with a `101 Switching Protocols` message, the client will proceed with a standard TCP handshake. For more information on HTTP Preflight, see the [HTTP Preflight](/docs/amps-user-guide/transports/http-preflight) section in the AMPS User Guide. ### Load-Balancing Considerations Some proxy systems are also intended to implement load-balancing. The approach that a given installation uses to determine how to distribute connections depends on the resource that the load-balancing system is managing, and the needs of the application. For example, an application where all subscribers produce roughly the same amount of traffic could use a simple round-robin load balancing system to distribute network load. On the other hand, a proxy for an application where subscribers execute complex aggregated subscriptions might monitor CPU load or memory consumption on a set of servers that host AMPS and attempt to route a new connection to the server with the lowest current load. Regardless of the approach used, it is important to keep in mind that, for a publisher or queue consumer, the same considerations for the proxy apply as would apply for specifying failover equivalents to a client application directly. In particular, a publisher or a subscriber that processes a queue should not fail over across a replication connection that uses asynchronous acknowledgment (or a connection that has been downgraded to be asynchronous). If this happens, there is a possibility of message loss and/or inconsistent transaction log contents. See the discussion of synchronous and asynchronous acknowledgment ([Sync vs Async Acknowledgment](../replication/configuring-replication.md#downstream-persistence-acknowledgment-sync-vs-async)) in the [Configuring Replication](../replication/configuring-replication) section for more details. --- # Upgrading AMPS ## Upgrading an AMPS Installation This chapter describes how to upgrade an existing installation of AMPS. The steps presented here focus on upgrading the installation itself, and should be the only steps you need for upgrades that change the HOTFIX version number or the FEATURE version number (as described in [AMPS Versioning and Certification](../intro/technical\_support.md#amps-versioning-and-certification)). For changes that update the MAJOR or MINOR version number, AMPS may add features, change file or network formats, or change behavior. For these upgrades, you may need to make changes to the AMPS configuration file or update applications to adapt to new features or changes in behavior. 60East recommends maintaining a test environment that you can use to test upgrades, particularly when an upgrade changes MAJOR or MINOR versions and you are taking advantage of new features or changed behavior. When the AMPS instance participates in replication, you must coordinate the instance upgrades when upgrading across AMPS versions. AMPS supports replication to and from versions 5.2.0.0 and later for the purposes of rolling upgrade. For long-term deployment, 60East recommends that all AMPS instances that replicate to each other have the same MAJOR and MINOR version number, and preferably run the same release of AMPS. ### Upgrade Steps Upgrading an AMPS installation involves the following steps: 1. Stop the running instance. 2. Install the new AMPS binaries. 3. If you are upgrading from an AMPS version prior to 5.0.0.0, upgrade any data files or configuration files that you want to retain. 4. If necessary, update the configuration file for the instance. 5. If necessary, update any applications that will use new features. 6. Restart the service. AMPS supports replication from version 5.2.0.0 and later to this version of AMPS for the purposes of rolling upgrade with no (or minimal) downtime. 60East recommends that production installations of AMPS have the same MAJOR and MINOR version number at a minimum, and preferably run identical versions of AMPS. ### Upgrading AMPS Data Files AMPS may change the format and content of data files when upgrading across versions, as specified by the MAJOR and MINOR version number. This most commonly occurs when new features are added to AMPS that require different or additional information in the persisted files. The HISTORY file for the AMPS release lists when changes have been made that require data file changes. When upgrading to AMPS 5.0 or later, from a release prior to 5.0, you must upgrade the data files. For versions of AMPS 5.0 and later, backward compatibility is maintained and therefore there have been no changes to the data file formats. The AMPS distribution includes the `amps_upgrade` utility to process and upgrade data files. Unless you are upgrading from a version of AMPS prior to 5.0, there is no need to use this utility when upgrading AMPS. ### Downgrading AMPS Data Files The contents of AMPS data files, including SOW topic files, transaction log journals, and the statistics database are not guaranteed to be backward compatible for versions that change the MAJOR, MINOR, or FEATURE version numbers. Even when the file format is compatible, newer versions of AMPS may include new options or use metadata in a way that older versions of AMPS are not aware of. Downgrading an instance of AMPS while preserving data files may produce unexpected or incorrect behavior. --- # Operation and Deployment This chapter contains guidelines and best practices to help plan and prepare an environment, to ensure optimal performance and stability for AMPS deployments. See the [Troubleshooting AMPS](/docs/amps-user-guide/troubleshooting) chapter for information on troubleshooting problems with AMPS, including information on using the utilities that come with AMPS. --- # Chaining Key Generator The AMPS distribution includes a module that can generate a SOW key for a set of chained messages. Message chains are most frequently used in FIX order processing systems to track a set of updates to an original order from a set of systems that use unique local identifiers for the order. As messages arrive, AMPS must update the record for the original order, regardless of whether the identifier on the current message is the original order, or is an order chained to the original order. A message chain allows an application to treat any update to an identifier in the chain as an update to the original message in the chain. The `libamps_id_chaining_key_generator` module supports this by generating the same SOW key for any message in the chain. To use this module, messages must have a field that identifies the current message and a field that identifies the previous message in the message chain, if one exists. ## Chained Message Sample Case Consider a message processing scheme that uses two fields to identify related messages. Each message contains a `DocumentNumber` field that indicates the current document. If the message updates or extends an existing document, the message contains a `ParentDocument` that, when present, refers to the `DocumentNumber` of the document that the message updates or extends. With the default SOW key generator, each of the following messages would be a distinct message in the SOW topic: ```javascript delta_publish: {"DocumentNumber":1, "Status":"Started"} delta_publish: {"DocumentNumber":2, "ParentDocument":1, "Order":"Antivenom"} delta_publish: {"DocumentNumber":3, "ParentDocument":2, "Order":"Sandwich"} delta_publish: {"DocumentNumber":4, "ParentDocument":1, "Status":"Pending"} ``` With the default SOW key generator, at the end of the publishing process, the SOW contains four distinct records: ```javascript {"DocumentNumber":1, "Status":"Started"} {"DocumentNumber":2, "ParentDocument":1, "Order":"Antivenom"} {"DocumentNumber":3, "ParentDocument":2, "Order":"Sandwich"} {"DocumentNumber":4, "ParentDocument":1, "Status":"Pending"} ``` However, with the chaining key generator, AMPS is able to combine these messages into a single chain and produces the following single record: ```javascript {"DocumentNumber":4, "ParentDocument":1 , "Order":"Sandwich", "Status":"Pending"} ``` The sequence of events for producing this message is as follows: * When the first message arrives with a `/DocumentNumber` of `1`, the module begins a new chain (since there is no `/ParentDocument` present). * When the second message arrives, the module knows that it is an update to the same message since the message contains a `/ParentDocument` value. In this case, because the value is `1`, the update is to the first message received. The module also adds a `/DocumentNumber` of `2` to the chain, so that subsequent messages that refer to a `/ParentDocument` of `2` are a part of the chain and update the same message. * The same process occurs for the third message: the module looks up the message that should be updated when the `/ParentDocument` is `2`, and traces the chain back to the original underlying message. The module adds a `/DocumentNumber` of `3` to the chain, so that updates with a `/ParentDocument` of `3` will update the same message. * When the last message arrives, the module knows that a `/ParentDocument` of `1` is still an update to the same message, since this is the original value. The module adds the value `4` to the chain. In each case, rather than simply using the fields in the message directly, the module creates a chain of linked identifiers: each identifier in the chain produces the same SOW key as the first identifier in the chain, so each update in the chain updates the same message. It is an error for a publisher to publish a message that resolves to two different message chains. If the module receives such a message, the module will not generate a SOW key, and the message is not processed by AMPS. ## Configuring the Chaining Key Generator To load the module in AMPS, add the following configuration item to the `Modules` block of the AMPS configuration file: ```xml showLineNumbers ... ... key-chaining libamps_id_chaining_key_generator.so ``` You then use the module as the `KeyGenerator` for each topic in the SOW that will use chaining key generation. The module accepts the following options: | Parameter | Description | | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Key` |

    A field to use in chaining. AMPS supports any number of `Key` fields.

    The first `Key` field specified in the configuration is the primary field to use in chaining. When the primary field is present on a message, and the value of the field is not in an existing chain of values, the module creates a new chain.

    When the message contains the primary field and there is no previous entry for the value of that field, this message is the head of the chain and is used to generate the SOW key.

    `Key` parameters specified after the first `Key` are secondary fields.

    When a secondary field is present on the message, the module generates a SOW key for this message as though the message contained a primary field with this value. In addition, the module stores the value of the primary field in the current, if any, message as equivalent to this value, enabling subsequent messages to be chained to this message.

    There is no default for this parameter. The parameter requires an AMPS field identifier, such as `/11` or `/Order/ClOrdID`.

    The module requires a primary field and at least one secondary field to be defined.

    | | `FileName` |

    Sets the name of the file that the module uses to store chaining data.

    This module persists existing chains between restarts of the AMPS server. If a file with the given name exists when AMPS starts, the module reads chaining data from the file. Otherwise, the module creates a new file.

    | | `Primary` |

    A synonym for `Key` that explicitly specifies that this field is the primary field.

    When this configuration element is present, AMPS uses the field specified in this element as the primary field and considers any field specified in a `Key` element to be a secondary field.

    | | `Secondary` | A synonym for `Key` that explicitly specifies that this field is a secondary field. | | `Validation` |

    Specifies whether the module validates that incoming messages are properly chained. When set to `true` or `1`, the module records extra data to attempt to detect errors in the sequencing of the chain. The module will consider it an error when it detects that two or more distinct chains share identifiers and would have been combined into a single chain had messages arrived in a different order. In some systems, this indicates an error in message processing.

    Default: This option defaults to `false`.

    | ### Example The example configuration file below shows one way to use the chaining key generator module. ```xml showLineNumbers ... key-chaining libamps_id_chaining_key_generator.so ... Orders json key-chaining /DocumentNumber /ParentDocument /RelatedDocument ./sow/Orders.chain ./sow/%n.sow ExternalOrders fix key-chaining /11 /41 ./sow/ExternalOrders.chain ./sow/%n.sow ``` Notice that once the module is loaded, it can be used for any message type, and can accept different configuration values for each topic in the SOW that uses the generator. --- # Configuring Modules The `Modules` section of the AMPS configuration file is used to load, configure and define any plug-in modules used for this installation of AMPS. AMPS supports a wide variety of plug-in modules, as described in the _Extending AMPS Guide_ (available from support). The following steps are required to use a plug-in module: 1. Load the module and declare the name of the module. 2. Define the AMPS object that the module contains, give the object a name and pass any required options. 3. Use the module in a specific context. For many modules, such as `Authentication` and `Entitlement` modules, steps 2 and 3 are performed at the same time. However, they are handled separately when a module must have the same definition across multiple contexts (for example, a `MessageType` which may be used in a Transport, a SOW, a View, and replicated to other instances). Described below are the features available for a `Module`. Expand each item for more details.
    `Name` (required) A plain text name for the module. This will be used as a reference when the module is used elsewhere in the AMPS configuration, and is also the name that AMPS will use for logging messages related to the module.
    `Library` (required) The shared object file that contains the compiled module. This must contain a path to the file. When using relative paths, those paths are evaluated relative to the current working directory of the AMPS process. For example, to load a file from the current working directory, you must specify the directory (`./my_awesome_module.so`). AMPS automatically searches the `lib` directory of the AMPS distribution for shared objects. If you install the shared object in the `lib` directory of the AMPS distribution, you can simply provide the filename of the shared object without using a path.
    `Options` A list of supported features for the implemented library. AMPS allows you to pass options to the module by specifying elements within the `Options` element. The exact options that the module requires, if any, are determined by the creator of the module.
    The following section provides an example of an AMPS configuration using an authentication and entitlement plug-in module. In our example, a custom authentication module named `libauthenticate_customer001.so` has been written to manage the authentication portion of AMPS authentication. Similarly, a custom entitlements module has been written named `libentitlement_customer001.so` to manage the permissions and access of the authenticated user. The first step is to define the global `Modules` section of the AMPS configuration, and then list the individual modules, as shown below: ```xml showLineNumbers ... authentication1 libauthenticate_customer001.so info debugging entitlement1 libentitlement_customer001.so error prod ... ``` We now have an authentication module and an entitlements module that we can reference elsewhere in the AMPS configuration file to enable authentication and/or entitlements for supported features. For example, we can create one type of `Authentication` module for the instance as a whole, and then create instances of a different type of `Authentication` and `Entitlement` modules for each `Transport`, to ensure that our `Transports` are properly enabling authentication and entitlements. As shown in this example, the `Authentication` and `Entitlement` modules configured for an individual `Transport` are used for that transport, and the instance level modules are used as a default for transports that do not specify any `Authentication` or `Entitlement`. This is accomplished via an entry along the lines of the following: ```xml showLineNumbers ... my_default_security/Module> my_default_security/Module> ... fix-tcp-001 ... authenticate_customer001 entitlement_customer001 fix-tcp-007 ... authenticate_customer007 entitlement_customer007 json-tcp ... ... ``` The above example shows how our `fix-tcp-001` transport is secured with the `authenticate_customer001` authentication module, and the `entitlement_customer001` entitlement module, which would be defined in a global `Modules` section. Similarly, the `fix-tcp-007` transport is secured with the `authenticate_customer007` authentication module and the `entitlement_customer007` entitlement module. Those modules would be defined in a global `Modules` section. In contrast, the `json-tcp` transport does not define modules, and instead uses the authentication and entitlement modules specified at the instance level. --- # Special-Purpose Functions This module contains User-Defined Functions (UDF) that are a part of the AMPS distribution, but which serve specialized use cases and are not currently intended for use outside of those scenarios. The current implementations of these UDFs are considered to be experimental and are only supported for use in the intended context -- a general-purpose version of these features might behave somewhat differently. In this release, the experimental UDF module contains one function for use in preprocessing or enrichment. ## Preprocessing/Enrichment The `VALUE_LOOKUP` function can be used to look up a value in another topic during preprocessing or enrichment of a SOW topic. To use the function, you must first load the experimental UDF module and register the lookups that will be used by the module. A lookup is declared using the Lookup option to the module. This option uses a semicolon-delimited list of parameters to define the lookup. All parameters are required. | Option | Definition | |----------------------- |--------------------------------------------------| | Name | The name to use for the lookup. | | Parameter Count | The number of fields to use for the lookup. | | Message Type | The message type of the topic to use for lookup. | | Topic Name | The name of the topic to use for lookup. | | Return Field | The field to return from the located message. | | Lookup Fields | The fields to use to look up the message. | For example, to create a lookup named `return-value-by-id`, that takes a single parameter and uses that to match the `/id` field in the `data` topic of `json` type, returning the `/value` field of the matched record, you would provide this option when loading the module: ```xml return-value-by-id;1;json;data;/value;/id ``` Likewise, to create a lookup named `getNotes` that uses a combination of `/customerId` and `/orderId` to return the `/notes` field of a matching record from the orders topic of `nvfix` type, you would provide this option when loading the module: ```xml getNotes;2;nvfix;orders;/notes;/customerId;/orderId ``` To use the UDF, provide the name of the lookup and the values to lookup: ```sql VALUE_LOOKUP("return-value-by-id", /thisId) ``` The above function call will take the `/thisId` field from the current message, use that to find a matching `/id` value in the data topic (of `json` type), and then return the `/value` field from a matching record. The `VALUE_LOOKUP` function uses an exact match for the fields specified. | Function | Parameters | Description | |-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`VALUE_LOOKUP`|

    name of lookup

    one or more values for lookup

    The number of values provided must match the definition for the lookup name.

    |

    Returns a value from the referenced topic as a string.

    If multiple records match the lookup, returns one of those records. If no records match the lookup, returns `NULL`.

    | To use this function, the module must be loaded and one or more lookup functions must be declared when the module is loaded, as described in this section. This function is not loaded by default. ### Limitations and Usage This UDF has the following limitations: - 60East makes no particular performance guarantees for this UDF, but recommends that the topics used for lookup be relatively small topics with infrequently updated data. - The UDF caches lookup results for performance. This means that the UDF will increase the memory footprint of the AMPS instance (typically by the size of the lookup values + the size of the return values + 32 bytes per record indexed in the target topic + a small amount of overhead per defined lookup). - The lookup returns the value retrieved as an AMPS string. As with all AMPS values, if the value of the string can be coerced to a number, it can be used as a numeric value. - If a lookup results in multiple matches, the UDF will return values for one of the matching records, but does not guarantee which result will be returned. - The lookup function should only be used in preprocessing and enrichment in cases where it is not possible to use a view instead. The function cannot be used in view definitions, since it cannot guarantee that data is available during view recovery. The function cannot be used for a SOW topic with `transient` durability and a recovery point other than `now`, since the function cannot guarantee that data is available during SOW recovery. - Values provided are cached for performance and the cache is updated asynchronously. This means that a publish to the source topic immediately followed by a call to the UDF may or may not produce the newly published value. - AMPS considers the lookup function non-deterministic, since the value returned depends on the state of a message stored in a SOW topic, rather than the message being currently evaluated. ## Loading the Module The experimental functions module is shipped in the AMPS `lib` directory, and is named `libamps_udf_experimental.so`. To use the module, you load it as shown below, providing any options necessary for the functions contained in the module. ```xml showLineNumbers experimental-udf libamps_udf_experimental.so value-by-id;1;json;source-sow;/value;/id customer-id-by-order-id;1;nvfix;orders;/id;/customerId ``` --- # Loadable Function Modules AMPS includes the following optional functions: - AMPS includes the `libamps_udf_legacy_compatibility` module that provides date and time handling functions similar to those provided by legacy messaging systems. - AMPS provides a `libamps_udf_experimental` module that contains special purpose functions. --- # Legacy Messaging Functions The AMPS distribution includes a library of legacy messaging compatibility functions. These functions are intended to ease migration to AMPS from legacy messaging systems that provide similar functions. In this release, the legacy messaging functions provide functions to make it easy to work with date and time. These functions are not loaded into AMPS by default. To enable them, you must load the legacy messaging compatibility module by adding a directive to the AMPS configuration file. Once the module is loaded, the functions become available. No further configuration is required. For example, adding the following configuration item to the `Modules` block of the AMPS configuration file loads the legacy messaging compatibility functions. ```xml showLineNumbers ... ... compatibility-functions-module libamps_udf_legacy_compatibility.so ``` ## TIMEZONEOFFSET --- ```sql TIMEZONEOFFSET() ``` Returns a `long` that contains the current timezone offset represented in seconds East of UTC. **Parameters** None. **Returns** The current timezone offset in seconds East of UTC. **Example** Calling `TIMEZONEOFFSET` in the PDT timezone returns `-25200` (-7 hours in seconds). ## YEAR --- ```sql YEAR(timestamp) ``` Returns the year for the provided timestamp in YYYY format. The year is calculated in the UTC timezone. **Parameters** * `timestamp`: UNIX timestamp. **Returns** The year represented by `timestamp` when interpreted in UTC. **Example** Calling `YEAR` on a timestamp that represents January 25, 2010 at 10:04 AM in UTC returns `2010`. ## MONTH --- ```sql MONTH(timestamp) ``` Returns the month (1-12) for the provided timestamp. The month is calculated in the UTC timezone. **Parameters** * `timestamp`: UNIX timestamp. **Returns** The month represented by `timestamp` when interpreted in UTC. **Example** Calling `MONTH` on a timestamp that represents January 25, 2010 at 10:04 AM in UTC returns `1`. ## DAY --- ```sql DAY(timestamp) ``` Returns the day (1-31) for the provided timestamp. The day is calculated in the UTC timezone. **Parameters** * `timestamp`: UNIX timestamp. **Returns** The day represented by `timestamp` when interpreted in UTC. **Example** Calling `DAY` on a timestamp that represents January 25, 2010 at 10:04 AM in UTC returns `25`. ## DATE_UTC --- ```sql DATE_UTC(timestamp) ``` Returns the UNIX timestamp for the beginning of the provided local calendar day (00:00:00) in UTC (i.e. the date is calculated in local time then the time is adjusted for when that day started in UTC). **Parameters** * `timestamp`: UNIX timestamp. **Returns** The UNIX timestamp for the UTC start of the provided local calendar date. **Example** Calling `DATE_UTC` in the PDT timezone on a timestamp that represents May 21, 2026 at 22:00:00 PDT will return a UNIX timestamp representing May 21, 2026 at 00:00:00 UTC even though the date for the given timestamp in UTC is May 22, 2026. ## DATE --- ```sql DATE(timestamp) ``` Returns the UNIX timestamp for the beginning of the provided day (00:00:00) in the local timezone. **Parameters** * `timestamp`: UNIX timestamp. **Returns** The UNIX timestamp for the start of the provided day in local time. **Example** Calling `DATE` in the PDT timezone on a timestamp that represents May 21, 2026 at 22:00:00 PDT will return a UNIX timestamp representing May 21, 2026 at 00:00:00 PDT. ## TODAY_UTC --- ```sql TODAY_UTC() ``` Returns the UNIX timestamp for the beginning of the current local calendar day (00:00:00) in UTC (i.e. the date is calculated in local time then the time is adjusted for when that day started in UTC). **Parameters** None. **Returns** The UNIX timestamp for the UTC start of the local calendar date. **Example** Calling `TODAY_UTC` in the PDT timezone on May 21, 2026 at 22:00:00 PDT will return a UNIX timestamp representing May 21, 2026 at 00:00:00 UTC even though current date in UTC is May 22, 2026. ## TODAY --- ```sql TODAY() ``` Returns the UNIX timestamp for the beginning of the current day (00:00:00) in the local timezone. **Parameters** None. **Returns** The UNIX timestamp for the start of the local day. **Example** Calling `TODAY` in the PDT timezone on May 21, 2026 at 22:00:00 PDT will return a UNIX timestamp representing May 21, 2026 at 00:00:00 PDT. --- # Optional SOW Key Generator In this release, AMPS includes an optional module that provides the ability to combine chains of updates into a single record in the SOW: * AMPS includes the `libamps_id_chaining_generator` module that provides chained SOW key generation. --- # Optionally-Loaded Modules The AMPS distribution provides several modules that extend AMPS with optional behavior. These modules are *not* loaded by default. This section describes the optional modules included with the AMPS distribution. --- # Conflated Subscriptions AMPS provides the ability for the server to _conflate_ messages to a subscription. When a subscription requests conflation, the server will retain messages for that subscription for a certain period of time, the _conflation interval_, and provide the latest update to that message once a message has been retained for that interval. In effect, AMPS guarantees that a subscription will receive _no more than_ one update for a given message per conflation interval. Conflated subscriptions provide a way to reduce the bandwidth and processing for a subscriber in cases where a subscriber needs periodic updates with the current state of a message, rather than the complete message stream. AMPS provides per-subscription conflation for cases where only a small number of subscribers require conflation, or if conflation is required only in unusual cases. If multiple subscribers will have the same conflation needs, consider using [Conflated Topics](../conflated-topics). For example, imagine an application that monitors selected stocks and displays the current prices on a large screen, which refreshes every few seconds. This application may use the same topics as a trading desk, but has very different needs for data freshness and completeness. Since updates to each symbol will only be displayed every few seconds, the application only needs point-in-time updates of the prices, rather than the full stream of price changes. To meet this need, the application could specify that the subscription conflates price updates by `tickerId` with a conflation interval of two seconds. For each distinct value of the `tickerId` field, AMPS will retain messages for two seconds. If another message with the same `tickerId` is processed for the subscription during the conflation interval, that message completely replaces the previous message. At the end of the two second conflation interval, the message is delivered to the application. This lets the application receive an up-to-date price at most every two seconds, without having to process a large number of updates that will never be displayed. This approach also ensures that the price is never more than two seconds out of date, which means that each time the screen is refreshed, the price is current. As mentioned in the example above, if the subscription uses `tickerId` for conflation and the following sequence of messages arrive during a conflation interval: ```javascript { "tickerId" : "IBM", "price" : 150.34 } { "tickerId" : "IBM", "price" : 149.76 } { "tickerId" : "IBM", "price" : 149.32 } { "tickerId" : "IBM", "price" : 151.10 } ``` AMPS delivers only the last message for that `tickerId`: ```javascript { "tickerId" : "IBM", "price" : 151.10 } ``` Notice that when a subscription is conflated, AMPS does not guarantee that messages are delivered precisely in the order in which they arrived in AMPS, since the latest update is delivered based on the conflation interval. When the `timestamp` option is used with conflated subscriptions, AMPS provides the timestamp for the first message conflated. ## When to Use Conflated Subscriptions Conflated subscriptions reduce the bandwidth for a subscription, and may reduce the processing resources required for a subscription. However, rather than immediately delivering messages, AMPS retains messages in memory for the conflation interval. This can increase the memory required for the subscription. AMPS contains other features for conflating messages and reducing bandwidth. Conflated subscriptions are most appropriate when: * **Network bandwidth is at a premium**, and you would like AMPS to spend slightly more processing time and potentially more memory to reduce the bandwidth needs of the application. * Each subscription has **different conflation needs**. For example, if each subscription has a dramatically different conflation interval, or needs to conflate by different fields. If most subscribers will use a similar conflation interval and use the same fields for conflation, using a _Conflated Topic_ can provide equivalent results with lower overhead. * The conflation needs are **relatively predictable and consistent** for the subscription. If you need the application to conflate messages _only_ when processing is slow or there are bursts of message traffic, _client-side conflation_ provides that ability and may be a better choice than a conflated subscription. See the developer guide for your programming language of choice for details. The considerations above are general guidance to help you consider options and choose a conflation strategy. You can also combine approaches as necessary. For example, if most of your subscriptions require a 3 second conflation interval by `tickerId`, while a few subscriptions require a 15 second interval, you could create a [Conflated Topic](../conflated-topics) with a 3 second interval. Those subscriptions that require a 15 second interval could subscribe with that interval. This provides both sets of subscriptions with the intervals that they need. ## Requesting Conflation on a Subscription To request conflation on a subscription, set the following options on the subscription: | Option | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `conflation=n` |

    Specifies whether to conflate this subscription. The value provided can be a time interval, `auto` or `none`.

    When present and set to a value other than none, enables conflation for the subscription.

    Can also be set to `auto`, which requests that AMPS attempt to determine an appropriate conflation interval based on client consumption.

    Recognizes the same time specifiers used in the AMPS configuration file (for example, `100ms` or `1s` or `1m`).

    Defaults to `none`.

    | | `conflation_key=[keys]` |

    When conflation is enabled, specifies the fields to use to determine message uniqueness. The format of this option is a comma-delimited list of XPath identifiers within brackets.

    For example, to conflate based on the value of the `/tickerId` and `/customerId` within a message. the value of this option would be:

    `[/tickerId,/customerId]`

    Defaults to the SOW key fields for SOW topics.

    No default for non-SOW topics. This option is required for non-SOW topics.

    This option is not valid with the `oof` option unless the keys provided are identical to the keys for the topic.

    This option can only be used when `conflation` is also specified.

    | For example, to request a 10 second conflation interval with messages conflated on the `[/orderId]` field, you would use the following options string: ```bash conflation=10s,conflation_key=[/orderId] ``` --- # Filtering Subscriptions by Content One thing that differentiates AMPS from classic messaging systems is its ability to route messages based on message content. Instead of a publisher declaring metadata describing the message for downstream consumers, the publisher can simply publish the message content to AMPS and let AMPS examine the native message content to determine how best to deliver the message. The ability to use content filters greatly reduces the problem of oversubscription that occurs when topics are the only facility for subscribing to message content. The topic space can be kept simple by using content filters to deliver only the desired messages. The topic space can reflect broad categories of messages and does not have to be polluted with metadata that is usually found in the content of the message. In addition, many of the advanced features of AMPS such as out-of-focus messaging, aggregation, views, and SOW topics rely on the ability to filter content. Content-based messaging is somewhat analogous to database queries that include a `WHERE` clause. Topics can be considered tables into which rows are inserted (or updated). A subscription is similar to issuing a `SELECT` from the topic table with a `WHERE` clause to limit the rows which are returned. Topic-based messaging is analogous to a `SELECT` on a table with no limiting `WHERE` clause. AMPS uses a combination of XPath-based identifiers and SQL-92 operators for content filtering. Some examples are shown below: ## Example Filter for a JSON Message: ```sql (/Order/Instrument/Symbol == 'IBM') AND (/Order/Px >= 90.00 AND /Order/Px < 91.00) ``` ## Example Filter for an XML Message: ```sql (/FIXML/Order/Instrmt/@Sym == 'IBM') AND (/FIXML/Order/@Px >= 90.00 AND /FIXML/Order/@Px < 91.0) ``` ## Example Filter for a FIX Message: ```sql /35 < 10 AND /34 == /9 ``` For more information about how content is handled within AMPS and the syntax of AMPS filters, details are presented at [AMPS Expressions](../amps-expressions) and [AMPS Functions](../amps-functions). :::info Unlike some other messaging systems, AMPS lets you use a relatively small set of topics to categorize messages at a high level and use content filters to retrieve specific data published to those topics. Examples of good, broad topic choices: `trades`, `positions`, `MarketData`, `Europe`, `alerts` This approach makes it easier to administer AMPS, easier for publishers to decide which topics to publish to and easier for subscribers to be sure that they've subscribed to all relevant topics. ::: --- # Messages in AMPS Communication between applications and the AMPS server uses AMPS messages. AMPS messages are received or sent for every operation in AMPS. Each AMPS message has a specific type and consists of a set of headers and a payload. The headers are defined by AMPS and formatted according to the protocol specified for the connection. Typically, applications use the standard `amps` protocol which uses a JSON document for headers. The payload, if one is present, is the content of the message and is in the format specified by the message type. Messages received from AMPS have the same format as messages to AMPS. These messages also have a specific type, with a header formatted according to the protocol and a payload of the specified message type. For example, AMPS uses `ack` messages, short for acknowledgment, to report the status of commands. AMPS uses `publish` messages to deliver messages on a subscription, and so on for other commands and other messages. Let's consider a complete interaction between an application and the AMPS server as an example. When a client subscribes to a topic in AMPS, the client sends a `subscribe` message to AMPS that contains the information about the requested subscription and, by default, a request for an acknowledgment that the subscription has been processed. AMPS returns an `ack` message when the subscription is processed that indicates whether the subscription succeeded or failed, and then begins providing `publish` messages for new messages on the subscription. The `publish` messages continue as messages that match the subscription arrive at the AMPS server. If the application needs to stop the subscription, the application sends an `unsubscribe` message to the AMPS server, indicating the subscription to end. Once the AMPS server processes the unsubscribe message, the server will no longer send messages for that subscription to the application. Should the application disconnect, the AMPS server removes all subscriptions for that connection (whether or not the application sends an `unsubscribe` command first). Messages to and from AMPS are described in more detail in the _AMPS Command Reference_, available on the 60East website and included in the AMPS client SDKs. In this version of AMPS, the communication transports used by AMPS accept message sizes of up to 200MB in a single command to AMPS. Messages larger than 200MB may be rejected by the transport as invalid. Should your use of AMPS require larger message sizes, contact 60East support. :::tip This version of AMPS limits messages to 200MB in total size ::: ### Introduction to AMPS Headers The _AMPS Command Reference_ contains a full list of headers for each command. The table below lists some commonly used headers. | Header | Description | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Topic |

    The topic that the message applies to.

    For commands to AMPS, this is the topic that AMPS will apply the command to. For messages from AMPS, this is the topic from which the message originated.

    | | Command |

    The command type of the message. Each message has a specific command type.

    For example, messages that contain data from a query over a SOW topic have a command of `sow`, while messages that contain data from a publish command have a command of `publish` and messages that acknowledge a command to AMPS have a command type of `ack`.

    | | CommandId |

    An identifier used to correlate responses from AMPS with an initial command.

    For example, `ack` messages returned by AMPS contain the CommandId provided with the command they acknowledge and subscriptions can be updated or removed using the CommandId provided with the `subscribe` command.

    | | SowKey |

    For messages received from a State of the World (or SOW) topic, an identifier that AMPS assigns to the record for this message. SOW topics are described the [State of the World Topics](../sow) section.

    This header is included on messages from a SOW topic by default. AMPS will omit this header when the subscription or SOW query includes the `no_sowkey` option.

    | | CorrelationId |

    A user-specified identifier for the message.

    Publishers can set this identifier on messages. AMPS does not parse, change, or interpret this identifier in any way.

    This header is limited to characters used in Base64 encoding.

    | | Status | Set on `ack` messages to indicate the results of the command, such as `Success` or `Failure`. | | Reason | Set on `ack` messages to indicate the reason for the Status acknowledgment. | | Timestamp |

    Optionally set on `publish` messages and `sow` messages to indicate the time at which the local AMPS instance processed the message.

    To receive a timestamp, the SOW query or subscription must include the `timestamp` option on the command that creates the subscription or runs the query. The timestamp is returned in ISO-8601 format.

    | This section presents a few of the commonly used headers. See the _AMPS Command Reference_ for a full description of AMPS messages. AMPS does not provide the ability to add custom header fields. However, AMPS composite message types provide an easy way to add an additional section to a message type that contains metadata for the message. Since composite message type parts fully support AMPS content filtering, this approach provides more flexibility and allows for more sophisticated metadata than simply adding a header field. See the [Composite Messages](../message-types/composite-messages) section for details. --- # Message Ordering AMPS guarantees that, for each AMPS instance, each subscription to a topic receives messages in the order in which AMPS received the messages (with the exception of messages that have been returned to a message queue for redelivery or the results of a query). Before a given message is delivered to a subscriber, all previous messages for that topic are delivered to the subscriber. AMPS does this by enforcing a total order across the instance for all messages received from publishers, including messages received via replication. When AMPS is using a transaction log, that order is preserved in the transaction log for the instance, and persists across instance restarts. When replaying from the transaction log, a subscriber will always receive messages in the same order in which messages were originally delivered by that instance of AMPS. This guarantee also applies across topics for subscriptions that involve multiple topics, for all topics _except_ views, queues, and conflated topics. Views and queues guarantee that every message on the view or the queue appears in the order in which the message was published. However, the computation involved in producing messages for views and queues may introduce some amount of processing latency, and AMPS does not delay messages on other topics while performing these computations. For a queue that provides `at-least-once` delivery, if a processor fails and returns a message to the queue, that message will be redelivered (which means that the new processor may receive the message out of order). Likewise, when AMPS is providing conflation (either through a conflated topic or the conflation options on a subscription), AMPS does not provide ordering guarantees for conflated messages. Applications often use this guarantee to publish checkpoint messages, indicating some external state of the system, to a checkpoint topic. For example, you might publish messages marking the beginning of a business day to a checkpoint topic, `MARKERS`, while the `ORDERS` topic records the orders during that day. Subscribers to the regular expression `^(ORDERS|MARKERS)$` are guaranteed to receive the message that marks the business day before any of the messages published to the `ORDERS` topic for that day, since AMPS preserves the original order of the messages. For messages constructed by AMPS, such as the output of a view, AMPS processes messages for each topic in the order in which they arrive (unless conflation is requested) and delivers each calculated message to subscribers as soon as the calculation is finished and a message is produced. This keeps the latency low for each individual topic. However, this means that while AMPS guarantees the order in which messages are produced within each view, messages produced for views that do simple operations will generally take less time to be produced than messages for views that perform complex calculations or require more complicated serialization. This means that AMPS guarantees ordering within view topics, but does not guarantee that messages for separate view topics arrive in a particular order. The figure below shows a possible ordering for messages received on an underlying topic and two views that use the topic: Notice that within each topic, AMPS enforces an absolute order. However, the Simple View produces the results of Message 3 before the Complex View produces the results of Message 2. AMPS delivers the message for each topic as soon as possible. ## Replicated Message Ordering When providing messages received via _replication_ (see [Replicating Messages Between Instances](../replication)), the principles on message ordering provided above still apply. AMPS records messages into the local transaction log in the order in which messages are received by the instance and provides messages to subscribers in that order. AMPS uses the sequence of publishes assigned by the original publisher and the order assigned by the upstream instance to ensure that all replicated messages are received and recorded in order with no gaps or duplicates. Each instance of AMPS replicates messages to downstream destinations in the order in which messages are recorded in the transaction log. AMPS does not enforce a global total ordering across a replication topology. This peer-to-peer approach means that an AMPS instance can continue accepting messages from publishers and providing messages to subscribers even when the remote side of a replication link is offline or if replication is delayed due to network congestion. However, if two messages are published to _different_ instances at the same time by different publishers, the two instances may record a different _overall_ message order for those messages, even though message order _from each publisher_ is preserved. --- # Replacing Subscriptions AMPS provides the ability to perform atomic subscription replacement. This allows you to replace the filter, change the topic, or update the options for a subscription. The most common use for this capability is for an application to change the filter for a subscription. For example, a GUI that is providing a view of a set of orders may need to add or remove an order from the set of orders being displayed. By replacing the content filter with a filter that tracks the updated set of orders, the application can do this without missing messages, getting duplicate messages, or having to manage more than one subscription. Replacing a filter is an atomic operation. That is, the application is guaranteed not to miss messages that are in both the original and replacement subscription, and is guaranteed to receive all messages for the new subscription as of the point at which the replacement happens. To replace a subscription, applications re-submit the subscription _using the subscription ID of the previous subscription_. See the Developer Guide of the client library you are using and the [AMPS Command Reference](../../amps-command-reference/) for details. When replacing a `sow_and_subscribe` command (described later in the guide), AMPS runs the SOW command again and provides any messages that were not previously in the result set to the application. See the section called [Replacing Subscriptions with SOW and Subscribe](../sow-queries/query-and-subscribe.md#replacing-subscriptions-with-sow-and-subscribe) for details. Notice that some options on an initial subscription limit the support for `replace` on a subscription. In those cases, the limitation is described when the option is described. ### Replacing the Content Filter on a Subscription AMPS allows you to replace the content filter on an existing subscription. When this happens, AMPS begins sending messages on the subscription that match the new filter. When an application needs to bring more messages into scope, this can be more efficient than creating another subscription. For example, an application might start off with a filter such as the following: ```sql /region = 'WesternUS' ``` The application might then need to bring other regions into scope, for example: ```sql /region IN ('WesternUS', 'Alaska', 'Hawaii') ``` ### Replacing the Topic on a Subscription AMPS allows a subscription to replace the topic on a subscription. When the topic is replaced, AMPS re-evaluates the subscription as it does when a filter is replaced. If the subscription is updated to include a topic that the user does not have permission to subscribe to, the `replace` operation succeeds, but no messages will be delivered on that topic. ### Replacing the Options on a Subscription AMPS allows a subscription to replace some of the options on the subscription. In this case, the subscription is evaluated as though the topic or filter has been replaced. Any new messages generated after the subscription is replaced use the new options. However, AMPS does not replay or re-query previous messages to apply the options. For example, if a `sow_and_subscribe` command did not previously specify Out-of-Focus tracking and adds this option, AMPS generates the appropriate Out-of-Focus messages from the replace point forward. AMPS does not recreate Out-of-Focus messages that would have previously been generated by the subscription. If the subscription uses pagination (see [Managing Result Sets](../sow-queries/managing-result-sets)), the replacement must contain the full set of pagination options provided on the original subscription. For a paginated subscription, the replacement _may not_ change the topic of the subscription. Instead, close the existing subscription and create a new subscription with a different topic. --- # Retrieving Part of a Message: Select List AMPS has the ability to allow a subscriber to retrieve only the relevant parts of a message, in the same way that a SQL query can retrieve only specified fields from a table. For example, consider a topic that stores an event ID, a short description, and a detailed event record. A UI that presents an overview of the contents of the topic might only need the event ID and short description to present a high-level view of the topic contents, while retrieving the detailed event record when a user explicitly requests the details for a specific record. With select lists, AMPS allows an individual subscription to control which fields are retrieved from a subscription or query. In the example above, the subscription would include a select list that requests that AMPS provide the event ID and description, while excluding any other field. To do this, the application would include the following option on the command used to retrieve data for the overview: `select=[-/,+/event_id,+/description]` When provided by an application as a part of a command to AMPS, a select list is applied after any content filtering is applied. The select list specifies the contents of the subscription, but does not affect the underlying messages, and the contents of the subscription select list do not affect filter evaluation or query results. :::info To use a select list, the message type format must allow partial serialization of messages. Message formats that require a full message, or that interpret missing fields as having a specific value, cannot be used with select lists, since a partial message would either create an invalid message, or change the way the data in the message is interpreted. ::: ## Creating Select Lists As mentioned above, to provide a select list on a command, add the keyword `select` and a comma-delimited list of _field directives_ to the options for a subscription or query in AMPS. Each _field directive_ is a combination of an _inclusion specifier_ and an _AMPS identifier_. For example, the _field directive_ `+/event_id` has an _inclusion\_specifier_ of `+` and the AMPS identifier of `/event_id`. This _field directive_ specifies that the `/event_id` field is included in the message returned to the subscriber. AMPS recognizes the following _inclusion specifier_ values: | Specifier | Meaning | | ------------- | ---------------------------------------------------------------------- | | `-` | Explicitly exclude the field for the identifier immediately following. | | `+` | Explicitly include the field for the identifier immediately following. | Identifiers for individual fields follow the syntax described in the [Identifiers](../amps-expressions/identifiers) section. For select lists, AMPS also recognizes the special field directive of `-/` to specify that all fields should be excluded and the special field directive of `+/` to specify that all fields should be included. If no field directive in the select list applies to a given field in a message, that field is included in the message. If a field is covered by multiple field directives, AMPS respects the most specific field directive. In other words, a select list that contains the field directives `+/,-/details` will include all fields except the details field. A select list that contains the field directives `-/event,+/event/description` will include the `/event/description` subfield, but no other contents of the `/event` field. (If an identifier is provided twice in the same select list, AMPS uses the first field specifier that contains the identifier.) With select lists, AMPS does not create fields that are not in the original message. This means that if the select list requests a field that does not exist in the original message, the message delivered to the subscriber will not contain that field. Notice that a select list only changes how a message is delivered to the subscriber that the select list applies to. The original message is unaffected, and the complete message is delivered to any subscriber that does not specify a select list. AMPS contains related functionality that may be more appropriate for some applications: * To modify a message as it is published to AMPS, use _Enrichment and Preprocessing_. With those features, the original publish message is modified and the modified message is stored in AMPS and sent to all subscribers. * AMPS also offers the ability to create a view of a set of messages that aggregates data across a set of messages and produces a result (for example, the total value of all open orders for each customer). See the chapter on _Aggregating and Analyzing Data in AMPS_ for more details. ## Select List Examples For example, consider an original message like the following JSON document: ```javascript showLineNumbers { "id": 42, "name":"Arthur", "day":"Thursday", "complaint":"Unannounced construction in neighborhood.", "pocket_contents": { "left":"twine", "right":"towel" } } ``` An application might only need to see the `id` and `complaint` description. To retrieve just those fields of a message, the application could add the following option to the command that retrieves the message: ```bash select=[-/,+/id,+/complaint] ``` This select list tells AMPS to remove all fields from the message except for the `/id` field and the `/complaint` field. With this select list, the message above will be delivered as: ```javascript showLineNumbers { "id": 42, "complaint":"Unannounced construction in neighborhood." } ``` Likewise, an application could want to know the name of the person making the complaint and the contents of that person's left pocket: ```bash select=[-/,+/name,+/pocket_contents/left] ``` From the original message, the result of providing this select list would be: ```javascript showLineNumbers { "name": "Arthur", "pocket_contents": { "left":"twine"} } ``` Last, consider an application that wants to see everything in the message except the `pocket_contents`. That application could provide an option such as: ```bash select=[-/pocket_contents] ``` With that specifier, AMPS provides any field in the message except the `pocket_contents`, producing the following result: ```javascript showLineNumbers { "id": 42, "name":"Arthur", "day":"Thursday", "complaint":"Unannounced construction in neighborhood." } ``` :::info Select lists are not available for `struct` message types, since these types represent a single, contiguous block of memory (and, therefore, cannot meaningfully have omitted fields). ::: --- # Topics A topic is a string that is used to declare a subject of interest for purposes of routing messages between publishers and subscribers. Topic-based Publish and Subscribe (e.g., Pub/Sub) is the simplest form of Pub/Sub filtering. All messages are published with a topic designation to the AMPS engine, and subscribers will receive messages for topics to which they have subscribed. For example, in the diagram above there are two publishers: Publisher 1 and Publisher 2 which publish to the topics `LN_ORDERS` and `NY_ORDERS`, respectively. Messages published to AMPS are filtered and routed to the subscribers of a respective topic. For example, Subscriber 1, which is subscribed to all messages for the `LN_ORDERS` topic will receive everything published by Publisher 1. Subscriber 2, which is subscribed to the regular expression topic `".*_ORDERS"` will receive all orders published by Publisher 1 and 2. Subscriber 3, which is subscribed to all messages for the `NY_ORDERS` topic will receive everything published by Publisher 2. Regular expression matching makes it easy to create topic paths in AMPS. Some messaging systems require a specific delimiter for paths. AMPS allows you the flexibility to use any delimiter. However, 60East recommends using characters that do not have significance in regular expressions, such as forward slashes. For example, rather than using `northamerica.orders` as a path, use `northamerica/orders`. :::tip AMPS does not restrict the characters that can be present in a topic name. However, notice that topic names that contain regular expression characters (such as `.` or `*`) will be interpreted as regular expressions by default, which may cause unexpected behavior. When a topic name contains regular expression characters, a subscription can use the `non_regex_topic` option to specify that AMPS will not treat the topic name as a regular expression. ::: Topics that begin with `/AMPS` are reserved. The AMPS server publishes messages to topics that begin with `/AMPS` as described in the [Event Topics](../events) section. When a transaction log is configured, the AMPS server may periodically write to an `/AMPS/Tx/Checkpoint` topic to assist in replication recovery. Some versions of the AMPS client libraries may internally publish to `/AMPS/devnull`. Your applications should not publish to topics that begin with `/AMPS`, as publishes to those topics may fail. Each topic has an associated message type. Each client connection to AMPS also has an associated message type. A given client connection can only publish to topics with the same message type, and can only receive messages from topics with the same message type. ## Ad Hoc Topics AMPS does not require explicit configuration of a topic for publishers to send messages to the topic and subscribers to receive messages from the topic. However, if there is no configuration for the topic, AMPS does not persist messages to the topic, so no features that depend on having a persisted message state (for example; replay, aggregation, State of the World, and so on) are available for that topic. The message will be delivered to subscriptions that are active when the message is published, but the message will not be persisted or retained. These "ad hoc" topics are useful for low-latency delivery of messages that are only useful at the time that they are published. ## Matching Multiple Topics With Regular Expressions With AMPS, a subscriber can use a regular expression to simultaneously subscribe to multiple topics that match the given pattern. This feature can be used to effectively subscribe to topics without knowing the topic names in advance. Notice that a message cannot be published to a topic pattern. The topic for a given message is unambiguously specified using a literal string. From the publisher’s point of view, it is publishing a message to a topic. A publisher does not publish to a topic pattern. When a subscription is sent to AMPS, the topic for the subscription is interpreted as a regular expression if the topic includes special regular expression characters. Otherwise, the topic must be an exact match. Some examples of regular expressions to match a set of topics are included in the table below: | Topic | Behavior | | ------------- | ------------------------------------------------------------ | | `^trade$` | Matches only "trade". | | `^client.*` | Matches "client", "clients", "client001", etc. | | `.*trade.*` | Matches "NYSEtrades", "ICEtrade", etc. | | `trade.info` | Matches "trade/info", "trade-info", "every/trade/info", etc. | For more information regarding the regular expression syntax supported within AMPS, please see the [Regular Expressions](/docs/amps-user-guide/amps-expressions/regular-expressions) section for details. AMPS can be configured to disallow regular expression topic matching for subscriptions. See [Instance-Level Configuration](/docs/amps-user-guide/configuring-amps/instance-configuration) for details. --- # Subscribe and Publish AMPS is a rich message delivery system. At the core of the system, the AMPS engine is highly-optimized for publish and subscribe delivery. In this style of messaging, publishers send messages to a message broker (such as AMPS) which then routes and delivers messages to the subscribers. "Pub/Sub" systems, as they are often called, are a key part of most enterprise message buses, where publishers broadcast messages without necessarily knowing all of the subscribers that will receive them. This decoupling of the publishers from the subscribers allows maximum flexibility when adding new data sources or consumers. AMPS can route messages from publishers to subscribers using a topic identifier and/or content within the message's payload. For example, in the figure above, there is a Publisher sending AMPS a message pertaining to the `LN_ORDERS` topic. The message being sent contains information on Ticker "IBM" with a Price of 125, both of these properties are contained within the message payload itself (i.e., the message content). AMPS routes the message to Subscriber 1 because it is subscribing to all messages on the `LN_ORDERS` topic. Similarly, AMPS routes the message to Subscriber 2 because it is subscribed to any messages having the Ticker equal to "IBM". Subscriber 3 is looking for a different Ticker value and is not sent the message. --- # Advanced Messaging and Queues Queues are implemented as AMPS topics which lets you use the advanced messaging features of AMPS to create your queues and provide insight into your queues. For example, consumers can use content filtering to select the messages from the queue that they want to consume. You can use content filters to select only a subset of messages published to an underlying topic to populate the queue. You can create a view that provides insight over the messages currently in the queue, including a view that joins reference data from another topic to make the information easier to understand. Since messages for queues are recorded in the transaction log, you can easily replay the messages that the queue provided to subscribers by using a bookmark subscription. ## Querying Queues as a View For each queue, AMPS provides a view of the currently available messages. Applications can query this queue just as though it were a view. For example, if you have a queue named `PendingOrders`, you can see the currently available messages in the queue by querying the queue as though it were a view, with a `sow` command. A query of a queue is _read-only_. AMPS does not lease the returned messages to the querying application, or remove them from the queue. ## Topic in the SOW as an Underlying Topic for a Queue AMPS fully supports a topic that maintains a SOW as an underlying topic for a queue. Since a queue records every individual publish to a topic (rather than simply preserving the current state of a distinct message identified by a SOW key), each publish to the SOW topic creates a new message in the queue. AMPS does not provide Out-of-Focus messages to the queue. Only publish messages are added to the queue. Deleting a message from an underlying topic that maintains a SOW does not remove the corresponding messages from the queue. Likewise, when a message expires from the SOW, it is not removed from the queue. When a message is published to a topic in the SOW, and that topic uses expiration, the message is stamped with the expiration value set for the topic before being written to the transaction log. This means that the message will have an expiration timestamp that can be used by the queue for expiration. The `ExpirationModel` parameter of the queue configuration determines how the queue manages expiration when a message has an expiration set. ## Delta Messaging with Queues AMPS delta subscriptions rely on being able to determine the last state of a message delivered on a subscription and providing a set of changes to the subscriber. With AMPS queues, AMPS treats each update to a SOW record as a new message, so there's no previous state that would generate a delta message. When an underlying topic of a queue is a SOW topic, AMPS supports delta publish to that underlying topic. The full, merged message is added to the queue. AMPS allows delta subscriptions to a queue, but treats each message as a new publish and delivers the full message. ## Views and Aggregated Subscriptions over Queues AMPS fully supports creating a view or an aggregated subscription with a queue as an underlying topic. In both cases, AMPS operates on the messages that are currently available in the queue. When a message is leased, that message is no longer available to the queue and does not appear in the view or the aggregated subscription. If the message is returned to the queue, then the message is again available to the view or aggregated subscription. When a message expires, that message is no longer available in the view or aggregated subscription. Views and aggregated subscriptions are considered to be a query of the queue, so they are _read-only_. Views and aggregated subscriptions do not lease messages from the queue and do not affect message delivery. Views over queues can be useful to show constantly-updated aggregates of the activity in the queue. For example, you could create an aggregate that shows the total value of unprocessed orders currently in the queue. ## Bookmark Subscriptions to Queues A queue itself supports `at_most_once` or `at_least_once` delivery. It does not provide a way to replay messages once a subscriber has acknowledged the message. To support replay of messages from a queue, AMPS translates a bookmark subscription to a queue to be a bookmark subscription to the underlying topic for the queue. This allows you to replay messages from the underlying topic _without_ queue delivery semantics. That is, a bookmark subscription to a queue becomes a publish/subscribe bookmark subscription to the underlying topic, _not_ a subscription to the queue. Messages from this subscription do not have `at-most-once` or `at-least-once` delivery, and do not need to be acknowledged. The subscription is a publish/subscribe bookmark subscription, just as though there was no queue for the topic. To get queuing semantics, do not include a bookmark on subscriptions to a queue. --- # Advanced Queue Configuration This section describes more advanced configuration settings for AMPS message queues. ## Using Multiple Underlying Topics AMPS queues can contain messages from any number of underlying topics. This provides a flexible delivery model, and allows applications to populate multiple queues with a single publish to AMPS, which simplifies publisher code, reduces bandwidth, and ensures that the message is provided to all queues from the same point in the message stream. To create a queue that includes messages from underlying topics, you provide a regular expression that matches the set of topic names that contain messages for the queue. You also provide a `DefaultPublishTarget` that specifies the topic name for AMPS to use when a message is published directly to the queue topic. For example, you might configure a set of topics as follows: ```xml showLineNumbers ... ORDERS_ANALYTICS json ^ORDERS$|^ORDERS_ANALYTICS_DIRECT$ ORDERS_ANALYTICS_DIRECT ORDERS_RISK json ^ORDERS$|^ORDERS_RISK_DIRECT$ ORDERS_RISK_DIRECT ... ``` In this case, when a message is published to the `ORDERS` topic, both the `ORDERS_ANALYTICS` and the `ORDERS_RISK` queues deliver the message. However, a publisher can also publish directly to each queue by publishing a message to the `_DIRECT` topic for that queue. Furthermore, any publish to the name of the queue will be routed to the appropriate `_DIRECT` topic. The following table demonstrates how messages are provided to topics with this configuration: | Publish To | Results | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `ORDERS` | Both `ORDERS_ANALYTICS` and `ORDERS_RISK` enqueue the message, since `ORDERS` matches the `UnderlyingTopic` of both queues. | | `ORDERS_ANALYTICS` |

    The message is published to the `DefaultPublishTarget` of `ORDERS_ANALYTICS`, which is `ORDERS_ANALYTICS_DIRECT`.

    The message is then enqueued to `ORDERS_ANALYTICS`, since `ORDERS_ANALYTICS_DIRECT` matches the `UnderlyingTopic` of `ORDERS_ANALYTICS`.

    | | `ORDERS_RISK` |

    The message is published to the `DefaultPublishTarget` of `ORDERS_RISK`, which is `ORDERS_RISK_DIRECT`.

    The message is then enqueued to `ORDERS_RISK`, since `ORDERS_RISK_DIRECT` matches the `UnderlyingTopic` `ORDERS_RISK`.

    | | `ORDERS_ANALYTICS_DIRECT` | The message is published to `ORDERS_ANALYTICS_DIRECT`, and is then enqueued to `ORDERS_ANALYTICS`. | | `ORDERS_RISK_DIRECT` | The message is published to `ORDERS_RISK_DIRECT`, and is then enqueued to `ORDERS_RISK`. | ## Priority Queues AMPS includes the ability to specify that messages from a queue are delivered in order of _priority_ rather than being delivered strictly in publication order. To enable prioritization on a `Queue`, add the `Priority` tag to the queue configuration and specify the field or expression to use to set the priority. When a queue definition specifies `Priority`, AMPS orders delivery based on a descending sort of the value calculated for the `Priority` rather than in descending order of age. `Priority` expressions use the same grammar as other expressions in AMPS, as described in [AMPS Expressions](../amps-expressions). The results of the `Priority` expression are interpreted as an `unsigned long`. That is, the result should be a positive integer that can be represented in 64 bits. Non-numeric values or `NULL` are treated as `NaN`, and have the lowest priority. For example, to create a queue that delivers messages in order of the product of the price and quantity specified in the message, you could use the following queue definition: ```xml showLineNumbers ... ... OrderPriority json at-least-once Orders /price * /qty ... ``` ## Synchronizing Work with Barrier Messages AMPS queues provide the ability to synchronize work across a set of subscribers by sending the same message to all subscribers simultaneously, when the queue is fully processed up to the point of the message. These messages are referred to as _barrier messages_, since they are roughly equivalent to the concept of a barrier in multithreaded programming. Barrier messages simplify coordination between current subscribers to a queue. They allow you to easily coordinate operations (such as end of day processing), update reference data at the precise point in the message stream that the update should take effect, send a shutdown message when all available work is processed, and so on. Since barrier messages are integrated with queue delivery, there is no need to write extra code to correlate the barrier message with the correct point in the queue, to manage multiple subscriptions, or to write code that guarantees delivery to all current subscribers. To specify which messages should be treated as barrier messages, the queue configuration includes a `BarrierExpression` element that contains a filter. All messages that match that filter are considered barrier expressions. When a message matches the `BarrierExpression`, AMPS delivers the message in the following way: * AMPS does not deliver the message until all previous messages in the queue are acknowledged. * AMPS does not deliver messages that are after the barrier message until the barrier message is sent to subscribers. * When all previous messages in the queue are acknowledged, AMPS delivers the barrier message to all current subscribers to the queue. * AMPS immediately removes the message from the queue, without requiring acknowledgment from any subscriber. A barrier message sent to a subscriber does not count toward that subscriber's backlog, since the barrier message is immediately acknowledged. When sending the barrier message, AMPS does not consider the current backlog of the subscriber (so, subscribers that request `max_backlog=0` or which use a regular expression topic and have their backlog filled by messages from other topics will still receive the barrier message). AMPS does, however, apply content filters and entitlement filters for each subscription when delivering the message. If the content filter or entitlement filter for a subscription does not match the barrier message, that subscription will not receive the message. The fact that this subscription does not receive the actual barrier message does not affect the behavior of the barrier itself. A subscription that does not match the message will still see message delivery pause until the barrier is released, but will not receive the actual barrier message. Each AMPS instance manages the delivery of barrier messages for the subscribers connected to that instance, when all messages prior to the barrier message in the local queue have been acknowledged. ## Limiting Currently Deliverable Messages In some cases, it may be important to limit the maximum number of messages that AMPS considers to be part of the currently active, deliverable part of the queue. This can be advantageous in cases where memory is limited, or in cases where it is expected that a queue is only processed infrequently (for example, a reconciliation process that runs at the end of each day). The `TargetQueueDepth` option on a queue allows you to set the target depth for the number of messages that AMPS will consider to be deliverable at a given time from the local AMPS instance. This is most suitable in cases where the queue will not be needed during times when the instance is active, but will be processed when other traffic is at a minimum. In this case, using `TargetQueueDepth` allows you to limit memory usage when the queue is not being consumed in return for increased CPU and I/O usage when the queue is being processed. :::tip Another strategy for reducing the amount of memory required for a queue is to specify [`FileBackedMetadata`](configuring-queues-in-sow#managing-queue-metadata) in the queue configuration. This setting stores the queue metadata in a file, which allows that metadata to be paged out of memory when it is not being used. This typically provides better performance during queue processing since AMPS can determine metadata for messages as they arrive rather than re-reading the transaction log to populate the queue as it is processed or as metadata maintenance commands (such as acknowledgements and transfer requests) arrive. ::: If the number of messages for the queue exceeds the `TargetQueueDepth` on an instance, messages beyond the specified depth are considered to be inactive (though still part of the queue). Those messages are not deliverable to subscribers on this instance. They do not appear in a `sow` query of the queue, and are not considered when a view over the queue is calculated. A `sow_delete` that uses a filter will only apply to the messages in the queue that are currently deliverable. As messages are acknowledged from the queue, AMPS will add messages to the active part of the queue until the active part of the queue once again reaches the `TargetQueueDepth`. For messages that are not currently deliverable, AMPS will not read those messages from the transaction log and will not maintain queue state for those messages. This also means that, if no messages within the `TargetQueueDepth` match a given subscription, no messages will be delivered to that subscription even if later messages in the transaction log match the subscription. Although AMPS does not consider messages that are currently inactive to be deliverable, AMPS will preserve queue delivery guarantees in the event that a message that is not currently deliverable is acknowledged before it enters the queue and becomes deliverable. To do this, AMPS keeps a list of acknowledgments that have been received for messages that are not in the queue. When a message that would be added to the active part of the queue has already been acknowledged, AMPS does not add that message to the queue. Applications that use `TargetQueueDepth` to limit memory growth should avoid a pattern where messages are acknowledged before they enter the queue to reduce memory consumption, CPU usage, and storage activity for acknowledgments that cannot yet be processed. Likewise, if a transfer request from another instance arrives for a message that is in the transaction log, but is beyond the limit set by the `TargetQueueDepth`, AMPS will process messages further into the queue until it reaches the message to be transferred. At that point, AMPS will transfer the message and stop processing further messages until either: * The current depth drops below the `TargetQueueDepth`, _or_ * AMPS once again receives a request for a message that is in the transaction log, but beyond the current active messages in the queue. :::info A `TargetQueueDepth` _cannot_ be set for a queue that specifies a `Priority` or a `BarrierExpression`. ::: --- # Configuring Queues in a SOW This section outlines the configuration parameters for queues. AMPS provides three distinct types of queues, each designed for specific use cases: `Queue`, `LocalQueue`, and `GroupLocalQueue`. ## Defining Queue Replication Type Queues are defined within the `SOW` element using the tags `Queue`, `LocalQueue`, or `GroupLocalQueue`. The tag that encloses a queue definition specifies the queue replication type. Full details can be found in the [Queue Replication Types](/docs/amps-user-guide/queues/getting-started-with-amps-queues.md#queue-replication-types) section. To summarize: * `Queue` tag: Defines a distributed queue. The queue may be consumed from any instance hosting the queue, and all instances that accept publishes for queue messages must host the queue. * `LocalQueue` tag: Defines a queue limited to the local instance only. This queue is independent of any other instance. * `GroupLocalQueue` tag: Defines a queue restricted to a subset of instances in a replicated set, typically all within the same group. This queue is hosted by some, but not all, instances in the replicated fabric. AMPS accepts `QueueDefinition` as a synonym for `Queue`. ## Queue Configuration Described below are the configuration items that apply to queues regardless of whether they are fully distributed, local, or distributed within a replication group. Expand each item for more details.
    `Name` (required) The name of the queue topic. This is the name that consumers subscribe to. If no `Name` is provided, AMPS accepts `Topic` as a synonym for `Name` in the `Queue` definition.
    `MessageType` (required) The message type of the queue.
    `UnderlyingTopic` A topic name or regular expression for the topic that contains the messages to capture in the queue. These topics must be recorded in a transaction log, and all must be of the same message type as the queue. Default: The `Name` of the queue. That is, if an `UnderlyingTopic` is not provided, the `UnderlyingTopic` defaults to the `Name` of the queue.
    `DefaultPublishTarget` (required if `UnderlyingTopic` contains regular expression characters) The topic to publish to when an application publishes a message to the queue. For simplicity, AMPS allows applications to publish messages to the queue, and for those messages to be routed to an `UnderlyingTopic`. The `DefaultPublishTarget` must be one of the topics included in the queue. This element is required if the `UnderlyingTopic` contains regular expression characters. Otherwise, this element is optional and defaults to the `UnderlyingTopic`.
    `LeasePeriod` The amount of time that a subscriber has ownership of the message before the message is returned to the queue. For `at-least-once` delivery semantics, the consumer must process and acknowledge the message within this lease period, or the message may be provided to another subscriber. The `LeasePeriod` is measured from the time that AMPS sends the message to the subscriber. Set the `LeasePeriod` to account for round trip network latency as well as the expected processing time for the subscribers. Default: unset (no expiration)
    `Semantics` The delivery semantics to use for this queue. Regardless of the delivery semantics, AMPS queues deliver a given message to a single subscriber at a time. When a subscriber fails to process a message (that is, the connection to the subscriber closes before the message is acknowledged, or the lease expires before the message is acknowledged), the semantics specify how the failure is handled. AMPS supports the following semantics:
    • `at-least-once` - With these semantics, you can guarantee that a message has been processed by at least one subscriber, as explained in the [Delivery Semantics](/docs/amps-user-guide/queues/understanding-amps-queuing#delivery-semantics) section. With this value, a subscriber must explicitly remove the message from the queue once the message is processed. If the subscriber connection closes before acknowledging the message, if the subscriber returns the message to the queue, or the lease expires, the AMPS server can deliver the message again to allow another attempt to process the message.
    • `at-most-once` - With these semantics, AMPS removes the message from the queue immediately when AMPS sends the message. This allows you to guarantee that no more than one subscriber will process the message, even if the subscriber that receives the message fails without acknowledging the message.
    Default: `at-least-once`
    `MaxBacklog` The maximum number of outstanding, unacknowledged messages in the queue at any one time. This parameter allows you to set limits on the number of pending messages from the queue overall. When the queue reaches the `MaxBacklog`, no incoming messages are delivered from the queue until a message is removed from the queue (either by expiring, or being acknowledged by a client). This parameter allows you to avoid overwhelming clients during periods of heavy activity. Notice that this does not set a limit of any sort on the capacity of the queue. This parameter allows you to limit the number of messages that the queue will make available to subscribers at a given time, but does not restrict the capacity of the queue to track messages. This backlog number is applied per instance of the queue. That is, each instance of AMPS that hosts an instance of a replicated queue will deliver messages up to the `MaxBacklog` if messages are available on that instance. Default: unset (no limit)
    `MaxPerSubscriptionBacklog` The maximum number of outstanding, unacknowledged messages in the queue for an individual subscription. This parameter allows you to avoid overwhelming a single subscriber during a period of heavy activity. Subscribers can declare the maximum number of messages that the subscription is prepared to lease at a given time. This maximum defaults to `1` when there is no maximum explicitly specified for a subscription. AMPS will lease the number specified in the subscription or the maximum set for the queue, whichever is lower. Notice that this does not set a limit of any sort on the capacity of the queue. This parameter allows you to limit the number of messages that the queue will make available for a single subscription at a given time, but does not restrict the capacity of the queue to track messages. Default: `1`
    `Expiration` Sets the queue default value for the length of time an individual message can remain in the queue before AMPS considers the message to be undeliverable. Messages may expire while a subscriber has a lease on the message. AMPS does not send an additional notification in this case. It is possible for a message to also have an expiration value assigned. In this case, the `ExpirationModel` parameter determines whether the queue value or the message value is used for an individual message. Default: unset (no limit)
    `ExpirationModel` Manages how AMPS applies the queue expiration period and the message expiration period to determine the expiration for the message. There are four models:
    • `queue` - Always use the queue expiration value for expiration, regardless of the value on the message.
    • `latest` - Use whichever expiration value is greatest; that is, use the value that will keep the message in the queue longest.
    • `earliest` - Use whichever expiration value is smallest; that is, use the value that will expire the message from the queue earliest.
    • `default` - Use the message expiration if the message expiration is set; otherwise use the queue expiration.
    Notice that an `ExpirationModel` may be set even if there is no `Expiration` set on the queue. In this case, the queue expiration is treated as though it is "unlimited" -- a message that uses the queue value will never expire, and the queue value will always be greater than a message value. Default: `default`
    `Filter` An AMPS `Filter` that is applied to the `UnderlyingTopic`. When a `Filter` is specified, only messages matching the `Filter` appear in the queue. By default, there is no filter and all messages from the `UnderlyingTopic` are presented in the queue.
    `RecoveryPoint` This option allows you to specify the point at which AMPS begins reviewing the transaction log to recover the state of the queue when AMPS restarts and there is no existing information about the state of the queue. By default, AMPS reviews the full log to determine the contents and state of the queue. If AMPS has a record of the last point in the transaction log at which all previous messages are acknowledged (stored in the `queues.ack` file), and the recovery point specified in the configuration file has not changed, AMPS will recover from that point rather than the configured `RecoveryPoint`, since using the cached information will reduce recovery time. The `RecoveryPoint` can be one of the following:
    • `epoch` - Recovery begins at the beginning of the transaction log.
    • `now` - Recovery begins at the time AMPS starts queue recovery, so only new messages are added to the queue.
    • `creation` - Recovery begins at the time the queue was created. In current releases of AMPS, this is identical to specifying `now`.
    • AMPS Bookmark - When an AMPS bookmark is provided, AMPS starts recovery at the specified bookmark.
    • ISO-8601 Timestamp - When a timestamp is provided, AMPS starts recovery at the specified timestamp. The timestamp must be provided in the format AMPS uses for timestamp bookmarks.
    Default: `epoch`
    `FairnessModel` AMPS provides different methods to distribute messages across active subscriptions:
    • `fast` - AMPS delivers to the first subscription found that can process the message.
    • `round-robin` - AMPS distributes to the next subscription found that can process the message.
    • `proportional` - AMPS delivers to the subscription with the lowest ratio of active messages to available backlog.
    Each instance of AMPS independently manages the fairness model for subscriptions on that instance. Fairness model information is not replicated across instances. Default: `proportional` for `at-least-once` queues, `round-robin` for `at-most-once` queues
    `Leasing` Ownership model for leased messages. AMPS supports the following models:
    • `strict` - AMPS allows a client to acknowledge (`sow_delete`) only messages that are leased to the client or currently unleased. If a client acknowledges a message leased to another client, there is no effect.
    • `sublet` - AMPS allows any client to acknowledge any message, regardless of whether another client has a lease on the message.
    Default: `sublet`
    `MaxDeliveries` Specifies an upper bound to the number of times AMPS may deliver a queue message before automatically expiring it. For example, if AMPS delivers a message twice and `MaxDeliveries` is set to `2`, the message will be expired if the subscriber disconnects or unsubscribes before acknowledging it. This counter is reset if the server restarts, and the counter is not replicated to other instances. Default: No maximum (`0`).
    `MaxCancels` Specifies a limit to the number of times a subscriber may cancel a lease on a message before it is expired. For example, if a message is canceled for the second time and `MaxCancels` is set to `2`, AMPS automatically expires the message instead of returning it to the message queue. This counter is reset if the server restarts, and the counter is not replicated to other instances. Default: No maximum (`0`).
    `Priority` Specifies the order in which messages will be distributed from the queue. When present, this element constructs a value that specifies the priority for messages in the queue. Higher priority messages are delivered first, regardless of the order in which messages have been published to the queue. The contents of the element can be either the name of a field in the message or an AMPS expression. Either way, the result is treated as an `unsigned long` value. For example, to order message delivery based on the `/price` field of the message, specify the following element: ```xml /price ``` To order message delivery based on the product of the `/price` field and the `/quantity` field in the message, specify the following element: ```xml /price * /quantity ``` This option cannot be specified on a queue if `BarrierExpression` is specified. Default: There is no default for this value. If not specified, the delivery order is the order in which messages were processed by this instance of AMPS.
    `BarrierExpression` Specifies the filter used to identify a barrier message (synchronization point) for this queue. When a message matches this expression, it will be delivered to all current subscribers on the queue when every previous message in the queue has been acknowledged. This provides a simple, reliable way to synchronize workers. For example, the following configuration item specifies that any message that contains a non-NULL `/isSyncPoint` value will be treated as a barrier message. ```xml /isSyncPoint IS NOT NULL ``` The `BarrierExpression` filter is evaluated when AMPS adds the message to the in-memory state of the queue, and is not re-evaluated (unless the in-memory state of the queue is rebuilt after an instance restart). This option cannot be specified on a queue if `Priority` is specified. Default: There is no default for this value. If not specified, no messages are considered to be barrier messages.
    ## Managing Queue Metadata Described below are the configuration items for managing queue metadata. Expand each item for more details.
    `FileBackedMetadata` Specifies whether AMPS should persist metadata about the queue in the journal directory. This could reduce the active memory footprint of an AMPS instance in cases where a queue has a large number of messages, but it is not being actively consumed. In cases where the queue has a large number of unacknowledged messages when AMPS is restarted, this may also improve recovery time. This option may also increase I/O to the journal filesystem as the file is created and maintained. This option can be unset or set to a value of `enabled`. Default: unset (which indicates that queue metadata will not be persisted)
    `TargetQueueDepth` The target number of messages to keep active state on. Providing this option limits the amount of state that AMPS keeps for unacknowledged messages. In most circumstances, the limit will be the number of messages specified in this parameter. If this instance receives a transfer request for a message that is in the transaction log, but outside the currently active set of messages, AMPS will process the queue until it reaches the message to be transferred. Messages that are present in the transaction log, but for which AMPS does not currently have active state, are not deliverable from the queue topic, cannot be queried, cannot be deleted using a `sow_delete` by filter, and so on. They are not yet in the set of messages that AMPS considers to be an active part of the queue. By default, this option is not set. When unset, all messages in the queue are active. If `FileBackedMetadata` is enabled, this option is typically unnecessary. In that case, AMPS will process the full state of the queue, but state that is not currently in use can be paged out to the file. This option cannot be set if `Priority` or `BarrierExpression` is specified. When this parameter is set, it should be set to a number that accounts for several seconds of traffic to the queue, considering replication and acknowledgment speeds. For example, if a queue is typically processed at a rate of 2500 messages per second within minimal replication delays, a reasonable minimum target value might be 12500 or 15000. Notice that, if subscribers use content filters to selectively process the queue, and no currently active messages match the subscription, no messages will be delivered (even if there are matching messages later in the transaction log that would be in the queue and match the subscription if they were active). It is also important to note that this option may increase CPU cost, latency, and disk activity for queues, since metadata for messages is loaded as messages are processed rather than when messages arrive from a publisher. Default: unset (no limit) Minimum: `1000`
    `DeferredAckExpiration` Specifies the amount of time for AMPS to retain information about an acknowledgment (`sow_delete`) message received when the corresponding message is not in the queue. This can occur during failover, when messages are received over replication, or in cases where an application that uses a publish store has been offline for an extended period of time. This element is configured as an interval, for example, `15m` or `2h`. The default value is generally recommended. However, in cases where an instance may receive large volumes of acknowledgments for messages that are not currently in the queue, and are not expected to arrive, setting this to a lower value may somewhat reduce the memory required for managing these acknowledgments. Default: `1d`
    ## Group Local Queue Configuration When a queue is defined as a `GroupLocalQueue`the following configuration item must be provided in addition to the mandatory elements above. Expand the item for more details.
    `InitialOwner` (required for `GroupLocalQueue`) For a group local queue, provides the instance `Name` of the instance that will own the message when the message is first published. This configuration element is required for a `GroupLocalQueue`. It is not supported for fully distributed queues or for local only queues (that is, a queue defined with the `Queue` or `LocalQueue` element). All instances in a replicated system that define this queue must define the same `InitialOwner`. Notice that AMPS does not validate that the `InitialOwner` exists and is reachable through replication from the current instance. When configuring a `GroupLocalQueue`, take care to validate the name and ensure that a replication path exists to and from that instance. There is no default for this element.
    When a queue is defined as a `GroupLocalQueue`, the following optional configuration element may be provided. Expand the item for more details.
    `GroupLocalQueueDomain` For a group local queue, provides a way to allow a queue that is hosted in instances that are in different replication groups to be identified as the same queue and function as a single distributed queue. By default, a `GroupLocalQueue` is identified using the queue `Name` and the `Group` of the AMPS instance that hosts the queue. Queues with the same identity are considered to be instances of the same distributed queue. When this element is defined, the `GroupLocalQueue` is identified using the queue `Name` and the `GroupLocalQueueDomain` rather than the `Group` of the instance that hosts the queue. This element was introduced in version 5.3.4. Previous versions do not consider the `Group` of the instance or the `GroupLocalQueueDomain`: instances of the queue with the same `Name` are considered to be the same distributed queue in those versions regardless of the `Group` that hosts the queue.
    The following configuration snippet shows one way to configure a queue: ```xml showLineNumbers MQ json ORDERS_.* ORDERS_DIRECT 60s 1d 3 ``` The following configuration snippet shows the configuration for a `GroupLocalQueue`. Any instance of the queue in the group named `CONSUME_INSTANCES` or that uses the `GroupLocalQueueDomain` tag of `CONSUME_INSTANCES` will be treated as an instance of the same queue. ```xml showLineNumbers GroupQueue json ORDERS_.* ORDERS_DIRECT CONSUME_INSTANCES ``` --- # Getting Started with AMPS Queues To add a simple queue to AMPS, add the following options to your configuration file. First, create a transaction log that will record the messages for the queue and the state of the queue, as described in [Record and Replay Messages](../txlog). You add the transaction log entry if your AMPS configuration does not already have one. Otherwise, you can simply add a `Topic` statement or modify an existing `Topic` statement to record the messages. The sample below captures any JSON messages published to the `Work` topic, and also tracks the state of the queue itself: ```xml showLineNumbers ... ./journals Work json WorkToDo json ... ``` Next, declare the queue topic itself. Queues are defined in the SOW element of the `AMPSConfig` file, as shown below: ```xml showLineNumbers ... WorkToDo json at-most-once Work ... ``` These simple configuration changes create an AMPS message queue. Notice that the `Topic` for the queue in this case is `WorkToDo`, which includes every message published to the underlying topic `Work`. You could also use a regular expression to include messages to more than one topic, or leave out the `UnderlyingTopic` to include only messages published to the topic with the same name as the queue. This simple queue provides each message that arrives for the queue to at most one subscriber. After AMPS delivers the message to one subscriber, AMPS removes the message from the queue without waiting for the subscriber to acknowledge the message. While it's easy to create a simple queue, AMPS offers a rich queuing model that is designed to meet a wide variety of queuing needs. The options are described in the following sections and the [Configuring Queues in a SOW](/docs/amps-user-guide/queues/configuring-queues-in-sow) section. ## Queue Replication Types AMPS supports three different replication types for a queue. The configuration tag used to define a queue specifies the replication type for the queue. | Configuration Tag | Replication Type | Description | Message Ownership | | ----------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | | `Queue` | Distributed queue | Fully distributed queue. |

    Instance that received the initial publish from an application.

    Other instances may request ownership.

    | | `LocalQueue` | Local queue |

    Queue exists on a single instance, delivery guarantees only apply to that instance.

    Cannot be replicated.

    |

    Instance that contains the queue.

    No other instance may request ownership.

    | | `GroupLocalQueue` | Queue replicated to specific instances, most often in the same replication `Group`. |

    Distributed queue on a specific set of instances.

    In version 5.3.4 and higher, the instances that contain the queue must all be in the same `Group` _or_ define the same `GroupLocalQueueDomain` tag.

    In earlier versions of AMPS, every queue with the same name in the replication fabric is treated as an instance of the same queue and must have the same definition and the same `InitialOwner`.

    |

    Instance specified in the `InitialOwner` tag for the queue.

    Other instances may request ownership.

    | By default, AMPS queues are _distributed queues_. That is, if the queue topic or the underlying topics are replicated, AMPS provides the queue delivery guarantees (that is, `at-most-once` or `at-least-once` delivery) as though all of the instances were delivering messages from a single queue. AMPS provides local queues (where each instance has a separate, independent queue) when a queue is defined with the `LocalQueue` tag. In this case, AMPS does not allow replication of the queue topic, and all management of the queue is local to the individual instance. AMPS also supports distributed queues that are restricted to a set of instances within a mesh of replicated instances. These queues are defined with the `GroupLocalQueue` tag, to indicate that the queue is restricted to a subset of the servers involved in replication. In this case, only the servers that define the `GroupLocalQueue` participate in message delivery for the queue. In addition, the group must choose the instance that will initially own messages for the queue, as described in the section on [replicated queues](../replication/queue\_replication) and [queue message ownership](../replication/queue\_replication.md#queue-message-ownership). In version 5.3.4 and later, the identity of a `GroupLocalQueue` is defined by the `Name` of the queue combined with the `GroupLocalQueueDomain`, if one is set, or the `Group` of the AMPS instance if the definition does not include a `GroupLocalQueueDomain` tag. In previous versions of AMPS, the identity of the queue is determined only by the `Name` of the group local queue, regardless of the instance that defines the queue. :::tip When defining a `GroupLocalQueue`, the instances that participate in the queue do not need to be in the same replication group: any instance in a replicated mesh that defines the queue can host an instance of the queue, regardless of whether the instance is in the same replication group or not. However, if the instances that host the queue are in different groups, the queue definitions should set the same `GroupLocalQueueDomain` to be considered to be the same queue. ::: Each instance of AMPS manages its own subscriptions and backlog, regardless of the replication model. Delivery guarantees are provided by managing the ownership of each message in the queue, as described in the [Replicated Queues](../replication/queue\_replication) section. Only messages owned by the current instance will be delivered to subscribers, and only the subscriptions on the current instance are considered for delivery fairness and limits such as `MaxBacklog` (the overall limit for leases from the queue). --- # Handling Unprocessed Messages Queuing applications occasionally encounter data that cannot be successfully processed or completed. AMPS allows you to configure actions that occur when a message expires from a message queue. You can create a dead-letter queue or topic by using the `amps-action-on-sow-expire-message` action in conjunction with the `MaxCancels` and `MaxDeliveries` options and the `cancel` and `expire` options on the `sow_delete` command. As an example, suppose your queue `Jobs` is used to coordinate computation jobs, and in the rare case where a job cannot be completed successfully, you would like AMPS to automatically move the job to another queue, `FailedJobs`. AMPS provides `MaxCancels` and `MaxDeliveries` options to specify how many attempts it should make to deliver a message to a subscriber: ```xml showLineNumbers ... Jobs json 3 10 FailedJobs json ``` In this configuration, AMPS will automatically expire a message from `Jobs` on the 3rd time it receives a `sow_delete` with the `cancel` operation for that message, or after the 10th unsuccessful delivery of the message. By default, expiring a message simply means the message is removed from the queue, but in this case, we also want the `FailedJobs` queue to receive the message so that it can be manually reviewed later. To accomplish this, AMPS provides the `amps-action-on-sow-expire-message` action which can be configured to do so: ```xml showLineNumbers ... amps-action-on-sow-expire-message Jobs json amps-action-do-publish-message FailedJobs json { "message": {{AMPS_DATA}}, "reason": "{{AMPS_REASON}}" } ... ``` With this action configured, any expired message from `Jobs` will result in a message automatically published to `FailedJobs` containing the original message inside the `message` element, and an AMPS-supplied reason code in the `reason` field. --- # Queue Subscriptions Compared to Bookmark Replays At first glance, a subscription to a queue can look very similar to a bookmark replay. In both cases, messages are provided in order from the transaction log. In both cases, a message that is not processed can be retrieved and redelivered. In both cases, AMPS allows a subscription to exactly specify the messages of interest using content filtering. There are a few main differences: * **Delivery Model** With a bookmark subscription, a message from a given topic can be delivered to any number of subscribers and processed multiple times. A given subscriber can replay the same messages any number of times, if needed. With a queue subscription, the server guarantees that once a message is processed, it is not delivered from that topic again. * **Delivery Limits** With a bookmark subscription, by default AMPS will deliver messages to the subscription as fast as the physical hardware, disk, and network allow. With a queue subscription, AMPS intentionally limits the number of messages delivered to a subscriber, to help ensure that messages are processed in a timely manner. Since each message should be delivered only once, delivering every message to a single subscriber would run the risk of having a slow consumer holding messages that it is unable to process, while a faster consumer sits idle. * **Acknowledgment** With a bookmark subscription, the application and client libraries are responsible for tracking which messages have been processed and providing the correct recovery point. There is no state stored in the AMPS server as to which messages have been consumed by a given subscription. With a queue subscription, the AMPS server tracks whether a given message has been processed. A queue subscriber must notify the server that it is finished with a given message and is ready for more work. The AMPS client libraries contain support for making this process easy, as well as the ability to batch acknowledgments to provide high throughput while maintaining efficient processing and the guarantees for queues. --- # Replacing Queue Subscriptions Queues support the `replace` option for subscriptions. As with subscriptions to other topics, queue subscriptions can replace the content filter, the topic, the options, or all of the above. Replacement is atomic. The queue consumer is guaranteed that, after the replace occurs, only messages that match the new subscription will be delivered. Replacing queue subscriptions differs from unsubscribing and resubscribing with new parameters in two ways: 1. AMPS does not break message leases or adjust the number of currently-unacknowledged messages for the subscription, even if the messages no longer match the current subscription. AMPS makes no assumptions about the state of the messages, and requires the subscriber to acknowledge them or allow the lease to expire. 2. AMPS may change the maximum backlog for the subscription if either the `max_backlog` option _or_ the topic for the subscription has changed. AMPS adjusts the backlog using the same logic as when the subscription was entered: the maximum backlog will be the smaller of the option set by the consumer or the limit on the queue. This can result in a situation where the consumer has more messages leased than the current maximum for the subscription, and no new messages will be delivered until that number drops below the current maximum. For example, if the consumer has a requested `max_backlog` of 10 and updates a subscription from a queue with a configured maximum of 10 to a queue with a configured maximum of 5, the new backlog for the subscription will be 5. However, the consumer may still have 10 messages outstanding. In all other ways, AMPS behaves as though the replaced subscription was a new subscription to the queue. --- # Understanding AMPS Queuing AMPS message queues take advantage of the full historical and transactional power of the AMPS engine. Each queue is implemented as a view over an underlying topic or set of topics. Each of the underlying topics must be recorded in a transaction log. Publishers publish to the underlying topic, and the messages are recorded in the transaction log. Consumers simply subscribe to the queue. AMPS tracks which messages have been delivered to subscribers and which messages have been processed by subscribers. AMPS delivers the next available message to the next subscriber. Unlike traditional queues, which require consumers to poll for messages, AMPS queues use a subscription model. In this model, each queue consumer requests that AMPS provide messages from the queue. The consumer can also request a maximum number of messages to have outstanding from the queue at any given time, referred to as the backlog for that consumer. When a message is available, and the consumer has fewer messages outstanding than the backlog for that consumer, AMPS delivers the message to the consumer. This improves latency and conserves bandwidth, since there is no need for consumers to repeatedly poll the queue to see if work is available. In addition, the server maintains an overall view of the consumers, which allows the server to control message distribution strategies to optimize for latency, optimize to deliver to clients with the most unused capacity, or optimize for general fairness. The following diagram presents a simplified view of an AMPS queue: As the diagram indicates, a queue tracks a set of messages in the transaction log. The messages the queue is currently tracking are considered to be in the queue. When the queue delivers a message, it marks the message as having been delivered (shown as _leased_ in the diagram above). Messages that have been processed are no longer tracked by the queue (for example, the message for the order 1 in the diagram above). When a message has been delivered and processed, that event is recorded in the transaction log to ensure that the queue meets the delivery guarantees even across restarts of AMPS. Since queues are implemented as views over underlying topics, AMPS allows you to create any number of queues over the same underlying topic. Each queue tracks messages to the topic independently, and can have different policies for delivery and fairness. When a queue topic has a different name than the underlying topic, you can subscribe to the underlying topic directly, and that subscription is to the underlying (non-queue) topic. When a queue topic has the same name as the underlying topic (the default), all subscriptions to that topic are to the queue. (Notice that bookmark subscriptions to a queue are pub/sub subscriptions that replay from the underlying topic in the transaction log, so the behavior in that case is the same as if the subscription was directly to the underlying topic.) Likewise, AMPS queues work seamlessly with the AMPS entitlement system. Permissions to queues are managed the same way permissions are managed to any other topic, as described in the [Entitlement](../securing/entitlement) section, _except_ that read permission to a queue also grants the ability to acknowledge messages (though not to publish messages to the queue). While a message is in a queue, AMPS does not retain an extra copy of the message. Instead, AMPS retains in memory a small data structure indicating the state of the message and the position of the message in the transaction log. The amount of memory consumed by a queue is approximately 200 bytes per message, regardless of the size of the messages. AMPS queues provide a variety of options to help you tailor the behavior of each queue to meet your application's needs. ## Delivery Semantics AMPS queues deliver a message to a single subscriber at a time. In the most common case, a message is delivered to exactly one subscriber, and that subscriber processes the message. In the case that a subscriber does not successfully process a message, AMPS provides two delivery semantics to precisely specify the handling of the unprocessed message: * With `at-least-once` delivery, AMPS delivers the message to one subscriber at a time, and expects that subscriber to explicitly remove the message from the queue when the message has been received and processed. With this guarantee, each message from the queue must be processed within a specified timeout, or lease period. AMPS tracks the amount of time since the message was sent to the subscriber. If the subscriber has not responded by removing the message within the lease period, AMPS revokes the lease and the message is available to another subscriber. AMPS allows you to set limits on the number of times a message is made available to another subscriber, using the `MaxCancels` and `MaxDeliveries` configuration options. In this model, receiving a message is the equivalent of a non-destructive get from a traditional queue. To acknowledge and remove the message, a subscriber uses the `sow_delete` command with the bookmark of the message. Leases are broken and messages are returned to the queue if the lease holder disconnects from AMPS. This ensures that, if a message processor fails or loses its connection to AMPS, the message can immediately be processed by another message processor. * With `at-most-once` delivery, AMPS removes the message from the queue as soon as the message is sent to a subscriber. However, the subscriber still needs to acknowledge that the message was processed, so that AMPS can track the subscription backlog, as described below. In this model, receiving a message is the equivalent of a destructive get from a traditional queue. The message is immediately removed by AMPS, and is no longer available in the queue. ### Choosing Delivery Semantics to Handle Failures Regardless of the delivery semantics you choose, during typical message processing from a queue, AMPS delivers each message in the queue to a single subscriber, with no duplicates or redelivery. The difference between `at-most-once` and `at-least-once` semantics is important in cases where a failure happens. With `at-most-once` delivery, the message will not be redelivered even if the subscriber fails to process the message. With `at-least-once` delivery, the message will be redelivered to subscribers until the message is either explicitly acknowledged, explicitly removed, or expired based on the queue policy for expiration or retries. With either setting, a message is delivered exactly once, to a single subscriber, during normal processing. Consider the following recommendations when deciding how you would like AMPS to handle cases where the subscriber that receives the message fails to process the message: 1. For ephemeral or lower-value data, where message loss in the case of failure is preferable to duplicating message delivery, consider `at-most-once` semantics. For example, processing a stream of events from sensors might fall into this category: each message should be processed once, in order, during normal operation, but missing a single data point from time to time may be less disruptive than reconciling duplicates. 2. For higher value data, where duplicate message delivery in the event of failure is preferable to message loss, consider `at-least-once` semantics. For example, if inserting the same data into a database twice would simply be an in-place update of the same record with the same information, having a duplicate insert happen occasionally in the event of failure may be preferable to losing a message when a failure occurs. 3. For higher value data, where manual intervention is required to reconcile messages when there is a question as to whether the message has been correctly processed, consider using `at-least-once` message delivery with a `MaxDeliveries` setting and an action to move failed messages to a dead-letter queue for manual reconciliation. 4. For higher value data that is part of a transactional dataflow, where processors maintain state, it can be useful to include information about the message processed in the transaction. This can be used with `at-least-once` messaging to guarantee that the message is only processed once: if a processor receives a message that is already marked as processed in the transactional store, the processor knows that the message has already been acted on and should be acknowledged without further processing. This is the most common pattern used to produce the result that each message is processed "exactly once". This pattern involves additional overhead and processing, trading off increased work in the application for stronger guarantees that a message is only processed one time. ## Subscription Backlog For efficiency, queues in AMPS use a push model of delivery, providing messages to consumers when the message becomes available rather than requiring the consumer to poll the queue. To manage the workload among consumers, AMPS queues keep track of a _subscription backlog_. This backlog is the number of messages that have been provided to an individual subscription that have not yet been acknowledged. This backlog helps AMPS provide strong delivery guarantees while still optimizing for high throughput processing. AMPS calculates the subscription backlog for each subscription by calculating the _minimum_ of the following: * The _minimum_ `MaxPerSubscriptionBacklog` setting for the queues matched by the subscription _or_ * The `max_backlog` specified on the subscribe command Notice that, if a subscriber does not provide a `max_backlog` on a subscription, AMPS defaults to a `max_backlog` of `1`. In practical terms, this means that an application must explicitly specify a backlog to be able to receive more than one message from a queue at a time, regardless of the queue configuration. Subscribers request a `max_backlog` by adding the request to the options string of the `subscribe` command. For example, to request a `max_backlog` of 10, a subscriber includes `max_backlog=10` in the options for the command. :::info To improve concurrency for subscribers, 60East recommends using a backlog of at least `2`. This allows efficient pipelined delivery, as the consumer can be processing one message while the previous message is being acknowledged. With a `max_backlog` higher than `1`, the consumer never needs to be stopped waiting for the next message from the queue. The optimum backlog depends on the number of subscribers, network speed, processing time, and so on. 60East recommends testing a realistic workload with a variety of backlog settings to determine a good setting for your application. (When testing, be sure that the `MaxPerSubscriptionBacklog` setting for the queue is equal to or larger than the backlog being tested.) ::: ## Delivery Fairness When a queue provides `at-least-once` delivery, AMPS provides three different algorithms for distributing messages among subscribers. Each algorithm has different performance and fairness guarantees. For `at-most-once` delivery, AMPS supports only the `round-robin` method of distributing messages. | Algorithm | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `fast` |

    This strategy optimizes for the lowest latency.

    AMPS delivers the message to the first subscription found that does not have a full backlog. With this algorithm, AMPS tries to minimize the time spent determining which subscription receives the message without attempting to distribute messages fairly across subscriptions.

    | | `round-robin` |

    This strategy optimizes for general fairness across subscriptions.

    AMPS delivers the message to the next available subscription that does not have a full backlog. With this algorithm, AMPS delivers messages evenly among the subscribers that have space in their backlog.

    | | `proportional` |

    This strategy optimizes for delivery to subscriptions with the most unused capacity.

    AMPS delivers the message to the subscription that has the highest proportion of backlog capacity unused. AMPS determines this by taking the ratio of unacknowledged messages to the maximum backlog.

    For example, if there are three active subscribers for the queue, with backlog settings and outstanding messages as follows:

    • Subscriber `Inky`: `max_backlog=2`, currently leased 1 message
    • Subscriber `Blinky`: `max_backlog=4`, currently leased 3 messages
    • Subscriber `Clyde`: `max_backlog=10`, currently leased 4 messages

    In this case, with `proportional` delivery, a new message for the queue will be delivered to Clyde, since that subscriber has only filled 40% of the backlog, as compared with 50% for Inky and 75% for Blinky.

    If more than one subscription has the same unused capacity, AMPS delivers the message to the first subscription found with that capacity.

    | AMPS defaults to `proportional` delivery for `at-least-once` queues and defaults to `round-robin` (the only valid delivery model) for `at-most-once` queues. :::info Each instance of AMPS manages delivery strategy from among the messages that it currently owns and the subscriptions that are present on that instance. Delivery strategies apply only to a single instance, and are not applied across instances. ::: ## Acknowledging Messages Subscribers must acknowledge each message to indicate to AMPS that a message has been processed. The point at which a subscriber acknowledges a message depends on the exact processing that the subscriber performs and the processing guarantees for the application. In general, applications acknowledge messages at the point at which the processing has a result that is durable and which would require an explicit action (such as another message) to change. Some common points at which to acknowledge a message are: * When processing is fully completed. * When work is performed that would require a compensating action (that is, when information is committed to a database or forwarded to a downstream system). * When work is submitted to a processor that is guaranteed to either succeed or explicitly indicate failure. To acknowledge a message, the subscriber typically uses the acknowledge convenience methods in the AMPS client. These commands issue a `sow_delete` command with the bookmark from the message to acknowledge. AMPS allows subscribers to acknowledge multiple messages simultaneously by providing a comma-delimited list of bookmarks in the `sow_delete` command: the AMPS clients provide facilities to batch acknowledgments for efficiency. AMPS allows an application to acknowledge messages by providing a filter on a `sow_delete` command. In this case, the `sow_delete` acknowledges all messages that match the filter, regardless of whether the application that sends the command has a current lease on a given message or not. (The `Leasing` parameter on the queue specifies whether AMPS allows a client to successfully acknowledge messages that it does not currently have leased.) For queues that use the `at-least-once` delivery model, there are two additional options available for acknowledging messages. * **To return a message to the queue without processing it**, the subscriber provides the `cancel` option on the acknowledgment. In this case, AMPS returns the message to the queue just as though the lease had expired. If the message is eligible for redelivery (that is, it has not exceeded the maximum time or maximum cancels configured for the queue), it is redelivered. Otherwise, the message is expired from the queue. The `MaxCancels` option allows you to configure how many times a message can be returned to the queue for redelivery before AMPS expires the message. * **To immediately remove a message from the queue**, the subscriber provides the `expire` option. In this case, AMPS does not return the message to the queue for redelivery, but instead immediately expires the message from the queue and triggers any configured `amps-action-on-sow-expire-message` actions that monitor the queue. The `expire` option can be especially helpful if your application can determine that the message cannot be successfully processed (for example, the message is unparseable or contains invalid data). Options for acknowledging messages delivered from `at-least-once` queues: | Option | Result | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | \ | Message is considered to be successfully processed and removed from the queue. | | `cancel` |

    Message is returned to the queue.

    The count of cancels for this message is incremented, and if the count is greater than the configured `MaxCancels` for the queue, the message is expired.

    | | `expire` | Message is immediately expired from the queue. | For `at-most-once` queues, these options to acknowledgments have no effect, since the message is guaranteed not to be redelivered even if processing fails. ## Persisting Metadata for Queues A queue can, optionally, save the state of the queue as metadata in the journal directory. This can reduce the active memory footprint required for large queues, since the state of the queue can be paged out if necessary. This can also potentially improve recovery time if there are a large number of unacknowledged messages in the queue when AMPS starts. To persist metadata for a queue, set the `FileBackedMetadata` option to `enabled` for the queue. :::info When `FileBackedMetadata` is enabled, queue state that is not currently needed can be paged out to the file. This can substantially reduce the active memory needed for large queues that are infrequently used. Notice that the state will only be paged out as necessary. While this option increases AMPS ability to manage queue state in low-memory situations, AMPS will still retain state in memory if memory is available to improve performance. ::: ## Optional Message Delivery Behaviors AMPS queues provide the following optional message delivery behaviors: * A `Priority` can be specified for a queue, in which case AMPS delivers messages in priority order rather than the order in which the instance received the messages. * A `BarrierExpression` can be specified to provide a synchronization point. When a message matches the expression, all previous messages must be acknowledged before AMPS will deliver the message that matches the expression, or subsequent messages. These options are described in more detail in the [Advanced Queue Configuration](advanced-queue-configuration) section. ## Message Flow for Queues The message flow for AMPS queues is as follows. The message flow differs depending on whether the queue is configured for `at-most-once` delivery or `at-least-once` delivery. When the queue is configured for `at-most-once` delivery: 1. A publisher publishes a message to an underlying topic. 2. The message becomes available in the queue. 3. For a message that is not a barrier message, the message is published to a subscriber when: * There is a subscription that matches the message, and the subscriber is entitled to see the message * The message is the oldest message in the queue that matches the subscription (if no `Priority` is specified for the queue) or it has the highest priority value for messages in the queue and is the oldest message at that priority (if `Priority` is specified). * The subscription has remaining capacity in its backlog * The subscription is the next subscription to receive the message as determined by the delivery fairness for the queue AMPS removes the message from the queue when the message is published. 4. For a message that is a barrier message, AMPS does not deliver the message until all previous messages have been acknowledged. AMPS does not deliver messages after the barrier message until the barrier message has been delivered. 5. If no subscription has requested the message, and the message has been in the queue longer than the `Expiration` time, AMPS removes the message from the queue. With AMPS queues, message expiration is considered to be a normal way for the message to leave the queue, and is not considered an error. The expiration time is calculated based on the time that the message was received on the local instance, the expiration time set on the individual message (if any), and the `ExpirationModel` for the queue. 6. The subscriber processes the message, and acknowledges the message when processing is finished to indicate to AMPS that the subscriber has capacity for another message. When the queue is configured for `at-least-once` delivery: 1. A publisher publishes a message to an underlying topic. 2. The message becomes available in the queue. 3. For a message that is not a barrier message, the message is published to a subscriber and marked as leased/unavailable when: * There is a subscription that matches the message * The message is the oldest message in the queue that matches the subscription (if no `Priority` is specified for the queue) or it has the highest priority value for messages in the queue and is the oldest message at that priority (if `Priority` is specified). * The subscription has remaining capacity in its backlog * The subscription is the next subscription to receive a message as determined by the delivery fairness for the queue AMPS calculates the lease time for the message and provides that time to the subscriber along with the message. 4. For a message that is a barrier message (that is, the queue has a `BarrierExpression` specified and the message matches that expression), AMPS does not deliver the message until all previous messages have been acknowledged. AMPS will not deliver messages after the barrier message until the barrier message is delivered. 5. If the message has been in the queue longer than the `Expiration` time, and there is no current lease on the message, AMPS removes the message from the queue. With AMPS queues, message expiration is considered to be a normal way for the message to leave the queue, and is not considered an error. The expiration time is calculated based on the time that the message was received on the local instance, the expiration time set on the individual message (if any), and the `ExpirationModel` for the queue. 6. If a subscriber has received the message, but has not removed the message from the queue at the time the lease expires, AMPS returns the message to the queue (by marking it as unleased/available) if the message has been in the queue less than the `Expiration` time, and it has been delivered fewer than the `MaxDeliveries` limit. If the message has been in the queue longer than the `Expiration` time or has been delivered more than the number of times specified by `MaxDeliveries`, AMPS removes the message from the queue when the lease expires. 7. The subscriber processes the message, and removes the message from the queue by acknowledging the message (which is translated by the AMPS client into the appropriate `sow_delete` command). ## Startup and Recovery for Queues When the metadata for a queue is not persisted to a file, or if that file is not fully up to date with the transaction log, AMPS recovers the current state of a queue from the transaction log when the instance starts. The point at which AMPS chooses to start recovery for a given queue is as follows: * If this is a new queue, or the configured `RecoveryPoint` for the queue has changed since the last time AMPS started, begin queue recovery at the configured recovery point. (This defaults to the beginning of the transaction log if no explicit `RecoveryPoint` is configured for the queue.) * Otherwise, start from the most recent point in the transaction log at which all previous messages have been acknowledged. This recovery point is stored in the `queues.ack` file in the journal directory for all queues in the instance. If that recovery point is older than the journals currently available, begin from the beginning of the transaction log. To recover the state of the queue, AMPS proceeds through the journal file from the recovery point and restores the state of the queue based on the operations in the transaction log for that queue. AMPS determines the oldest recovery point for all of the queues in the instance, and reads journals from that point forward to the end of the transaction log. AMPS rebuilds the state of each queue as the replay progresses past the recovery point for that queue. --- # Message Queues AMPS includes high performance queuing built on the AMPS messaging engine and transaction log. AMPS message queues combine elements of classic message queuing with the advanced messaging features of AMPS, including content filtering, aggregation and projection, historical replay, and so on. AMPS message queues help you easily solve some common messaging problems: * Ensuring that a message is only processed once. * Distributing tasks across workers in a fair manner. * Ensuring that a message is delivered to and processed by a worker. * Ensuring that when a worker fails to process a message, that message is redelivered. * Setting limits for retries in processing a message, and taking an action when those limits are exceeded (such as moving the message to a dead-letter queue). While it's possible to create applications with these properties by using the other features of AMPS, message queues provide these functions built into the AMPS server. In addition, message queues allow you to: * Replicate messages between AMPS instances while preserving delivery guarantees. * Create views and aggregates based on the current contents of a queue. * Filter messages into and out of a queue. * Provide a single published message to multiple queues. * Aggregate multiple topics into a single queue. :::tip Use message queues when it is important to ensure that a message is processed once, by a single consumer. When it is important to distribute messages to a number of consumers, use the AMPS publish/subscribe delivery model rather than a message queue. ::: The following diagram illustrates a simple usage of a queue to distribute work across three consumers: This diagram shows a simple use of AMPS queues to distribute work. In the diagram, the transaction log is configured to record a topic named `Work`. AMPS is also configured with a queue named `WorkToDo`, which is based on the underlying topic `Work`. The publisher publishes three messages to the topic `Work`, and AMPS includes those messages in the `WorkToDo` queue. Each message is delivered to one of the three subscribers to the `WorkToDo` queue. Unlike pub/sub messaging, each subscriber only receives one message, and each message is delivered to only one subscriber. Notice that, even though AMPS provides queue semantics over the `WorkToDo` topic, the messages are recorded in the transaction log once, in the `Work` topic. Other subscribers could subscribe to the `Work` topic to receive the full stream of messages, or do a bookmark replay over the `Work` topic to recreate the message flow or audit the messages published to that topic. Also notice that, while the `Work` topic (the underlying topic) must be recorded in the transaction log, there is no need to define the `Work` topic in the SOW section of the configuration file. Queues use the transaction log to determine which messages to enqueue, and when a message is acknowledged, the acknowledgment is also recorded in the transaction log. --- # Configuring Incoming Replication Transports `Transports` enable connections to the AMPS instance. For replication, messages always flow from a replication destination to a transport defined in the `Transport` section. To receive incoming messages via replication, an instance must define the `amps-replication` transport in its list of `Transports`. Note that an instance can have _only one_ incoming replication transport. Additionally, transports for application use must be configured separately, as an `amps-replication` transport is used solely for replication purposes. For more information on `Transports`, see the [Configuring Transports](/docs/amps-user-guide/transports/configuring-transports) section. For inbound replication connections secured with SSL, use the `amps-replication-secure` transport. This transport type requires a certificate and private key to be set. :::tip This section describes configuring an **AMPS replication transport**, that is, a transport that allows the flow of incoming replication messages from a replication source. Configuring a replication source involves defining a `Destination` within the `Replication` block, which specifies replication targets. See the [Configuring Outgoing Replication Destinations](/docs/amps-user-guide/replication/config-outgoing-replication) section and the [Replicating Messages Between Instances](/docs/amps-user-guide/replication) section for details. ::: :::warning The configuration details below apply specifically to the transport types `amps-replication` and `amps-replication-secure`, which are used for setting up replication. To configure a `Transport` for handling application connections, see [Configuring Transports](/docs/amps-user-guide/transports/configuring-transports). ::: ## Transport: Defining Incoming Messages The `amps-replication` `Transport` defines an incoming flow of messages via replication. --- **`Transport`(amps-replication)** (required) * Defines how AMPS accepts connections for incoming replication messages. * Required parent tag, which is defined to receive incoming messages via replication. * For replication, `Type` should be `amps-replication` or `amps-replication-secure` (for connections that use SSL). --- ### Replication Transport Configuration Described below are the configuration items available for `Transport` when configuring replication. Expand each item for more details.
    `Name` (required) The name to use for this `Transport`. This name appears in the AMPS log for messages related to the transport. When the `Type` of the `Transport` is `amps-replication` or `amps-replication-secure`, 60East recommends that the `Name` of the `Transport` match the value of the `Type` to help make debugging replication easier. There is no default for this value.
    `Type` (required) Specifies the type of connection to make. The `Type` of a replication `Transport` must always be either `amps-replication` or `amps-replication-secure`. When the `Type` is set to `amps-replication-secure`, the incoming connection will use TLS/SSL. The `Type` of the outgoing connection must match the `Type` of the `Transport` that this instance is connecting to.
    `InetAddr` The port on which AMPS will listen for this transport. This element can also specify an IP address, in which case AMPS listens only on that address. If no IP address is specified, AMPS listens on all available addresses. Starting with version 5.3.3, both IPv4 and IPv6 address formats are fully supported for use with specifying the network address of a transport. If no address is specified and the host supports IPv6, AMPS will listen for incoming connections on both IPv4 and IPv6 protocols. If you wish to limit AMPS to listen for addresses of only a specific IP protocol you may specify the `ANY` address for that protocol. For example: `0.0.0.0:9007` will cause AMPS to listen on port `9007` for only IPv4 addresses. `[::]:9007` will cause AMPS to listen on port `9007` for only IPv6 addresses. This element is not required for transports of the `amps-unix` `Type` but is required for all other `Type` values.
    **TLS/SSL Parameters** Described below are the configuration items needed to set up and enable TLS/SSL, if the transport `Type` is `amps-replication-secure`. Expand each item for more details.
    `Certificate` (required if `Type` is `amps-replication-secure`) A `Transport` element that specifies `amps-replication-secure` as the transport type must provide a certificate to use for the TLS/SSL connection. There is no default for this option.
    `PrivateKey` (required if `Type` is `amps-replication-secure`) A `Transport` element that specifies `amps-replication-secure` as the transport type must provide a private key to use for the SSL connection. There is no default for this option.
    `Ciphers` (optional, only supported if `Type` is `amps-replication-secure`) A `Transport` element that specifies `amps-replication-secure` as the transport type may provide a cipher list to use for the SSL connection. When provided, this connection is restricted to the specified ciphers. Default: No restriction on the ciphers supported by the SSL implementation.
    `VerifyClient` (optional, only supported if `Type` is `amps-replication-secure`) When set to `true`, this destination will verify certificates provided for TLS using the `CAFile` or `CAFileLocation` specified. Default: `false`
    `CAFile` (one of `CAFile` or `CAPath` must be specified if `VerifyClient` is `true`) When `VerifyClient` is set to `true`, specifies a `.pem` file containing trusted certificates used to verify certificates provided by the other side of the replication connection. There is no default for this option.
    `CAPath` (one of `CAFile` or `CAPath` must be specified if `VerifyClient` is `true`) When `VerifyClient` is set to `true`, specifies a path to a directory containing `.pem` files that contain trusted certificates used to verify certificates provided by the other side of the replication connection. When this parameter is provided and `VerifyClient` is set to `true`, AMPS will use every `.pem` file in the directory for verification. There is no default for this option.
    ### Sample Replication Transport Configuration This section shows a sample that configures an AMPS instance to receive messages via replication. ```xml showLineNumbers any-tcp tcp 9007 amps amps-replication amps-replication localhost:10004 ``` --- # Configuring Outgoing Replication Destinations An AMPS replication target is defined in the `Replication` section of the configuration file, which contains one or more `Destination` blocks specifying unique replication targets. To replicate messages to another instance, the configuration file must include at least one `Destination`. Multiple `Destination` configurations can be defined, with each one specifying a single outgoing replication connection and message flow from the AMPS instance. :::tip This section describes configuring an instance as a **replication source**, that is, as an instance that provides messages to another instance. Configuring an instance to _receive_ replication connections is similar to configuring it for application connections — both require creating a `Transport` to handle incoming connections. See the [Configuring Incoming Replication Transports](/docs/amps-user-guide/replication/config-incoming-replication) section and the [Configuring Transports](/docs/amps-user-guide/transports/configuring-transports) section for details. ::: For further details on replication, refer to other parts of this section - [Replicating Messages Between Instances](/docs/amps-user-guide/replication). Additionally, the [Highly Available AMPS Installations](/docs/amps-user-guide/ha) section describes how to use the features of AMPS, including replication, to create highly-available AMPS installations. ## Destination: Defining Outgoing Messages Each `Destination` defines an outgoing flow of messages. --- **`Destination`** (required) * Defines a destination for outgoing replication messages. * Required parent tag, which defines a unique replication target. * The `Destination` block must include all required configuration items listed below. --- ### Destination Configuration Described below are the configuration items available for `Destination`. Expand each item for more details.
    `SyncType` (required) Defines whether the destination is considered when determining if a message is safely replicated and fully persisted before acknowledging the message to the source. There are two acknowledgment modes: * `sync` This `Destination` must acknowledge a message as persisted for this instance to acknowledge the message as safely persisted. * `async` This `Destination` does not need to acknowledge a message as persisted for this instance to acknowledge the message as safely persisted. There is no other behavioral difference between `sync` and `async` acknowledgment. That is, there is no difference in replication speed, replication priority, or the requirement to replicate messages to the destination. However, a destination using `sync` acknowledgment can be considered safe for failover from this instance, while an `async` destination should be considered to be in an unknown and potentially unsafe state for failover. To avoid publishers having to retain messages for an extended period of time if an instance is offline, it is possible to downgrade a `Destination` configured to use `sync` acknowledgment to use `async` acknowledgment. When a `Destination` is downgraded to `async` acknowledgment, it must be considered unsafe for failover See [Downgrading Acknowledgments for a Destination](/docs/amps-user-guide/replication/configuring-replication.md#downgrading-acknowledgments-for-a-destination) for details.
    `Transport` (required) The message type, network location, and connection details for making an outgoing connection to replicate messages. AMPS supports multiple `Transport` items within a `Destination`. When multiple Transports are provided, AMPS interprets these as transports for identical redundant servers, listed in priority order. If AMPS cannot connect to any of the internet addresses in a transport, AMPS tries the next `Transport`, in the order in which the `Transport` items appear in the file. When AMPS has tried all of the `Transport` items, AMPS tries again at the beginning of the list of transports. To provide failover, use multiple `InetAddr` elements within a single `Transport` for servers that can use the same `Authenticator` context (that is, the same credentials provided with the same authentication scheme). Use multiple `Transport` elements if the failover servers require different authentication. For details on the contents of the `Transport` element, see the section on [Transports in Destinations](config-outgoing-replication#transports-in-destinations) below.
    `Group` (required) The group that the downstream destination is a member of. The `Group` of the downstream instance must match the `Group` specified in this destination, or AMPS reports an error and will not replicate to that destination. There is no built-in default for this value. AMPS requires that a destination have a `Group` defined. If a `Name` is specified, and no `Group` is specified, AMPS will use the value of the `Name` as the value for the `Group`. This behavior is for convenience to match the behavior of an AMPS instance when no `Group` is specified at the instance level configuration. A `Group` is required. However, if a `Name` is specified for this `Destination` and no `Group` is specified, AMPS uses the value of the `Name` as the value of the `Group`. (Likewise, if `Group` is specified and no `Name` is specified, AMPS will use the `Group` value as the `Name` for this destination.) Notice that the `Name` must be unique within an AMPS replication fabric. If your replication configuration requires more than one `Destination` that replicates to the same `Group`, and does not want AMPS replication to treat all of those servers as identical, use the `Name` element instead of the `Group` element.
    `Name` The name of the destination. This name appears in the AMPS logs when AMPS logs a message about this destination. The `Name` must be unique in the AMPS replication fabric. When not present, AMPS uses the `Group` provided as the destination `Name`. The `Name` should be either the `Name` or the `Group` of the remote instance. 60East recommends setting the `Name` only when your replication configuration replicates to more than one instance in a given group and the configuration does not need to treat the servers within the group as interchangeable. If it is important to replicate to a specific AMPS instance, rather than any server in the `Group`, set the `Name` rather than the `Group`, and use the `Name` of that instance. For example, if you have three servers in the `AMPS-LA` group, the server `AMPS-LA-1` would have separate `Destination` configurations for `AMPS-LA-2` and `AMPS-LA-3`. Those `Destination` configurations would use the `Name` of the remote server (`AMPS-LA-2` or `AMPS-LA-3`) rather than the `Group` that is common to all of the servers. There is no default for this value. If a `Group` is specified and no `Name` is specified, AMPS uses the value of the `Group` as the `Name` of the destination.
    `Topic` Defines one or more topics to replicate to this destination. A `Destination` may contain any number of `Topic` elements. All `Topic` elements will be used to determine the topics to be replicated to the destination. That is, a topic that matches any of the `Topic` elements will be replicated. See the following section [Topics in Destinations](config-outgoing-replication#topics-in-destinations) for details.
    `PassThrough` Specifies source groups to pass through to this destination. The value of this element is a regular expression which is matched against the group name of the instance that sent the replication message to this instance. When the regular expression matches, the replication message is eligible for passthrough, and will be sent to the destination if the `Topic` specifications match the message. For installations that involve more than two AMPS instances, or installations that use queues, a `PassThrough` specification may be necessary for replication to distribute the full set of messages in this instance. See the [PassThrough Replication](/docs/amps-user-guide/replication/passthrough-replication) section for details. Using a regular expression that matches all groups (such as `.*`) provides the highest level of reliability. In some topologies, it may be possible to further refine the groups specified in a `PassThrough` directive to reduce bandwidth by partially replicating the local transaction log. Default: There is no default for this value. If no value is configured, then only messages published directly to this instance from an application will be replicated. If there are more than two instances in the set of instances that replicate to each other, it is strongly recommended to set `PassThrough` to cover all of the groups that should have the same messages.
    `Compression` Specifies whether to use compression for this destination. When set to `enabled`, AMPS compresses traffic to this destination. Default: `disabled`
    `CompressionType` Specifies the library to use for compression for this destination. This option has no effect unless the `Compression` option is set to `enabled`. Supported values for this release are `zstd` and `zlib`. Default: `zlib`
    `AckConflationInterval` Specifies the interval to use for conflating acknowledgment messages from this destination. This value must be an interval of less than 1 second. See the AMPS User Guide chapter on acknowledgment messages for details on this setting. Default: `1s`
    `ResyncPassThrough` Specifies source instances to pass through to this destination while replication is in the process of resynchronizing after a connection is made. The value of this element is a regular expression which is matched against the `Group` name of the instance that sent the replication message to this instance. When the regular expression matches, the replication message is eligible for passthrough, and will be sent to the destination if the `Topic` specifications match the message. When present, this value is used during resynchronization instead of the `PassThrough` value specified. If configured, this value should match every `Group` that the `PassThrough` value matches as well as any `Group` that replicates messages to this instance that are intended to be present on the downstream instance, but which are not included in the `PassThrough` configuration of this instance. This element is not required in most replication topologies. However, it can be useful to prevent gaps in replication during failover for topologies that intentionally rely on incomplete replication along certain paths. (Although a topology that does not fully replicate is not recommended, using this configuration item can help reduce the risk of missing messages in a topology that does not replicate messages along all paths that reach an instance that must receive the message). Default: There is no default for this value. If no value is configured, the `PassThrough` value, if any, is used for all replication to this destination.
    ### Transports in Destinations An outgoing `Destination` uses one or more `Transport` elements to specify how the instance will make an outgoing connection to the remote instance of AMPS. Described below are the configuration items available for a `Transport` in a `Replication/Destination`. Expand each item for more details.
    `Type` (required) Specifies the type of connection to make to the destination. The `Type` of a replication destination `Transport` must always be one of `amps-replication` or `amps-replication-secure`. When the `Type` is set to `amps-replication-secure`, the outgoing connection will use TLS/SSL. The `Type` of the outgoing connection must match the `Type` of the `Transport` that this instance is connecting to.
    `InetAddr` A `Transport` for a replication destination can contain one or more `InetAddr` elements. An `InetAddr` element can be specified as an IPv4 address, an IPv6 address, or a resolvable hostname. When a single `InetAddr` element is present, AMPS connects to that address for replication. When more than one `InetAddr` element is present, AMPS uses the list of addresses as a prioritized list of failover servers to provide high availability. The list is in priority order, with the most preferred server at the beginning of the list. Each time AMPS needs to make a connection for this `Destination`, AMPS starts with the first address in the list and tries each address in order until a connection succeeds. If no connection succeeds, AMPS waits for a timeout period and then either moves to the next `Transport` (if more than one `Transport` is present in the destination) or starts again with the first address in the list. Each time AMPS tries all of the addresses in the list without a successful connection, AMPS increases the timeout period between tries, up to a maximum timeout. The first time through the list, upon startup, AMPS gives addresses extra time, up to 60 seconds, to connect successfully. If no `InetAddr` is specified, then this `Destination` does not make an outgoing connection. Instead, this instance will wait for the remote instance specified in the `Destination` to connect, and replicate to that instance once the connection is established. Use this configuration with caution, as this configuration requires another instance of AMPS to connect. This configuration is only recommended in cases where this instance must replicate messages to another instance, but is unable to make a connection to that instance (for example, firewall rules block outgoing connections from the system that hosts the instance).
    `ReconnectTimeout` A list of intervals that specifies how long AMPS will attempt to connect to a given `InetAddr` before attempting to connect to the next entry. This configuration item can contain a single interval, which will be used for all `InetAddr` entries in the `Transport`, or a comma-delimited list of intervals. When a comma-delimited list is provided, the number of intervals provided must match the number of `InetAddr` entries in the `Transport`. Each interval is used for the corresponding `InetAddr`. (In other words, the first entry in the list is used for the first `InetAddr`, the second entry in the list is used for the second `InetAddr`, and so on.)
    `IpProtocolPrefer` A `Transport` element within a `Destination` may contain an `IpProtocolPrefer` element, which specifies the IP protocol to prefer during DNS resolution of hostnames when establishing a bidirectional replication connection. Currently allowed values for `IpProtocolPrefer` are: * `v4` to prefer resolving hostnames to IPv4 addresses * `v6` to prefer resolving hostnames to IPv6 addresses If this element is not specified, AMPS will prefer IPv4 name resolution by default. All of the `InetAddr` elements specified within a `Transport` use the same `IpProtocolPrefer` preference. If a valid DNS entry of the preferred IP protocol cannot be found, AMPS will fall back to the other *non-preferred* IP protocol type. If an `InetAddr` is specified as an explicit IP address, the protocol is determined from the format of the IP address and this setting has no effect.
    `Authenticator` A `Transport` element within a `Destination` may contain an `Authenticator` element, which specifies a module that provides credentials to use when connecting to the destination. All `InetAddr` elements specified within a `Transport` use the same `Authenticator`.
    `Authentication` A `Transport` element within a `Destination` may contain an `Authentication` element, which specifies the `Authentication` module to use when establishing a bidirectional replication connection. That is, if the other side of the connection will attempt to log in using an outgoing connection from this `Destination`, this sets the authentication to use for that connection. All of the `InetAddr` elements specified within a `Transport` use the same `Authentication` module. If this element is not specified, AMPS will use the `Authentication` module specified for the incoming `Transport` with a `Name` matching the `Type` of this `Transport`, or the `Authentication` for the instance if no such `Transport` is present.
    `Entitlement` A `Transport` element within a `Destination` may contain an `Entitlement` element, which specifies the `Entitlement` module to use when establishing a bidirectional replication connection. All of the `InetAddr` elements specified within a `Transport` use the same `Entitlement` module. If this element is not specified, AMPS will use the `Entitlement` module specified for the incoming `Transport` with a `Name` matching the `Type` of this `Transport`, or the `Entitlement` for the instance if no such `Transport` is present.
    **TLS/SSL Parameters** Described below are the configuration items needed to set up and enable TLS/SSL. Expand each item for more details.
    `Certificate` (required if `Type` is `amps-replication-secure`) A `Transport` element that specifies `amps-replication-secure` as the transport type must provide a certificate to use for the TLS/SSL connection. There is no default for this option.
    `PrivateKey` (required if `Type` is `amps-replication-secure`) A `Transport` element that specifies `amps-replication-secure` as the transport type must provide a private key to use for the SSL connection. There is no default for this option.
    `Ciphers` (optional, only supported if `Type` is `amps-replication-secure`) A `Transport` element that specifies `amps-replication-secure` as the transport type may provide a cipher list to use for the SSL connection. When provided, this connection is restricted to the specified ciphers. Default: No restriction on the ciphers supported by the SSL implementation.
    `VerifyClient` (optional, only supported if `Type` is `amps-replication-secure`) When set to `true`, this destination will verify certificates provided for TLS using the `CAFile` or `CAFileLocation` specified. Default: `false`
    `CAFile` (one of `CAFile` or `CAPath` must be specified if `VerifyClient` is `true`) When `VerifyClient` is set to `true`, specifies a `.pem` file containing trusted certificates used to verify certificates provided by the other side of the replication connection. There is no default for this option.
    `CAPath` (one of `CAFile` or `CAPath` must be specified if `VerifyClient` is `true`) When `VerifyClient` is set to `true`, specifies a path to a directory containing `.pem` files that contain trusted certificates used to verify certificates provided by the other side of the replication connection. When this parameter is provided and `VerifyClient` is set to `true`, AMPS will use every `.pem` file in the directory for verification. There is no default for this option.
    ### Topics in Destinations A replication destination can contain any number of `Topic` definition elements. For simplicity in working with the configuration file, 60East recommends using a few `Topic` elements with regular expression patterns over large numbers of individual topic declarations. When a `Destination` contains multiple `Topic` elements, messages that match _any_ of the `Topic` elements will be replicated. When matching a `Topic` element, literal (non-regular expression) topic names take priority over regular expression topic names. Described below are the configuration items available for a `Topic` in a `Replication/Destination`. Expand each item for more details.
    `Name` (required) The topic or topics to replicate. The `Name` can be either a literal topic name or a regular expression. When `Name` is a literal topic, a topic with that name and the specified message type must be captured in a transaction log. When `Name` is a regular expression, then topics that match the expression, match the message type, and are present in a transaction log are replicated. Defaults to the regular expression `.*`, which matches any topic name, if no value is explicitly set.
    `MessageType` (required) The message type of the topic or topics to replicate.
    `Filter` A content filter to apply to the topic or topics. When present, only messages that match the filter are replicated. This filter follows the standard AMPS filter syntax.
    `IncludeValidation` The set of configuration checks to validate for this topic. See [Replication Validation](/docs/amps-user-guide/replication/replication-configuration-validation) section for details. Default: All validation options listed are included by default.
    `ExcludeValidation` The set of configuration checks to exclude for this topic. If the same check appears in both `IncludeValidation` and `ExcludeValidation`, `ExcludeValidation` takes precedence and the check will not be run. See [Replication Validation](/docs/amps-user-guide/replication/replication-configuration-validation) section for details. Default: None of the validation options are excluded by default.
    ### Sample Replication Configuration This section shows a sample that configures an AMPS instance to provide messages to two downstream destinations. Notice that this sample does not configure the AMPS instance to _receive_ replication messages. Configuring an instance to receive replication is done in the `Transports` configuration for the instance. An example of this can be found here - [Configuring Incoming Replication Transports](/docs/amps-user-guide/replication/config-incoming-replication). ```xml showLineNumbers Data-Center-NYC-1 ORDER_STATE-Replication xml REFERENCE_INFO-.* json /state = 'published' sync enabled amps-replication interface1.example.com:19005 interface2.example.com:19080 my-credentials-store-module .* NYC-View-Server-Group ORDER_STATE xml replicate,cascade,sow async enabled amps-replication view-server-a.example.com:19005 view-server-b.example.com:19080 .* ``` --- # Configuring Replication Replication configuration involves the configuration of two or more instances of AMPS. For testing purposes both instances of AMPS can reside on the same physical host before deployment into a production environment. When running both instances on one machine, the performance characteristics will differ from production, making this setup more useful for testing configuration correctness than testing overall performance. Any instance that is intended to receive messages via replication must define an incoming replication transport as one of the `Transports` for the instance. An instance may have only one incoming replication transport. Any instance that is intended to replicate messages to another instance must specify a `Replication` stanza in the configuration file with at least one `Destination`. An instance can have multiple `Destination` declarations: each one defines a single outgoing replication connection. For details on setting up a replication transport, refer to [Configuring Incoming Replication Transports](/docs/amps-user-guide/replication/config-incoming-replication) and for configuring destinations, see [Configuring Outgoing Replication Destinations](/docs/amps-user-guide/replication/config-outgoing-replication). In AMPS replication, instances should only be configured as part of the same `Group` if they are fully equivalent. That is, not only should they contain the same messages, but they should be considered failover alternatives for applications and other AMPS servers. If two servers are not intended to be fully replicated (for example, if there is one-way replication between a production server and a test server), they should have different `Group` values. :::warning It's important to make sure that when running multiple AMPS instances on the same host there are no conflicting ports. AMPS will emit an error message and will not start properly if it detects that a port specified in the configuration file is already in use. ::: For the purposes of explaining this example, we're going to assume a simple hot-hot replication case where we have two instances of AMPS - the first host is named `amps-1` and the second host is named `amps-2`. Each of the instances are configured to replicate data to the other. That is, all messages published to `amps-1` are replicated to `amps-2` and vice versa. This configuration ensures that a message published to one instance is available on the other instance in the case of a failover (although, of course, the publishers and subscribers should also be configured for failover). :::info Every instance of AMPS that will participate in replication must have a unique `Name` among _all_ of the instances that are part of replication. All instances that have the same `Group` must be able to be treated as equivalent by AMPS replication and AMPS clients. If two instances of AMPS should be treated differently (for example, one instance receives publishes while the other is a read-only instance that receives one-way replication), those instances should be in different groups. ::: ## Replication Setup Example We will first show the relevant portion of the configuration used in `amps-1`, and then we will show the relevant configuration for `amps-2`. :::tip All topics to be replicated must be recorded in the transaction log. The examples below omit the transaction log configuration for brevity. Please reference the [Record and Replay Messages](../txlog) chapter for information on how to configure a transaction log and choose which topics are recorded in the transaction log. ::: ```xml showLineNumbers amps-1 DataCenter-NYC-1 ... amps-replication amps-replication 10004 ... transports for client use here ... ... fix topic json ^/orders/ amps-2 .* DataCenter-NYC-1 sync amps-2-server.example.com:10005 amps-replication ... ``` For the configuration of `amps-2`, we will use the following example. While this example is similar, only the differences between the `amps-1` configuration will be called out. ```xml showLineNumbers amps-2 DataCenter-NYC-1 ... amps-replication amps-replication 10005 ... fix topic json ^/orders/ amps-1 .* DataCenter-NYC-1 sync amps-1-server.example.com:10004 amps-replication ... ``` These example configurations replicate the topic named `topic` of the message type `nvfix` and any topic of the message type `json` that begins with `/orders/` between the two instances of AMPS. To replicate more topics, these instances could add additional `Topic` blocks. ## Downstream Persistence Acknowledgment: Sync vs Async When publishing to a topic that is recorded in the transaction log, it is recommended that publishers request a `persisted` acknowledgment. The `persisted` acknowledgment message is how AMPS notifies the publisher that a message received by AMPS is considered to be safely persisted, as specified in the configuration. (The AMPS client libraries automatically request this acknowledgment on each `publish` command when a publish store is present for the client -- that is, any time that the client is configured to ensure that the publish is received by the AMPS server.) Depending on the replication destination configuration for the AMPS instance that receives the message, that `persisted` acknowledgment message will be delivered to the publisher at different times in the replication process. There are two options: `sync` (synchronous) or `async` (asynchronous) acknowledgment. These two acknowledgment `SyncType` options control when the instance of AMPS will acknowledge the message as persisted. In other words, this controls when the message publisher will receive a `persisted` acknowledgment. AMPS will not return a `persisted` acknowledgment to the publisher for a message until: * The message has been stored to the local transaction log (and SOW as applicable), _**and**_ * All downstream replication destinations using `sync` acknowledgment have acknowledged the message. The acknowledgment type (`SyncType`) has no effect on how an instance of AMPS replicates the message to other instances of AMPS. The process of sending messages is identical for the instance sending messages. The instance that receives the messages has no information on the acknowledgment type the upstream link has configured, so all incoming messages are processed in the same way. The acknowledgment type _only_ affects whether the instance pushing the message must receive an acknowledgment from that `Destination` before it will acknowledge a message as having been persisted. It's typical for an instance of AMPS to have multiple destinations with different acknowledgment types. When this is the case, the instance can acknowledge a message when all destinations using `sync` acknowledgment have acknowledged the message. No destinations using `async` acknowledgment are considered. The figure below shows the cycle of a message being published in a replicated instance, and the persisted acknowledgment message being returned back to the publisher. Notice that, with this configuration, the publisher will not receive an acknowledgment if the remote destination is unavailable. 60East recommends that when you use `sync` replication, you consider setting a policy for downgrading the link when a destination is offline, as described in [Downgrading Acknowledgments for a Destination](configuring-replication.md#downgrading-acknowledgments-for-a-destination). The sequence for a destination that uses `async` acknowledgment is different. For a destination that uses `async` acknowledgment, the AMPS instance replicating the message can send a `persisted` acknowledgment message back to the publisher as soon as the message is stored in the local transaction log and SOW stores. The instance does not wait for acknowledgment from the destination, which means that acknowledgment can happen before the replicated instance has stored the message. The figure below shows the cycle of a message being published with a `SyncType` configuration set to `async` acknowledgment. By default, replication destinations do not affect when a message is delivered to a subscription. Optionally, a subscriber can request the `fully_durable` option on a bookmark subscription (that is, a replay from the transaction log). When the `fully_durable` option is specified, AMPS does not deliver a message to that subscriber until all replication destinations using `sync` acknowledgment have acknowledged the message. :::info Every instance of AMPS that accepts publish commands, SOW delete commands or allows consumption of messages from queues, should specify at least one destination that uses `sync` acknowledgment. If a publish or queue consumer may fail over between two (or more) instances of AMPS, those instances should specify `sync` acknowledgment between them to prevent a situation where a message could be lost if an instance fails immediately after acknowledging a message to a publisher. ::: A destination configured for `sync` acknowledgment can be downgraded to `async` acknowledgment while AMPS is running. This can be useful in cases where a server is offline for an extended period of time due to hardware failure or persistent network issues. While the destination is downgraded, AMPS considers that destination to be using `async` acknowledgment, as described in the next section. ## Downgrading Acknowledgments for a Destination AMPS provides the ability to temporarily downgrade a replication link from _synchronous_ to _asynchronous_ acknowledgment. This feature is useful to relieve memory or storage pressure on publishers should a downstream AMPS instance prove unstable, unresponsive, or be experiencing excessive latency to the point that it should be considered to be offline. A link can be downgraded using an action or explicitly downgraded from the AMPS administrative console. Likewise, a link that has previously been downgraded can be upgraded using an action or from the AMPS administrative console. When a replication link is downgraded, that link will use `async` acknowledgment until the link upgrades or until AMPS restarts. Downgrading a replication link to using `async` (asynchronous) acknowledgment means that any `persisted` acknowledgment message that a publisher may be waiting on will no longer wait for the downstream instance to confirm that it has committed the message to its downstream Transaction Log or SOW store. AMPS immediately considers the downstream instance to have acknowledged the message for existing messages, which means that if AMPS was waiting for acknowledgment from that instance to deliver a `persisted` acknowledgment, AMPS immediately sends the `persisted` acknowledgment when the instance is downgraded. Downgrading the acknowledgment type reduces the reliability guarantees provided by that replication link. Because those guarantees are reduced, the publisher can remove messages that it would have to retain if the guarantees were enforced. The result of a link being downgraded is: * The number of messages that the publisher must retain is reduced, _but_ * The downgraded link is unsafe for the publisher to fail over to * The downgraded link is unsafe for a bookmark subscriber to fail over to Automatic downgrade is most suitable for a situation where an instance should be considered offline or unavailable. If an instance is configured to use an action to downgrade the acknowledgment type, it should also be configured to use an action to upgrade acknowledgment. :::danger Downgrading a destination means that this instance will not wait for that destination to acknowledge a message before acknowledging that message to publishers or upstream instances. It does not affect any other behavior of the instance. A publisher or queue consumer must not fail over from this instance to a destination that has been downgraded to `async` acknowledgment. This can cause message loss, since the upstream instance may have acknowledged a message that the downstream instance has not yet processed. A bookmark subscriber must not fail over from this instance to a destination that has been downgraded to `async` acknowledgment. This can cause replay gaps, since that destination is no longer considered when determining whether a message is persisted. ::: #### Automatically Downgrading and Upgrading Acknowledgment AMPS can be configured to automatically downgrade a replication link to `async` if the remote side of the link cannot keep up with persisting messages or becomes unresponsive. This option prevents unreliable links from holding up publishers but increases the chances of a single instance failure resulting in message loss, as described above. AMPS can also be configured to automatically upgrade a replication link that has previously been downgraded. Since downgrading a link to a destination affects the consistency and durability guarantees provided by the set of AMPS instances as a whole, use caution when configuring the parameters. In general, it's a good idea to set an interval that is larger than the amount of time at which an instance would be considered to be unresponsive or offline. Automatic downgrade is implemented as an AMPS action. To configure automatic downgrade, add the appropriate action to the configuration file as shown below: ```xml showLineNumbers ... amps-action-on-schedule 15s amps-action-do-downgrade-replication 300s amps-action-do-upgrade-replication 10s ... ``` In this configuration file, AMPS checks every 15 seconds to see if a destination has fallen behind by 300 seconds. If a destination has fallen behind by more than 300 seconds, that destination should no longer be considered online. Typically, this would be set to a duration longer than the time at which monitoring of that instance would produce alerts that the instance is unavailable. AMPS downgrades the destination to `async` acknowledgment. That destination will no longer be considered when acknowledging messages to publishers. Once the link to the destination is downgraded, connections to this instance should not consider that destination to be safe for failover until the link has again been upgraded. :::info All publishers using a publish store should be able to hold a number of messages equal to the number of messages published, at peak message volume, for a time period equal to the periodicity of the downgrade check plus the threshold for downgrade. With the configuration above, a publisher that publishes at a peak rate of 10,000 messages per second should, at a minimum, be able to allocate a publish store that holds 750,000 messages. ::: In some cases, it is important that a destination maintain a minimum number of destinations that use `sync` acknowledgment. For those cases, an instance-level `Tuning` parameter is available that will prevent the action from downgrading a connection if doing so would reduce the number of destinations that use `sync` acknowledgment below the configured limit. This parameter does not guarantee whether a specific destination will continue using `sync` acknowledgment. This parameter only limits whether AMPS will downgrade a destination that meets downgrade criteria. AMPS will not upgrade a destination that has previously been downgraded if a connection is lost, even if this means that the number of currently connected destinations that use `sync` acknowledgment is less than the configured minimum. See the section on [Instance-Level Configuration](/docs/amps-user-guide/configuring-amps/instance-configuration) for details. --- # Destination Server Failover Your replication plan may include replication to a server that is part of a highly-available group. There are two common approaches to destination server failover: 1. **Wide IP** - AMPS replication works transparently with wide IP and many installations use wide IP for destination server failover. The advantage of this approach is that it requires no additional configuration in AMPS and redundant servers can be added or removed from the wide IP group without reconfiguring the instances that replicate to the group. A disadvantage to this approach is that failover can require several seconds and messages are not replicated during the time that it takes for failover to occur. 2. **AMPS Failover** - AMPS allows you to specify multiple downstream servers in the `InetAddr` element of a destination. In this case, AMPS treats the defined list of servers as a list of equivalent servers, listed in order of priority. When multiple addresses are specified for a destination, each time AMPS needs to make a connection to a destination and there is no incoming connection from a server in that destination, AMPS starts at the beginning of the list and attempts to connect to each address in the list. If AMPS is unable to connect to any address in the list, AMPS waits for a timeout period, then begins again with the first server on the list. Each time AMPS reaches the end of the list without establishing a connection, AMPS increases the timeout period. If an incoming connection from one of the servers on the list exists, AMPS will use that connection for outgoing replication. If multiple incoming connections from servers in the list exist, AMPS will choose one of the incoming connections to use for outgoing traffic. This capability allows you to easily set up replication to a highly-available group. If the server you are replicating to fails over, AMPS uses the prioritized list of servers to re-establish a connection. --- # Guarantees on Ordering For each publisher, on a single topic, AMPS is guaranteed to deliver messages to subscribers in the same order that the messages were published by the original publisher. This guarantee holds true regardless of how many publishers or how many subscribers are connected to AMPS at any one time. For each instance, AMPS is guaranteed to deliver messages in the order in which the messages were received by the instance, regardless of whether a message is received directly from a publisher or indirectly via replication. The message order for the instance is recorded in the transaction log, and is guaranteed to remain consistent across server restarts. These guarantees mean that subscribers will not spend unnecessary CPU cycles checking timestamps or other message content to verify which message is the most recent, or reordering messages during playback. This frees up subscriber resources to do more important work. AMPS preserves an absolute order across topics for a single subscription for all topics _except_ views, queues, and conflated topics. Applications often rely on this behavior to correlate the times at which messages to different topics were processed by AMPS. See [Message Ordering](../pub-sub/ordering) for more information. --- # PassThrough Replication PassThrough Replication is a term used to describe the ability of an AMPS instance to pass along replicated messages to another AMPS instance. This allows you to easily keep multiple failover or DR destinations in sync from a single AMPS instance. Unless passthrough replication is configured, an AMPS instance only replicates messages directly published to that instance from an application. By default, an instance _does not_ re-replicate messages received over replication. PassThrough replication uses the name of the originating AMPS group to indicate that messages that arrive at this instance of AMPS directly from that group are to be replicated to the specified destination. PassThrough replication supports regular expressions to specify groups, and allows multiple server groups per destination. Notice that if the destination instance does not specify a `Group` in its instance config, the group name is the `Name` of the instance. To ensure that an instance replicates a full copy of its transaction log downstream (which is typically the intended result), include a `PassThrough` configuration item that matches any group name. With care, some topologies can use a `PassThrough` configuration that only replicates messages directly published to the instance and a subset of messages received over replication. This can result in significant bandwidth reduction in some topologies, but must be configured with care to ensure that messages do not fail to reach all of the instances, since in this case AMPS is configured to replicate only a _part_ of the transaction log of the local instance. ```xml showLineNumbers AMPS2-HKG amps-replication amps-replication secondaryhost:10010 /rep_topic fix /rep_topic2 fix sync ^((?!HKG).)*$ ``` When a message is eligible for passthrough replication, topic and content filters in the replication destination still apply. The passthrough directive simply means that the message is eligible for replication from this instance if it comes from an instance in the specified group. AMPS protects against loops in passthrough replication by tracking the instance names or group names that a message has passed through. AMPS does not allow a message to travel through the same instance and group name more than once. :::info When using passthrough, AMPS uses the path that the message has taken to reach this instance to protect against replication loops. ::: If an instance replicates a queue (distributed queue) or a group local queue, it _must also_ provide passthrough for any incoming replication group that replicates that topic (even if the incoming replication connection is from the same group that this instance belongs to). The reason for this is simple: AMPS must ensure that messages for a replicated queue, including acknowledgments and transfer messages, are able to reach every instance that hosts the queue if possible, even if a network connection fails or an instance goes offline. Therefore, this instance must pass through messages received from other instances that affect the queue. --- # Replicated Queues AMPS provides a unique approach to replicating queues. This approach is designed to offer high performance in the most common cases, while continuing to provide delivery model guarantees, resilience and failover in the event that one of the replicated instances goes offline. When a queue is replicated, AMPS replicates the `publish` commands to the underlying topic, the `sow_delete` commands that contain the acknowledgment messages, and special queue management commands that are internal to AMPS. ### Queue Message Ownership To guarantee that no message is processed more than once, AMPS tracks ownership of the message within the network of replicated instances. For a distributed queue (that is, a queue defined with the `Queue` configuration element), the instance that first receives the publish command from a client owns the message. Although all replicated instances downstream record the publish command in their transaction logs, they do not provide the message to queue subscribers unless that instance owns the message. For a group local queue (that is, a queue defined with the `GroupLocalQueue` tag), the instance specified in the `InitialOwner` element for the queue owns a message when the message first enters the queue, regardless of where the message was originally published. Only one instance can own a message at any given time. Other instances can request that the current owner transfer ownership of a message. When a message is published, the instance that owns the message depends on the type of queue: | Queue Type | Initial Owner | | ------------- | -------- | | `Queue` | Instance where the message was published. | | `GroupLocalQueue` | Instance specified in the `InitialOwner` tag. | |

    `LocalQueue`

    cannot be replicated

    |

    N/A

    Each instance owns its copy of the message. The queue is not replicated. Each instance will independently deliver its copy of the message.

    | To transfer ownership, an instance that does not currently own the message makes a request to the current message owner. The owning instance makes an explicit decision to transfer ownership, and replicates the transfer notification to all instances to which the queue topic is replicated. The instance that owns a message will always deliver the message to a local subscriber if possible. This means that performance for local subscribers is unaffected by the number of downstream instances. However, this also means that if the local subscribers are keeping up with the message volume being published to the queue, the owning instance will never need to grant a transfer of ownership. A downstream instance will request ownership transfer if: 1. The downstream instance has subscriptions for that topic with available backlog, _and_ 2. The amount of time since the message arrived at the instance is greater than the typical time between the replicated message arriving and the replicated acknowledgment arriving. Notice that this approach is intended to minimize ungranted transfer requests. In normal circumstances, the typical processing time reflects the speed at which the local processors are consuming messages at a steady state. Downstream instances will only request messages that have been seen to exceed that time, indicating that the processors are not keeping up with the incoming message rate. The instance that owns the message will grant ownership to a requesting instance if: 1. The request is the first request received for this message, _and_ 2. There are no subscribers on the owning instance that can accept the message When the owning instance grants the request, it logs the transfer in its transaction log and sends the transfer of ownership to all instances that are receiving replicated messages for the queue. When the owning instance does not grant the transfer of ownership, it takes no action. Notice that your replication topology must be able to replicate acknowledgments to all instances that receive messages for the queue. Otherwise, an instance that does not receive the acknowledgments will not consider the messages to be processed. Replication validation can help to identify topologies that do not meet this requirement. :::tip A _barrier message_ is delivered immediately when there are no unacknowledged messages ahead of the barrier message in the queue on this instance, _regardless of which instance owns the message_. This means that for a distributed queue or group local queue, every queue that contains the barrier message will deliver the barrier message when all previous messages on that instance have been acknowledged. ::: #### Disaster Recovery and Queue Message Ownership When an instance that contains a queue fails or is shut down, that instance is no longer able to grant ownership requests for the messages that it owns. This means that those messages cannot be delivered to subscribers since the owner cannot transfer ownership. AMPS provides a way to change the handling of transfer requests so those messages can be delivered. Through the admin console, you can choose to `enable_proxied_transfer`, which allows an instance to act as an ownership proxy for an instance that has gone offline. That is, an instance with proxied transfer enabled can choose to process transfer requests itself rather than sending the requests to an instance that is known to be offline. An instance may claim ownership of a message directly rather than sending a transfer request when: * Proxied transfer is enabled, _and_ * An active subscription could receive the message, _and_ * The current owner is not known to be reachable through replication An instance may grant a transfer request for a message it does not own when: * Proxied transfer is enabled, _and_ * A transfer request is received for a message currently in the queue, _and_ * The current owner is not known to be reachable through replication When an instance assumes ownership of a message that instance will write an ownership transfer to the transaction log (which will then be replicated). This means that other instances that are still available will know that this instance has taken ownership. :::danger _Use this setting with care_ When proxied transfer is enabled it is possible for messages to be delivered twice or for queues to reach an inconsistent state if an instance that currently owns messages in the queue is online and has subscribers to the queue, or if multiple instances enable proxy transfer for the same queue. ::: ##### Using Proxied Transfer for Disaster Recovery Proxied transfer is used as a temporary recovery step while an instance that owns messages for a queue is offline and the level of service for the queue requires those messages to be delivered before that instance is recovered. In this case, the recommended procedure is to: 1. Choose *one* of the remaining instances that hosts the queue 2. Fail over clients with subscriptions to that queue to that instance 3. Enable proxied transfer for that queue on that instance 4. Recover the offline instance without allowing client connections to that instance. Notice that, once replication reconnects, that instance will again be available to process transfer requests for messages that it owns. 5. Once the offline instance is recovered, disable proxied transfer on the queue on the instance where it was enabled. 6. Enable client connections to the recovered instance. Notice that, as described earlier, enabling proxied transfer *only* affects how an instance will handle a transfer request for an offline instance. When this is enabled on a queue, it does not cause the queue to immediately take ownership of messages, nor does it immediately send transfer requests. The setting only affects messages that AMPS is trying to deliver to a subscription, which limits the risk of duplicate delivery to only those messages that will be consumed immediately. ### Configuration for Queue Replication To provide replication for a distributed queue, AMPS requires that the replication configuration meet the following requirements: 1. Provide bidirectional replication between the instances. In other words, if instance A replicates a queue to instance B, instance B must also replicate that queue to instance A. 2. If the topic is a queue on one instance, it must be a queue on all replicated instances. 3. On all replicated instances, the queue must use the same underlying topic definition and filters. For queues that use a regular expression as the topic definition, this means that the regular expression must be the same. For a `GroupLocalQueue`, the `InitialOwner` must be the same on all instances that contain the queue. 4. The underlying topics must be replicated to all replicated instances (since this is where the messages for the queue are stored). 5. Replicated instances must provide passthrough for instances that replicate queues. For example, consider the following replication topology: Instance A in group One replicates a queue to instance B in group Two. Instance B in group Two replicates the queue to instance C in group Three. For this configuration, instance B must provide passthrough for group Three to instance A, and must also provide passthrough for group One to instance C. The reason for this is to ensure that ownership transfer and acknowledgment messages can reach all instances that maintain a copy of the queue. Likewise, consider a topology where Instance X in GroupOne replicates a queue to Instance Y in GroupOne. Instance X must provide passthrough for GroupOne, since any incoming replication messages for the queue (for example, from Instance Z) that arrive at Instance X must be _guaranteed_ to reach Instance Y. Otherwise, it would be possible for the queue on Instance Y to have different messages than Instance X and Instance Z if Instance Z does not replicate to Instance Y (or if the network connection between Instance Z and Instance Y fails). Notice that _the requirements above apply only to queue topics_. If the underlying topic uses a different name than the queue topic, it is possible to replicate the messages from the underlying topic _without_ replicating the queue itself. This approach can be convenient for simply recording and storing the messages provided to the queue on an archival or auditing instance. When only the underlying topic (or topics) are replicated, the requirements above do not apply, since AMPS does not provide queuing behavior for the underlying topics. A queue defined with `LocalQueue` cannot be replicated. The data from the underlying topics for the queue can be replicated without special restrictions. The queue topic itself, however, cannot be replicated. AMPS reports an error if any `LocalQueue` topic is replicated. --- # Replication Basics Before planning an AMPS replication topology, it can be helpful to understand the basics of how AMPS replication works. This section presents a general overview of the concepts that are discussed in more detail in the following sections. * **Replication is point-to-point**. Each replication connection involves exactly two AMPS instances: a source (that provides messages) and a destination that receives messages. * **Replication is always "push" replication**. In AMPS, the source configures a destination, and pushes messages to that destination. (Notice that it is possible to configure the source to wait for the destination to connect rather than actively making an outgoing connection, but replication is still a "push" from the source to the destination once that connection is made). The source must be configured to push messages to the destination, and the source guarantees that all messages to be replicated must be acknowledged by the destination before they can be removed from the transaction log. * **Replication is one-link by default**. By default, an instance of AMPS _only_ replicates messages that are published to directly to that instance by a client. Optionally, an instance of AMPS can be configured to also replicate messages that arrive over replication. Adding this configuration is typically required if there are more than two instances in a replicated set of AMPS instances. See the [PassThrough Replication](passthrough-replication) topic for details. * **Replication relies on the transaction log**. AMPS replicates the commands as preserved in the transaction log. This means, for example, that the results of delta publishes are replicated as fully-merged messages, since fully-merged messages are stored in the transaction log. Likewise, if duplicate messages arrive over different paths, only the first message to arrive will be stored in the transaction log, and that message is the one that will be replicated. Replication always provides messages to a destination in the order in which the messages are recorded in the transaction log of the instance sending the message. Messages that are not stored in the transaction log cannot be replicated. * **Replication provides a command stream**. In AMPS replication, the server replicates the results of `publish`, `delta_publish` and `sow_delete` commands once those results are written to the transaction log. Each individual command is replicated, for low latency and fine-grained control of what is replicated. If a command is not in the transaction log (for example, a maintenance action has removed the journal that contains that command, or the command is for a topic that is not recorded in the transaction log), that command will not be replicated. Replication is intended to guarantee that the command stream for a set of topics on one instance is present on the other instance, with the ordering of each message source preserved. This means that there can be only _one_ connection from a given upstream instance to a given downstream instance. * **Replication is customizable by topic, message type, and content**. AMPS can be configured to replicate the entire transaction log, or any subset of the transaction log. This makes it easy to use replication to populate view servers, test environments, or similar instances that require only partial views of the source data. * **Replication guarantees delivery**. AMPS will not remove a journal file until _all_ messages in that journal file have been replicated to, and acknowledged by, the destination. * **Replication is composable**. AMPS is capable of building a sophisticated replication topology by composing connections. For example, full replication between two servers is two point-to-point connections, one in each direction. The basic point-to-point nature of connections makes it easy to reason about a single connection, and the composable nature of AMPS replication allows you to build replication networks that provide data distribution and high availability for applications across data centers and around the globe. * **Replication acknowledgment is configurable**. The acknowledgment mode provides different guarantees: _async_ acknowledgment provides durability guarantees for the local instance, whereas _sync_ acknowledgment provides durability guarantees for the local instance and the downstream instance. * **Group identifies a set of instances that are intended to be fully equivalent.** This identification is for the purposes of message contents, application failover, and AMPS replication failover. Instances that are not intended to be fully equivalent for all of these purposes should be given a different `Group` name, even if they are in the same data center or geographic location, or if they would be treated as equivalent for some, but not all, purposes. More details on each of these points is provided in this section. ## Benefits of Replication Replication can serve two purposes in AMPS: 1. It can increase the fault-tolerance of AMPS by creating another instance to be used should an instance fail or be taken offline. 2. Replication can be used in message delivery to a remote site. In order to provide fault tolerance and reliable remote site message delivery, for the best possible messaging experience, there are some guarantees and features that AMPS has implemented. Those features are discussed in the following sections. Replication in AMPS supports filtering by both topic and by message content. This granularity in filtering allows replication sources to have complete control over what messages are sent to their downstream replication instances. Additionally, replication can improve availability of AMPS by creating a redundant instance of an AMPS server. Using replication, all of the messages which flow into a primary instance of AMPS can be replicated to a secondary spare instance. This way, if the primary instance should become unresponsive for any reason, then the secondary AMPS instance can be swapped in to begin processing message streams and requests. :::tip When an AMPS instance is a replication source, that instance _guarantees_ that messages will not be removed from the transaction log until all destinations have acknowledged the message. ::: --- # Replication Best Practices For your application to work properly in an environment that uses AMPS replication, it is important to follow these best practices: * _Every client that changes the state of AMPS must have a distinct client name_ - Although AMPS only enforces this requirement for an individual instance, if two clients with the same name are connected to two different instances, and both clients publish messages, delete messages, or acknowledge messages from a queue, the messages present on each instance of AMPS can become inconsistent. * _Use replication filters with caution, especially for queue topics_ - Using a replication filter will create different topic contents on each instance. In addition, using a replication filter for a message queue topic (or the underlying topic for a message queue) can create different queue contents on different instances, and messages that are not replicated must be consumed from the instance where they were originally published. * _Do not manually set client sequence numbers on published messages_ - The publish store classes in the AMPS client libraries manage sequence numbers for published messages to ensure that there is no message loss or duplication in a high availability environment. 60East recommends using those publish stores to manage sequence numbers rather than setting them manually. Since AMPS uses the sequence number to identify duplicate messages, setting sequence numbers manually can cause AMPS to discard messages or lead to inconsistent state across instances. * _Default to PassThrough for every group_ - PassThrough for every group guarantees that each upstream instance will provide the full set of messages that it has to downstream groups. For some topologies, it is possible to reduce traffic to downstream instances by using the PassThrough configuration to avoid replicating messages that are guaranteed to arrive via another route (even in cases of network or server failure), but this should be done with caution. * _Do not allow a publisher or queue consumer to fail over between two instances that are replicating using async acknowledgment_ - The `async` acknowledgment mode means that messages are acknowledged to a publisher before the downstream replication instance has acknowledged that it has received the message. Allowing a publisher to fail over between two instances that are using `async` acknowledgment runs the risk of creating inconsistent state or message loss. --- # Replication Compression AMPS provides the ability to compress the replication connection. In typical use, using replication compression can greatly reduce the bandwidth required between AMPS instances. The precise amount of compression that AMPS can achieve depends on the content of the replicated messages. Compression is configured at the replication source, and does not need to be enabled in the transport configuration at the instance receiving the replicated messages. For AMPS instances that are receiving replicated messages, no additional configuration is necessary. AMPS automatically recognizes when an incoming replication connection uses compression. See the [Configuring Outgoing Replication Destinations](/docs/amps-user-guide/replication/config-outgoing-replication) section for enabling compression and choosing a compression algorithm. --- # Replication Configuration Validation Replication configuration validation helps to ensure that any configuration that could result in message loss or inconsistent message contents between two instances of AMPS is explicitly designed into the replication topology and not the result of accidental misconfiguration. Replication can involve coordinating configuration among a large number of AMPS instances. It can sometimes be difficult to ensure that all of the instances are configured correctly, and to ensure that a configuration change for one instance is also made at the replication destinations. For example, if a high-availability pair replicates the topics ORDERS, INVENTORY, and CUSTOMERS to a downstream disaster recovery site, but the disaster recovery site only replicates ORDERS and INVENTORY back to the high-availability pair, disaster recovery may not occur as planned. Likewise, if only one member of the HA pair replicates ORDERS to the other member of the pair, the two instances will contain different messages, which could cause problems for the system. Starting in the 5.0 release, AMPS automatic replication configuration validation makes it easier to keep configuration items consistent across a replication fabric. Replication configuration validation happens when a replication connection is made between two instances. The validation compares the configuration of those two instances. By default, any difference in configuration that could result in message loss, different behavior between the source instance and the destination instance, or different replication guarantees between the source instance and the destination instance is reported as an error. :::warning When replication validation reports an error, the reason for the error is logged to the event log on the instance that detects the problem, and the connection is closed. To troubleshoot the issue, it is typically necessary to check the logs and configuration on both instances. ::: Automatic configuration validation is enabled for all elements of the replication configuration by default. You can turn off validation for specific elements of the configuration, as described below. AMPS replication uses a leaderless, "all nodes hot" model. This means that no single AMPS instance has a view of the entire replication fabric, and a single AMPS instance will always assume that there are instances in the replication fabric that it is not aware of. The replication validation rules are designed with this assumption. The advantage of this assumption is that if instances are added to the replication fabric, it is typically only necessary to change configuration on the instances that they directly communicate with for replication to function as expected. The tradeoff, however, is that it is sometimes necessary to configure an instance as though it were part of a larger fabric (or exclude a validation rule) even in a case where the instance is part of a much simpler replication design. Each `Topic` in a replication `Destination` can configure a unique set of validation checks. By default, all of the checks apply to all topics in the `Destination`. When troubleshooting a configuration validation error, it is important to look at the AMPS logs on _both_ sides of the connection. Typically, the AMPS instance that detects the error will log complete information on the part of validation that failed and the changes required for the connection to succeed, while the other side of the connection will simply note that the connection failed validation. This means that if a validation error is reported on one instance, but details are not present, the other side of the connection detected the error and will have logged relevant details. :::danger Excluding a validation check directs AMPS to make a replication connection that could result in inconsistent state or data loss. Use caution when excluding validation checks. See the table below for details on each validation check. ::: By default, replication validation treats the downstream instance as though it is intended to be a full highly-available failover partner for any topic that is replicated. For situations where that is not the case, many validation rules can be excluded. For example, if the downstream instance is a view server that does not accept publishes and, therefore, is not configured to replicate a particular topic back to this instance, the `replicate` validation check might need to be excluded. Removing validation checks should be done with caution. Removing a validation check states that this configuration is intended to create instances that may differ in contents. AMPS performs the following validation checks. Expand each item for more details. In this discussion "this instance" refers to the instance sending messages via replication and the "downstream instance" refers to the instance receiving messages via replication.
    `txlog` Validates that the topic is contained in the transaction log of the downstream instance. An error on this validation check indicates that this instance is replicating a topic that is not in the transaction log on the downstream instance. This means that the downstream instance is not persisting the messages in a way that can be used for replication, replay, or used as the basis for a queue.
    `replicate` Validates that the topic is replicated from the downstream instance back to this instance. An error on this validation check indicates that this instance is replicating a topic to the downstream instance that is not being replicated back to this instance. This means that any publishes or updates to the topic on the downstream instance are not replicated back to this instance. If this is intentional (for example, replicating messages to a read-only view server), the upstream instance can exclude this validation check.
    `sow` Validates that if the topic is a `SOW/Topic` in this instance, it must also be a `SOW/Topic` in the downstream instance. An error on this validation check indicates that this instance is replicating a topic to the downstream instance that is a `SOW/Topic` on this instance but is not a `SOW/Topic` on the downstream instance. This means that the topic has different behavior on the downstream instance, and does not maintain the current value of records in the topic in the SOW.
    `cascade` Validates that the downstream instance must enforce the same set of validation checks for this `Topic` as this instance does. When relaxing validation rules for a topic that the downstream instance itself replicates, it is usually necessary to add a `cascade` exclusion for this instance as well. An error on this validation check indicates that this instance enforces a validation check for a topic that the downstream instance does not enforce when that instance replicates the topic. To understand the impact of this validation check, consider the validation checks that the downstream instance enforces. If the downstream instance enforces the appropriate validation checks, this instance can exclude the `cascade` check. For example, if the downstream instance replicates to a view server that does not replicate back, the downstream instance may exclude the replicate check, and this instance would need to exclude the cascade check to indicate that replicating to an instance that excludes checks is intentional. It is sometimes necessary to exclude this check as part of a rolling upgrade, and then to leave this exclusion in place until all instances can be taken offline at the same time. If the `cascade` check is the only check being excluded on any instance, the topology can be considered to meet validation rules (and the `cascade` exclusion can be safely removed during a maintenance window when all of the instances can be updated simultaneously). :::warning Replication topologies that intentionally have asymmetrical replication typically require this exclusion (for example, replication to a read-only view server as mentioned earlier). This exclusion can also become necessary as part of a rolling update where topics are being added or changed, even if the final state of the replication fabric would not require this exclusion. In this case, it is typically necessary to leave the exclusion in place until all instances can be taken offline at the same time (so the `cascade` exclusion can be removed from all of the configurations at once). If the `cascade` validation check is the *only* check that is excluded throughout a replication fabric, the topology can be considered to fully meet validation rules. :::
    `queue` Validates that if the topic is a queue in this instance, it must also be a queue in the downstream instance. This is a _mandatory validation check_, and _cannot be excluded_. A distributed queue (defined with the `SOW/Queue` or `SOW/GroupLocalQueue` tags) will not function correctly if one of the instances it is replicated to does not define the topic as a queue. An error in this validation check means that the queue will not function correctly, and the appropriate queue definition must be added to the downstream instance.
    `keys` Validates that if the topic is a `SOW/Topic` in this instance, it must also be a `SOW/Topic` in the downstream instance and the `SOW/Topic` in the downstream instance must use the same `Key` definitions. An error on this validation check indicates that this instance is replicating a topic to the downstream instance that is a `SOW/Topic` on both instances, but that the definition of message identity (the `Key` configuration for the topic) does not match on the two instances. This means that the contents of this topic may be different on these two instances for the same set of messages published.
    `replicate_filter` Validates that if this topic uses a replication filter, the downstream instance must use the same replication filter for replication back to this instance. An error on this validation check indicates that this instance uses a replication filter for a topic that the downstream instance does not use when it replicates the topic. The result is that, for a given set of messages, the downstream instance may replicate a different set of messages than it received. This would produce inconsistent data across the set of replicated instances. In some cases (for example, partitioning a global stream of messages into particular regions), this is the intended result.
    `queue_passthrough` Validates that if the topic is a queue in this instance, the downstream instance must support passthrough from this group to its replication destinations. An error on this validation check indicates that this instance does not pass through messages for one or more groups that the queue is replicated from. This could lead to a situation where a queue message is undeliverable if a network connection is unavailable or if additional instances are added to the set of instances that contain the queue.
    `queue_underlying` Validates that if the topic is a queue in this instance, it must use the same underlying topic definition and filters in the downstream instance. This is a _mandatory validation check_, and _cannot be excluded_. A distributed queue (defined with the `SOW/Queue` or `SOW/GroupLocalQueue` tags) will not function correctly if one of the instances it is replicated to does not contain the same messages as the other instances that host the queue. An error in this validation check means that the queue definitions will not contain the same messages, so the queue will not function correctly. The underlying topics in the queue definition must be identical on this instance and the downstream instance.
    ### Example The sample below shows how to exclude validation checks for a replication destination. In this sample, the `Topic` does not require the downstream destination to replicate back to this instance, and does not require that the downstream destination enforce the same configuration checks for any downstream replication of this topic. ```xml showLineNumbers ... json MyStuff-VIEW replicate,cascade ... ``` --- # Replication Resynchronization When a replication connection is established between AMPS instances, the upstream instance publishes any messages that it contains in its transaction log that the downstream instance may not have previously received. This process is called "replication resync". During resync, the upstream instance replays from the transaction log, replicating messages that match the `Topic` (and, optionally, `Filter`) specification(s) for the downstream `Destination`. When a replication connection is established between AMPS servers that are both version 5.3.3.0 or higher, the servers exchange information about the messages present in the transaction log to determine the earliest message in the transaction log on the upstream instance that is not present on the downstream instance. Replication resynchronization will begin at that point. This approach to finding the resynchronization point applies whether or not these two instances have had a replication connection before. Messages automatically recorded by each instance in the `/AMPS/Tx/Checkpoint` topic are used to help with this determination, since an instance can assume that the message stream is up to date with the last checkpoint received that originated at each instance. These messages are replicated as though the `/AMPS/Tx/Checkpoint` topic were explicitly configured for replication to each downstream destination. For replication between older versions of AMPS, replication resynchronization begins at the last point in the upstream instance's transaction log that the downstream instance has received. For those versions of AMPS, messages from other instances are not considered when determining the resynchronization point. --- # Replication Security AMPS allows authorization and entitlement to be configured on replication destinations. For the instance that receives connections, you simply configure `Authentication` and `Entitlement` for the transport definition for the destination, as shown below: ```xml showLineNumbers amps-replication amps-replication 10005 amps-default-entitlement-module amps-default-authentication-module ... ``` For incoming connections, configuration is the same as for other types of transports. For connections from AMPS to replication destinations, you can configure an `Authenticator` module for the destination transport. `Authenticator` modules provide credentials for outgoing connections from AMPS. For authentication protocols that require a challenge and response, the `Authenticator` module handles the responses for the instance requesting access. ```xml showLineNumbers fix topic amps-1 async amps-1-server.example.com:10004 amps-replication amps-default-authenticator-module ``` --- # Two-Way Replication Two-way replication, sometimes called _Back Replication_, is a term used to describe a replication scenario where there are two instances of AMPS -- termed `AMPS-A` and `AMPS-B` for this example. In a two-way replication configuration, messages that are published to `AMPS-A` are replicated to `AMPS-B`. Likewise, messages which are published to `AMPS-B` are replicated to `AMPS-A`. This replication scheme is used when both instances of AMPS need to be in sync with each other to handle a failover scenario with no loss of messages between them. This way, if `AMPS-A` should fail at any point, applications can immediately fail over to the `AMPS-B` instance, allowing message flow to resume with as little downtime as possible. To enable two-way replication, each AMPS instance defines a replication `Transport` to receive incoming messages and a replication `Destination` to deliver messages to the other instance. For details on configuring a replication transport, see [Configuring Incoming Replication Transports](/docs/amps-user-guide/replication/config-incoming-replication) and for configuring destinations, see [Configuring Outgoing Replication Destinations](/docs/amps-user-guide/replication/config-outgoing-replication). Notice that servers are intended to function as failover partners. Since a publisher may fail over between these two instances, the `Destination` on each instance that replicates to the other instance is configured to use `sync` message acknowledgment. This ensures that a publisher does not consider a message to be persisted until all of the failover partners have received and persisted the message. :::danger When configuring a set of instances for failover, it is important that the instances use `sync` message acknowledgment among the set of instances that a given client will consider for failover. It should never be possible for a publisher to fail over from one instance to another instance if the replication link between those instances is configured for `async` acknowledgments. ::: Starting with the 5.0 release, when AMPS detects back replication between a pair of instances, AMPS will prefer using a single network connection between the servers, replicating messages in both directions (that is, to both destinations) over a single connection. This can be particularly useful for situations where you need to have messages replicated, but only one server can initiate a connection. For example, when one of the servers is in a DMZ, and cannot make a connection to a server within the company. AMPS also allows you to specify a replication destination with no InetAddr provided. In this case, the instance will replicate once the destination establishes a connection, but will not initiate a connection. When both instances specify an InetAddr, AMPS may temporarily create two connections between the instances while replication is being established. In this case, after detecting that there are two connections active, AMPS will close one of the network connections and allow both AMPS instances to use the remaining network connection to publish messages to the other instance. Notice that using a single network connection is simply an optimization to more efficiently use the available sockets. This does not change the way messages are replicated or the replication protocol, nor does it change the requirement that all messages in a journal are replicated to all destinations before a journal can be removed. --- # Understanding Replication Message Routing An instance of AMPS will replicate a message to a given `Destination` when all of the following conditions are met (and there is an active replication connection to the `Destination`): 1. The message must be recorded in the transaction log for this instance (that is, the topic that the message is published to must be recorded in the transaction log with the appropriate message type). 2. The AMPS instance must be configured to replicate messages with that topic and message type to the `Destination`. (If a content filter is included in the replication configuration, the message must also match the content filter.) 3. The message must *either* have been directly published to this instance, or if the message was received via replication, the `Destination` must specify a `PassThrough` rule that matches the AMPS instance that this instance received the message from. 4. The message must not have previously passed through the `Destination` being replicated to; replication loops are not permitted. Each instance that receives a message evaluates the conditions above for each `Destination`. The same process is followed when replaying messages from the transaction log while resynchronizing the downstream replication destination. To verify that a given set of configurations replicates the appropriate messages, start at each instance that will receive publishes and trace the possible replication paths, applying these rules. If, at any time, applying these rules creates a situation where a message does not reach an instance of AMPS that is intended to receive the message, revise the rules (typically, by adjusting the topics replicated to a `Destination` or the `PassThrough` configuration for a `Destination`) until messages reach the intended instances regardless of route. --- # Replicating Messages Between Instances AMPS has the ability to replicate messages to downstream AMPS instances once those messages are stored to a transaction log. Replication in AMPS involves the configuration of two or more instances designed to share some or all of the published messages. With AMPS replication, an upstream (or source) instance delivers messages to a downstream instance. Replication is typically used to improve the availability of a set of AMPS instances by creating a set of servers that hold the same messages, where each server can take over for the others in the event of a network or server failure. Since AMPS replication operates on individual messages, replication is also an efficient way to split and share message streams between multiple sites where each downstream site may only want a subset of the messages from the upstream instances. AMPS replication uses a leaderless, "all nodes hot" model. Any instance in a replication fabric can accept publishes and updates at any time, and there is no need for a message to be replicated downstream before it is delivered to subscribers (unless a subscriber explicitly requests otherwise using the `fully_durable` option on a bookmark replay). :::tip The only communication between instances of AMPS is through replication. AMPS instances do not share state through the filesystem or any out-of-band communication. AMPS high availability and replication do not rely on a quorum, controller instance, or leader instance. Each instance of AMPS processes messages independently. Each instance of AMPS manages connections and subscriptions locally. ::: AMPS supports two forms of message acknowledgment for replication links: _synchronous_ and _asynchronous_; these settings control when AMPS considers a message persisted. This controls when AMPS sends publishers `persisted` acknowledgments and the point at which AMPS considers a message to be fully persisted for bookmark subscribers. These settings do not affect when or how messages are replicated, or when or how messages are delivered to subscribers unless a subscriber explicitly requests this behavior. These settings only affect when AMPS acknowledges to the publisher that the message has been persisted. They do not affect the speed of replication, the priority of replication, or the guarantees that AMPS makes for ensuring that all replicated messages are acknowledged by the downstream instance before they can be removed from the transaction log. AMPS replication consists of a message stream (or, more precisely, a command stream) provided to downstream instances. AMPS replicates the messages produced as a result of `publish` and `delta_publish` and replicates `sow_delete` commands. AMPS does not replicate messages produced internally by AMPS, such as the results of `Views` or updates sent to a `ConflatedTopic`. When replicating queues, AMPS also uses the replication connection to send and receive administrative commands related to queues, as described in the section on [Replicated Queues](replication/queue\_replication). :::danger 60East recommends that any server that participates in replication _and_ that accepts publishes, SOW deletes, or queue acknowledgments directly from applications is configured with at least one `sync` replication destination. Without at least one destination that uses `sync` acknowledgment, an AMPS instance could be a single point of failure, resulting in possible message loss if the instance has an unrecoverable failure (such as a hardware failure) between the time that the server acknowledges a command to a client, and the time that a replication destination receives and persists the command. ::: Notice that AMPS replicates fully processed and merged messages when replicating the results of a `delta_publish` command or a publish to a topic that provides preprocessing and enrichment. That is, AMPS replicates exactly the information that is written to the local transaction log. AMPS does _not_ save the original command to the transaction log or replicate the original command. Instead, for topics recorded in the transaction log, AMPS stores the message produced by an update and replicates that complete message. --- # Loadable Authentication/Entitlements Modules In this release, AMPS includes optional modules that provide authentication and/or entitlement functionality: - The `libamps_http_entitlement` module makes requests to an external web service to validate authentication credentials and retrieve the set of applicable entitlements. With this module, you can provide both authentication and entitlement infrastructure. See [RESTFul Authentication and Entitlements](./http-auth-module) for details. - The `libamps_multi_authentication` module supports multi-mechanism authentication. In this release, the module supports both LDAP and Kerberos. This module does not provide an entitlements implementation. See [Multimethod Authentication Module](./multi-authentication-module) for details. - The `libamps_simple_access_entitlement` module restricts access to specific resources. This module is most often used to provide limited access to the AMPS Admin console. This module does not provide an authentication implementation, and does not consider individual user entitlements -- the restrictions applied by this module apply to all users. See [Simple Access Entitlements Module](./simple-access-module) for details. - The `libamps_oauth_authentication.so` module provides the ability to authenticate users using OAuth 2.0. See [OAuth Authentication](./oauth-module) for details. --- # Authentication The first part of securing AMPS is developing a strategy to verify the identity of connected clients. AMPS maintains an identity for each client connection, and uses that identity for entitlement requests. Once an identity is assigned to a connection, that identity stays the same for the lifetime of the connection. If an application needs to use different identities to work with AMPS, that application needs to make a separate connection for each identity. There are two ways that AMPS assigns an identity to a client: 1. When an application explicitly sends a `logon` command, AMPS uses the credentials in the message for the authentication process. If authentication is successful, AMPS associates the username provided in the initial logon with the connection. If authentication fails, AMPS closes the connection. 2. When an application issues any other command after connecting, but before sending a `logon` command, AMPS treats this as an _implicit_ logon and begins the authentication process with an empty username and password. If authentication is successful, AMPS associates an empty username with the connection. If authentication fails, AMPS closes the connection. AMPS does not allow implicit logon by default in 5.0 and later versions. However, you can enable implicit logon as described below. In both cases, authentication occurs through the AMPS security infrastructure. When authenticating a client, AMPS locates the authentication module in use for the client's `Transport` (or, for the admin interface, the special `amps-admin` transport). If there is an authentication module specified for that `Transport`, AMPS uses that module. Otherwise, the `Transport` uses an instance of the authentication module specified for the instance. When the configuration for the instance doesn't include an instance level authentication module, the default module for the `Transport` is `amps-default-authentication-module`, which requires a logon, but accepts any username and password provided and sets the authenticated username to an empty string. Once AMPS has located the module instance, AMPS provides the username and the password to that instance of the module. The module can accept the credentials, reject the credentials, or return a challenge that the application must respond to. When the module returns a challenge, the connection remains unauthenticated until the application requesting authentication responds to the challenge and the module accepts the response. For most production systems, AMPS security is integrated with the overall security fabric of the organization. 60East provides the _AMPS Server SDK_ to help developers create authentication modules that implement the unique policies and procedures required by a particular organization. AMPS does not, itself, enforce an explicit timeout on the authentication process. However, the thread used to authenticate is managed by AMPS, so if the authentication process takes an extended period of time, AMPS may report a potentially stuck thread or (in extreme cases) prompt a server shutdown. 60East recommends that the authentication process complete as quickly as possible, especially since the authentication process will limit how quickly clients can connect to AMPS. The process should generally be subsecond if possible, and 60East recommends that the process not take longer than 30 seconds, even if the authentication server is under load. ## Provided Authentication Modules AMPS loads three simple authentication modules by default. These modules provide very simple policies for authentication, and are most useful in testing and development environments. | Module | Description | | --------------------------------------- | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `amps-default-authentication-module` | Allows any username and password. Does not allow implicit logon by default. Does not provide the username to AMPS by default. | | `amps-implicit-authentication-module` | Allows any username and password. Allows implicit logon by default. Does not provide the username to AMPS by default. | | `amps-default-no-authentication-module` |

    Does not allow authentication regardless of the username and password provided.

    This can be useful for testing application behavior when logon is denied, or for setting a policy for the instance that individual transports must override.

    | For more information about the available configuration options for these modules, refer to [Configuring Authentication](/docs/amps-user-guide/securing/configuring-authentication). AMPS also includes two modules that integrate with other authentication systems. These modules must be explicitly loaded and configured. | Module | Description | | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `libamps_multi_authentication.so` | This module supports multiple authentication mechanisms. In this release, LDAP and Kerberos are provided. See [Multimethod Authentication Module](./multi-authentication-module) for details. | | `libamps_http_entitlement.so` | This module makes requests to an external RESTful service for authentication. This module will typically also handle entitlements for the Transports that use it. See [RESTful Authentication and Entitlements](./http-auth-module) for details. | ## Enabling Implicit Logon 60East recommends using explicit logon commands in your applications wherever possible, and the default authentication module disallows implicit logons. For backward compatibility with older versions of AMPS, AMPS includes the `amps-implicit-authentication-module` which allows implicit logon to restore the behavior of the previous AMPS versions. To use the `amps-implicit-authentication-module` for all of the transports in the instance, set the instance level `Authentication` to use this module, as shown below: ```xml showLineNumbers ... amps-implicit-authentication-module ... ``` --- # Loadable Authenticator Modules - AMPS includes a module, `libamps_multi_authenticator`, to provide credentials for outgoing replication connections. Notice that the default authenticator module, which is automatically loaded, can also be configured to provide credentials. See [Multimethod Authenticator](./multi-authenticator-module) for details. - AMPS includes a module, `libamps_exec_authenticator`, that executes an external application and provides the output of that process as the credentials for an outgoing replication connection. See [Command Execution Authenticator](./exec-authenticator-module) for details. --- # Configuring Authentication The `Authentication` element specifies the module to use for validating user identity. AMPS allows you to set the default `Authentication` for the instance as a whole, and also to set the `Authentication` on each `Transport` individually. `Authentication` elements are not required. The instance authentication defaults to using the `amps-default-authentication-module` if no `Authentication` element is specified for the instance. An individual `Transport` defaults to using the instance `Authentication` if no `Authentication` element is provided for that `Transport`. The [Authentication](/docs/amps-user-guide/securing/authentication) section describes how AMPS handles authentication and the default modules in more detail. Described below are the configuration items for setting `Authentication`. Expand each item for more details.
    `Module` This element specifies the name of the module that will be used for authentication. The value must be the `Name` of an authentication module loaded in the `Modules` section of the configuration file or one of the authentication modules that AMPS loads by default. By default, AMPS loads the authentication modules described in the following section.
    `Options` A list of supported features for the implemented library. AMPS allows you to pass options to the module by specifying elements within the `Options` element. The exact options that the module requires, if any, are determined by the creator of the module.
    ### Authentication Modules Loaded by Default AMPS loads the following authentication modules by default. Expand each item for more details.
    `amps-default-authentication-module` Authenticate any user, regardless of the credentials provided. Does not provide the user name to AMPS by default, and does not allow implicit authentication by default. This module accepts the following options:
    • `AllowSpoofing` - When set to `true`, this module provides the user name to AMPS. This option is set to `false` by default.
    • `RequireLogon` - When set to `true`, this module does not allow implicit logon. Connections must explicitly logon or the module will refuse to authenticate them. This option is set to `true` by default.
    • `RequireUsername` - When set to `true`, this module does not allow a logon unless a user name is provided. This option is set to `false` by default.
    `amps-implicit-authentication-module` Authenticate any user, regardless of the credentials provided. Allows implicit authentication. Does not provide the user name to AMPS by default. This module accepts the following option:
    • `AllowSpoofing` - When set to `true`, this module provides the user name to AMPS. This option is set to `false` by default.
    This module is provided to mimic the default behavior of the `amps-default-authentication-module` in versions prior to 5.0. To restore that behavior, set `amps-implicit-authentication-module` to the `Authenticator` for the instance.
    `amps-default-no-authentication-module` Do not authenticate any user.
    --- # Configuring Entitlement The `Entitlement` element specifies the module to use for validating permissions to resources within AMPS. AMPS allows you to set the default `Entitlement` for the instance as a whole, and also to set the `Entitlement` on each `Transport` individually. `Entitlement` elements are not required. The instance authentication defaults to using the `amps-default-entitlement-module` if no `Entitlement` element is specified for the instance. An individual `Transport` defaults to using the instance `Entitlement` if no `Entitlement` element is provided for that `Transport`. The [Entitlement](/docs/amps-user-guide/securing/entitlement) section describes how AMPS handles entitlements and the default modules in more detail. Described below are the configuration items for setting `Entitlement`. Expand each item for more details.
    `Module` This element specifies the name of the module that will be used for entitlement. The value of this element must be the `Name` of an entitlement module loaded in the `Modules` section of the configuration file or one of the entitlement modules that AMPS loads by default. By default, AMPS loads the entitlement modules described in the following section.
    `Options` A list of options to provide to the module for this instance of the module. AMPS allows you to pass options to the module by specifying elements within the `Options` element. The exact options that the module requires, if any, are determined by the creator of the module.
    ### Entitlement Modules Loaded by Default AMPS loads two entitlement modules by default. For more information about these modules, refer to the [Entitlement](/docs/amps-user-guide/securing/entitlement) section. --- # Entitlement The AMPS entitlement system controls access to individual resources in AMPS. Each entitlement request consists of a user, a specific action and, where applicable, the type of resource and the resource name. For example, an entitlement request might arrive for the user `Janice` to `write` (that is, publish) to the `topic` named `/orders/northamerica`. Another entitlement request might be for the user `Phil` to `logon` to the instance. A third request might be for the user `Jill` to `read` (that is, subscribe or run a SOW query) from the topic named `/orders/pacific/palau`. When checking entitlements, AMPS locates the entitlement module in use for the `Transport` that the client is connecting on (or, for the admin interface, the special `amps-admin` transport). If there is an entitlement module specified for the `Transport`, AMPS uses that module. Otherwise, AMPS uses an instance of the entitlement module specified for the instance. When the configuration file for the instance doesn't specify an instance level entitlement module, the default module for the `Transport` is `amps-default-entitlement-module`, which allows all permissions for any user. AMPS caches the results of the entitlement check until the cache is explicitly reset, the transport is disabled, entitlements are disabled and re-enabled, or the AMPS server restarts. You can clear the entitlement cache for all users using the AMPS Administrative Actions. You can clear the entitlement cache for a single user using the AMPS external API. When the entitlement cache is cleared, AMPS disconnects the user. This ensures that, when the user reconnects, the user only has access to resources that match the current set of entitlements. AMPS checks entitlements for a command when processing the command and does not recheck permissions after the command is processed. For example, when `Jill` subscribes to `/orders/pacific/palau`, AMPS checks entitlements when creating the subscription. If the entitlement check returns an entitlement content filter, AMPS includes that entitlement filter on the subscription. Once the subscription has been created, AMPS applies the filter as a part of the standard filtering process, but AMPS does not check entitlements for the subscription as further messages arrive. ## Entitlement Resource and Permission Types The following table lists the resource types that AMPS provides: | Resource Type | Description | | ------------------- | -------------------------------------------------------------------------------------- | | `logon` | Permission to log on to the AMPS instance. | | `replication_logon` | Permission to log on to the AMPS instance as a replication source. | | `topic` | Permission to receive from or publish to a specific topic. | | `admin` | Permission to read admin statistics or perform admin functions from the web interface. | For the `topic` and `admin` resource types, AMPS also provides the name of the resource and whether the request is for a permission type of `read` from the resource or `write` to the resource. The table below shows how AMPS commands translate to entitlement types: | AMPS Command | Entitlement Type | | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | |

    `delta_subscribe`,

    `sow`, `sow_and_subscribe`,

    `subscribe`, `sow_and_delta_subscribe`

    | `read` | |

    `delta_publish`, `publish`,

    `sow_delete`

    | `write` | | _Commands received over replication_ | `replication allowed` | :::tip Notice that topics that contain messages and requests to the administrative interface are considered to be a different resource type, and are treated differently by most entitlement systems. This allows you to set permissions to statistics and administrative actions separately from access to message data. For example, a system administrator could have access to all of the information in the administrative interface by being granted access to any resource of `admin` type, but could be denied access to any `topic` resource to disallow access to message data. ::: ### Adminstrator Actions using HTTP Administrator actions are implemented using an HTTP `GET` of the path that runs the action. This applies to both actions added using the `amps-action-on-admin` and built-in actions. This means that the AMPS entitlement system treats a request for an administrator action as a `read` request to an `admin` resource type. A user that is not entitled to `read` that resource will not be able to run the action. However, `write` access to the resource is **not required** to run the action. To limit access to administrator actions, restrict `read` access to those paths. Galvanometer will deactivate controls for actions that the current user is not entitled to, as as described in [Entitlement to Administrator Actions](../monitoring/galvanometer#entitlement-to-administrator-actions). ## Entitlement Caching AMPS does not present a request to the entitlement module each time that an entitlement check is needed. Instead, AMPS presents the request the first time the entitlement is needed, and then caches the results from the module for subsequent entitlement checks. This improves performance, although it also means that when a module that reads entitlements from an external source (such as a central directory of permissions), that may change without requiring a restart of the AMPS instance, that module will need to establish a policy for resetting the entitlement cache. ## Regular Expression Subscriptions Each request from AMPS is for a specific resource name. When a client requests a regular expression subscription, AMPS makes a request for each topic that matches the subscription at the point that AMPS has a message to deliver for that topic. For example, if the user `Nina` enters a subscription for `/parts/(mechanical|electrical)`, AMPS will make a request to the entitlement module for `/parts/mechanical` when there is a message to deliver for that topic, and will make a separate request for `/parts/electrical` when there is a message to deliver for that topic. ## Content Filtered Entitlements The entitlement system offers the ability to enforce content restrictions on subscriptions. When AMPS requests `read` access to a `topic`, the module that performs entitlement can also return a filter to AMPS. This filter is evaluated independently of any filter on the subscription, and messages must match both the subscription filter and the filter provided by the entitlement to be returned to the application. If a message does not match the entitlement filter, the message is not delivered, regardless of whether the message matches the filters provided by the application. AMPS also offers the ability to enforce content restrictions on `publish` commands. When AMPS requests `write` access to a `topic`, the module that performs entitlement can return a filter to AMPS. This filter is then evaluated against messages published to that topic by that user. If the message being published matches the filter, AMPS allows the message. Otherwise, AMPS rejects the message. For `delta_publish` commands, the content filter applies to the incoming delta message rather than the existing message in the SOW or the merged message that is the result of the `delta_publish`. For `sow_delete` commands, content filtered entitlements apply to the message being removed. If the message to be removed matches the content filter, AMPS allows the delete. Otherwise, AMPS refuses to delete the message. A `sow_delete` command can specify a regular expression topic, which can match multiple topics. In this case, AMPS applies the permissions and entitlement filter for each topic before deleting messages in that topic. For example, in an instance that keeps the State of the World for topics `T1`, `T2`, and `T3`, a `sow_delete` command that specifies `^T.$` as the topic would match all three of those topics. For this command, AMPS will check the write entitlements for `T1` and apply the entitlement filter for topic `T1` to the delete from that topic, check the write entitlements for `T2` and apply the entitlement filter for `T2` to the delete from that topic, and check the write entitlements for `T3` and apply the write entitlement filter for `T3` to the delete from that topic. The topics that the delete applies to and the entitlement filters applied, are stored in the transaction log. ## Entitlement Select Lists For `read` entitlements, AMPS also allows the ability to restrict access to specific fields of a message. In this case, the entitlement module returns a _select list_. That select list will be applied to all messages delivered on that topic for that user. When both a content filter and a select list are provided, the content filter is applied before the select list is applied. This means that an entitlement system can filter on fields that a given user is not allowed to view. As with content filters in entitlements, an entitlement select list is evaluated independently of any select list provided by the subscriber. An entitlement select list is evaluated _before_ a select list provided by the subscriber, and the subscriber select list applies to the output of the entitlement select list. The following table shows some examples: | Message | Entitlement Select List | Subscriber Select List | Result | | -------------------------------------------------------- | ------------------------- | ----------------------- | ----------------- | | `{"a":1, "b":2}` | `-/a` | `+/a` | `{"b":2}` | | `{"a":1, "b":2}` | `-/,+/b` | (none) | `{"b":2}` | | `{"a":1, "b":2, "c": {"c1":1, "c2":2, "c3":3} }` | `-/,+/b,+/c/c2` | `-/,+/c/c1,+/c/c2` | `{"c":{"c2":2}}` | ## Message Queues Message queues, since they are implemented as views over topics in the transaction log, present a special situation for the AMPS entitlement system in two ways. First, receiving a message from a queue implies that the subscriber has the ability to modify the contents of the queue. Second, a queue can specify a `DefaultPublishTopic` to receive publishes. The AMPS entitlement system treats queues differently than other topics as follows: * `read` entitlement on a queue also grants a user the ability to delete (acknowledge) messages from the queue. No other write permissions are implied. * `write` entitlement on a queue grants the ability to publish to the queue, even in cases where AMPS translates that publish to the `DefaultPublishTopic` configured for the queue. No other permissions are implied. In particular, granting the `write` entitlement on a queue does not grant any entitlements on the `DefaultPublishTopic` directly. Even though the message is delivered to the `DefaultPublishTopic`, the `publish` command must publish to the queue topic. In all other respects, entitlements for message queues behave in the same way as entitlements for any other topic. ## Multiple Logical Topics in a Physical SOW Topic As described in the section on [using a single physical topic to hold multiple logical topics](../sow/pattern\_topics), entitlements for the topic are applied to the physical topic -- that is, the set of logical topics as whole. When entitlements are checked for this topic, AMPS provides the `Name` of the physical topic to the entitlement system. AMPS uses the permissions returned for that `Name` for every topic in the physical topic. AMPS does not support providing different entitlements to individual topics within a physical topic. However, an entitlement filter that uses the `TOPIC_NAME()` function (as described in the section on [Message Functions](../builtin\_functions/message-functions) in the [AMPS Functions](../amps-functions) section) can be used to restrict access to specific topics, since the `TOPIC_NAME()` will return the logical topic name. ## Disabling Entitlement When AMPS starts, the entitlement system is always enabled. AMPS provides an administrative action, `amps-do-disable-entitlements` (see the [Manage Security](/docs/amps-user-guide/actions/do-elements/do-manage-security) topic in the section on [Configuring AMPS for Automation with Actions](/docs/amps-user-guide/actions)), that disables the entitlement system until AMPS is restarted, or the system is explicitly re-enabled with an action. When the entitlement system is disabled: * AMPS no longer checks new requests for entitlements with the configured entitlement module. * All entitlement requests succeed (even requests for operations that have previously been disallowed). * AMPS does not cache the entitlement results for any operation. Notice that this means that all subscriptions succeed, no entitlement filters or entitlement select lists are applied to new subscriptions, and so on. In effect, any time that AMPS would check the entitlement cache or query the entitlement module, the operation immediately succeeds with full permissions. Disabling entitlements is designed to help mitigate failures in the entitlement system (including external systems that manage entitlements), allowing an administrator to maintain system availability at the cost of allowing full access to AMPS. This is most commonly used in shared development instances that are simultaneously doing application development and testing while working on updates to the set of allowable actions and/or the entitlement system itself. Entitlements can be re-enabled with the `amps-action-do-enable-entitlements` action (see [Manage Security](/docs/amps-user-guide/actions/do-elements/do-manage-security) in the section on [Configuring AMPS for Automation with Actions](/docs/amps-user-guide/actions)). When entitlements are re-enabled, AMPS: * Initializes new entitlement contexts for the instance and destroys previous contexts. * Clears the entitlement cache. * Again consults the entitlement module or the entitlement cache for new requests. Notice that when entitlements are re-enabled, AMPS _does not_ validate the current client logons or the current subscriptions to determine if the entitlement policy allows those logons or subscriptions. Likewise, AMPS _does not_ update entitlement filters or entitlement select lists when entitlements are re-enabled. Unlike an entitlement reset, AMPS does not disconnect connected clients when the entitlement system is re-enabled. This means that any existing client connections are maintained (whether or not they would be allowed with the entitlement system enabled) and any existing subscriptions are maintained (whether or not they would be allowed with the entitlement system enabled). 60East recommends running an entitlement reset (see [Manage Security](/docs/amps-user-guide/actions/do-elements/do-manage-security) in the section on [Configuring AMPS for Automation with Actions](/docs/amps-user-guide/actions)) after re-enabling entitlement to ensure that all connections and subscriptions use the current entitlement policy. Otherwise, the state of connections and subscriptions may not match the current policy. This can lead to a client receiving messages that it is not currently entitled to (if the current policy is more restrictive than when the connection and subscription were created), or not receiving messages that it is currently entitled to (if the current policy is less restrictive than when the connection and subscription were created). ## Provided Entitlement Modules AMPS loads two simple entitlement modules by default. These modules provide very simple policies for entitlements, and are most useful in testing and development environments. | Module | Description | | --------------------------------------- | ---------------------------------------------------------------------------- | | `amps-default-entitlement-module` | Allows any user to access any resource. | | `amps-default-no-entitlement-module` | Denies access to all resources for all users. | AMPS also includes two modules that can be used to manage entitlements. These modules must be explicitly loaded and configured. | Module | Description | | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `libamps_simple_access_entitlement.so` | This module provides a simple allow/deny list for all users on the transport. See [Simple Access Entitlements Module](./simple-access-module) for details. | | `libamps_http_entitlement.so` | This module makes requests to an external RESTful service for authentication. This module will typically also handle entitlements for the Transports that use it. See [RESTful Authentication and Entitlements](./http-auth-module) for details. | --- # Command Execution Authenticator AMPS includes a module that can provide credentials for outgoing replication connections using the results of an external process. This is designed for cases where a site has an authentication system that requires short-lived tokens, and where the installation does not use Kerberos, so the multimechanism authentication module cannot be used. In this release, the exec authenticator module is provided with AMPS, but is not loaded by default. This module is an optional extension to the AMPS product, and while it is included with the AMPS distribution, the module must be explicitly loaded, enabled, and configured. :::danger This module runs an external application with the credentials of the AMPS server itself. Avoid using this module unless you fully trust the application, have verified that the command line provided is correct and cannot use another module (or a custom module) for authentication. ::: ### When to Use the Exec Authenticator Module 60East recommends using this module when a replication connection is authenticated, when a system other than Kerberos is in use, when the credentials to be used cannot be provided in the configuration file or stored in the filesystem and when an executable program or script is available that can produce the credentials to use for a replication connection. * If the credentials can be provided in the configuration file or stored in a file in the file system, use the `amps-default-authenticator-module`. * If Kerberos is in use, use the [Multimethod Authenticator](multi-authenticator-module). ### Configuring AMPS to use the Exec Authenticator Module The exec authenticator module is included in the AMPS distribution but is not loaded in AMPS by default. To load the module, add the following configuration item to the `Modules` block in your AMPS configuration: ```xml showLineNumbers ... exec-authenticator libamps_exec_authenticator.so ... ``` This module does not require any options as a part of the module configuration and ignores any options provided when the module is loaded. This module supports the following options when used in an `Authenticator` block: | Option | Description | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Command` |

    Sets the command to run.

    This can be either an absolute path or a relative path based on the current working directory of the AMPS server process.

    The command supports the following expansions when the command runs:

    • `AMPS_USER_NAME` The user name that the authenticator will provide to the remote server.

      For example, the following `Command` runs the script `give_me_a_token.sh` from the path `/opt/site/gateways/`, passing the user name as a parameter:

      ` /opt/site/gateways/give_me_a_token.sh "{{AMPS_USER_NAME}}" `

    The authenticator will read the stdout of that command and provide the result as the authentication token for the connection.

    There is no default for this parameter.

    | | `MaxLength` |

    Sets the maximum number of bytes to read from the command.

    Default: `1024`

    | | `UserName` |

    Sets the user name to provide on this connection.

    There is no default for this parameter.

    | The module must be configured with a `Command` and `UserName`. Otherwise, the module fails to initialize and AMPS will halt the startup process. For example, the following configuration loads and configures the module to run the `auth-widget` program, in the current working directory of the AMPS process, in order to provide credentials. The `UserName` provided to AMPS, as well as on the command line for `auth-widget`, is sourced from the `USER` environment variable used during AMPS startup. The options provided on the `auth-widget` command line are needed for the program to generate the token and write it to standard output. ```xml showLineNumbers ... run-proc-authenticator libamps_exec_authenticator.so ... ... ... amps-replication my-failover-partner:4000 run-proc-authenticator ${USER} ./auth-widget --user "{{AMPS_USER_NAME}}" --output=stdout ``` --- # RESTful Authentication and Entitlements The AMPS distribution includes a module that provides authentication and entitlement via an external Web Service. For some installations, this module provides a convenient way to integrate with an existing authentication and entitlement infrastructure without creating an entitlement plugin. In this release, the HTTP authentication module is provided with AMPS, but is not loaded by default. This module is an optional extension to the AMPS product, and while it is included with the AMPS distribution, the module must be explicitly loaded, enabled, and configured. When using this module, AMPS requests permissions documents from an external service using `http` or `https`. The request to retrieve the permissions document includes the credentials provided with the client logon. If the request succeeds, the module considers the user to have successfully authenticated to AMPS. If the request to retrieve the permissions document fails, the module considers the user to have failed authentication. When authentication succeeds, the contents of the document returned specify the permissions that the module grants to the user. The web service module expects that the web service endpoint will follow RESTful (HTTP) semantics. For example, the module uses standard HTTP headers for authentication and expects that if the credentials provided are not valid, the endpoint will return an HTTP 403 status code (or equivalent). ## When to Use the Web Service Module The AMPS Web Service Authentication and Entitlement module can be a good option when: * The site does not have an existing authentication and entitlements infrastructure for AMPS. * It is more feasible to develop and test a standalone web service than to develop a server plugin for AMPS. * Applications need to integrate with an existing authentication and entitlement system that offers limited Linux or C/C++ support. * An application that will use another authentication scheme in production needs an easy way to test changes to entitlements and entitlement scenarios that are difficult to replicate in the production system. ## Permissions Document Format This section describes the format of the permissions documents used by the Web Service Authentication and Entitlement module. All documents are in JSON format, and consist of a set of permissions. The document _is not required_ to contain the user name: this is intentional, and allows systems to easily provide identical permissions for all users in a group without having to create unique documents. The entitlement document expresses each permission as a field of a JSON document. The following is an example of a permissions document: ```javascript showLineNumbers { "logon": true, "replication-logon" : false, "topic": [ { "topic": "test", "read": "/priority = 1", "write": false }, { "topic": ".*", "read": true, "write": true } ], "admin": [ { "topic": "^/amps/instance/.*", "read": true, "write": false }, { "topic": ".*", "read": false, "write": false } ] } ``` The Web Authentication Module processes the entitlements in document order. Going through the document in order, this set of entitlements specifies the following permissions: * This user is authenticated as having provided valid credentials for the user name submitted in the logon request. * This user has permission to log on to AMPS, as set by the logon field. * This user does not have permission to make a replication connection to AMPS. A replication connection that uses these credentials will be refused. * This user has read permissions to the topic `test` for messages that match the filter `/priority = 1`. This user does not have write permissions to the topic. * The user has read and write permissions to every other topic in the instance without content restrictions. * The user has read permissions to the administrative interface for information under the `/amps/instance` path. * The user has no other permissions to the administrative interface. The Web Service Authentication and Entitlement Module also allows fine-grained control of the topics that a given user is allowed to publish to via replication with the `replicated-topics` configuration element. The following permissions document shows sample permissions for a replication connection: ```javascript showLineNumbers { "replication-logon": true, "logon": false, "replicated-topics":["^/orders/NYC/.*", "/events/P1"], "user_name": "replication-user" } ``` This document specifies that the credentials provided are valid for the username `replication-user`. This user can only log on to AMPS via replication connections. The user has permission to replicate messages to the topic `/events/P1` (using an exact match) and topics that begin with `/orders/NYC`, as specified by the regular expression `^/orders/NYC/.*`. The user has no other permissions. In particular, the user cannot log on from an AMPS client, and could not publish to or subscribe to any topics even if `logon` was changed to `true` without an explicit `topic` permission. The structure of the permissions document is as follows: ### Client Transport Permissions These fields control access for applications to connect, including the `SQL` tab of the Galvanometer (which is an AMPS client application). | Field | Value | | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `logon` |

    Specifies permission for an application to log on to AMPS.

    When this field is present, the value of the field must be a boolean `true` or `false`.

    | | `topic` |

    Controls access to topics within AMPS.

    When this field is present, the value of the field must be a permission list, as described below.

    If this field is not present, the user cannot publish to or subscribe to any topics.

    | ### Admin Transport Permissions This field controls access to the administrative interface and statistics, including Galvanometer statistics. | Field | Value | | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `admin` |

    Controls access to the administrative interface.

    When this field is present, the value of the field must be a permission list, as described below.

    If this field is not present, the user has no access to the administrative interface.

    Notice that, because the Admin console uses HTTP, there is no equivalent of the logon or replication-logon permission for the Admin transport.

    | ### Replication Transport Permissions These fields control permission to replicate messages to this instance of AMPS. | Field | Value | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `replication-logon` |

    Controls permission for a replication connection to log on to AMPS.

    When this field is present, the value of this field must be a boolean `true` or `false`.

    | | `replicated-topics` |

    An array containing the topics that this user can replicate to.

    When this field is present, the value of the field must be an array of strings that specify topic names or regular expressions. For example, the following entry allows this user to replicate ONLY to topics that begin with `/orders/NYC`

    ` "replicated-topics":["^/orders/NYC/.*"] `


    If this field is not present in the permissions document, the user cannot replicate to any topics.

    | ### Connection Properties This field allows you to set the user name for this connection to a value different than that used for logon. | Field | Value | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `user_name` |

    Specifies that the module should set the authenticated user name of the connection to the provided value.

    This option is ignored when the module is in `EntitlementsOnly` mode.

    If this field is not present in the returned permissions document, the request authenticates the logon with the user name provided with the `logon` request from the client.

    | ### Permissions Lists Permissions lists within this document are arrays of entries. Each entry in the array is a JSON object with the following format: | Field | Value | | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `topic` |

    The name of the topic this permission definition applies to.

    This name can be either a literal value, or a regular expression to use to match topic names.

    A name is interpreted as a regular expression if it contains any characters used in regular expression matching (for example; `^`, `$`, `*`, `.` and so on). Regular expression matching provides full support for PCRE regular expressions.

    | | `read` |

    Defines the read permission for the topic.

    The value of this field can be either `true`, `false` or an AMPS filter.

    When the value is `true`, the module grants read permission to the topic with no restrictions. When the value is `false`, the module denies read permission. When the value is a filter, the module grants read permission to the topic only for those messages that match the filter.

    | | `write` |

    Defines the write permission for the topic.

    The value of this field can be either `true`, `false` or an AMPS filter.

    When the value is `true`, the module grants write permission to the topic with no restrictions. When the value is `false`, the module denies write permission. When the value is a filter, the module grants write permission to the topic only for those messages that match the filter.

    | | `select` |

    Defines the entitlement select list for this topic.

    When this value is present, the module applies the select list provided to the topic entitlement.

    Notice that this element should include only the select list specifier and not the enclosing brackets for the select list. For example, a valid `select` element might be:

    `"select":"-/,+/id,+/home/range" `
    | ### Indicating Authentication Failure To indicate authentication failure, the web service should typically return a `403` HTTP status code and return no authentication document. To provide default permissions in the event of an authentication failure (for example, during transition from an unauthenticated instance to an authenticated instance), the web service should return a document that validates the request as being allowed for a default user. For example, the following document specifies that the connection will use the user name `default-unauthenticated-permissions`, regardless of the user the connection attempted logon for. This user can make an application connection to AMPS for read-only access to topics that start with `PUBLIC-`. This user has no permission to the administrative interface. ```javascript showLineNumbers { "user_name":"default-unauthenticated-permissions", "replication-logon": false, "logon": true, "topic": [ { "topic": "^PUBLIC-", "read": true, "write": false} ] } ``` :::tip All connections for the same user name on the same Transport will use the same permissions. If a logon fails to validate, or should use default permissions, any permissions document returned should use the `user_name` directive to set a default user name or a user name with no permissions. Otherwise, the permissions document confirms that the request is valid for the provided user name, which can lead to unexpected or incorrect results. ::: ## Configuring AMPS to use Web Service Authentication and Entitlements The web service authentication and entitlement module is included in the AMPS distribution, but is not loaded by default. To load the module in AMPS, add the following configuration item to the `Modules` block of the AMPS configuration file: ```xml showLineNumbers ... web-entitlements libamps_http_entitlement.so ... ``` Options for the module may be set when the module is loaded, when the module is used for `Authentication` or `Entitlement`, or in both places. Options set when the module is loaded are inherited as the default values for all uses of the module in the instance. Options specified in an `Authentication` or `Entitlement` block override options set when the module is loaded. For `Authentication`, the module supports the following options: | Option | Description | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ResourceURI` |

    The URI to request when a user logs into AMPS using this module. This option is required.

    For this option, AMPS substitutes the placeholder `{{USER_NAME}}` with the name of the user being authenticated. For example, here are two possible values for the `ResourceURI`:

    `http://cred-server:8080/admin/group_policy.json`

    `http://cred-server:8080/{{USER_NAME}}.json`


    In the first case, AMPS requests a document with the current user name of the user logging on substituted for the `{{USER_NAME}}` component of the URI. In the second case, the document is requested with the credentials of the user connecting, but AMPS requests the same document for each user.

    There is no default for this parameter.

    | | `CredentialStore` |

    Identifier for the entitlement store where the retrieved permission sets will be stored. When used for authentication, this is the name of the entitlement cache that the module stores parsed information into until AMPS requests the information.

    In most cases, there is no need to set this parameter. When multiple transports use different `ResourceURI` values, but those transports are expected to maintain the same information, specifying a `CredentialStore` value can speed the logon process and reduce the number of copies of the parsed permissions document in memory. Likewise, if a module must ensure that permission sets are kept separate (so that, for example, an internal user named `'johndoe'` and a web user named `'johndoe'` have separate credentials), explicitly setting a `CredentialStore` value can make it easier to verify that the permission sets are completely separate.

    Default: The literal value of the `ResourceURI` parameter.

    | | `ConnectionTimeout` |

    The maximum amount of time to wait for a connection to the web server for each request, in milliseconds. If a connection is not made within the specified time, the module stops the connection attempt and the request fails.

    Default: `2000`

    | | `RequestTimeout` |

    The maximum amount of time to wait for the web server to return a permissions document on each request, in milliseconds. If the server does not return a permissions document within the specified time, the module closes the connection and the request fails.

    Increasing this timeout above the default value may produce potential stuck thread warnings if the web service is slow to respond.

    Default: `5000`

    | | `RetryCount` |

    Sets the number of times to retry the request if retrieving the authentication document fails for any reason.

    Default: `0` (only try once)

    | | `HTTPHeader` |

    Sets a header to add to the HTTP request.

    The configuration can specify any number of `HTTPHeader` elements, and the module will provide each of the specified headers with the authentication request.

    There is no default for this option. If no `HTTPHeader` option is included, the module provides a standard set of headers.

    This option supports variable replacement during an authentication request, as described in the following table.

    This option supports expansion of the `AMPS_USER_NAME` (and the `USER_NAME` backward compatibility form) when the module is in `EntitlementsOnly` mode.

    | | `EntitlementTimeout` |

    Optionally, sets the amount of time to consider an entitlements document for a given user to be valid.

    After this timeout period, AMPS will check each entitlements document returned for that user to see if the entitlements have changed. If an authentication request returns a different entitlements document, the module will reset permissions for all connections currently connected with that username (which will disconnect those connections and clear the entitlement cache).

    When this option is not provided, the module retains permissions until all connections with the current user name are disconnected, then removes cached permissions in the module and clears the AMPS entitlement cache.

    This option is not supported in `EntitlementsOnly` mode.


    This option accepts a number of milliseconds or an AMPS interval.

    The granularity for the `EntitlementTimeout` is a second. Fractions of a second, or values less than one second, may be rounded or truncated.

    | | `ReuseConnections` |

    Optionally, sets the module to reuse connections to the entitlement service when possible. By default, the module creates a new connection for each entitlement request.

    When set to `enabled`, the module will reuse HTTP connections when possible.

    This option is only supported within the `Modules` definition, and is set for all uses of the module within the configuration. The option will be ignored if it is set within the `Authentication` or `Entitlement` blocks.

    Default: `disabled`

    | | `ServerAcceptsEmptyAuthId` |

    By default, the module will refuse authentication requests that do not contain an auth ID without allowing the request to reach the server. Instead, the module returns an authentication failure to AMPS immediately.

    This is designed to protect the authentication web service from requests that cannot succeed.


    In some environments, the web service providing authentication will authenticate a user based entirely on the authentication token provided as the password, and will return the authentication ID for AMPS to use in the permissions document.

    When this option is set to `true`, the module will submit authentication requests to the service even when no authentication ID is provided on the logon request from the AMPS client.

    An authentication ID is still required for AMPS to be able to track permissions. When this option is set to `true`, the authentication service must set the auth ID in the permissions document when no auth ID is provided in the permissions request. Otherwise, the authentication request will fail.

    Default: `false`

    | The following tokens are expanded in the `HTTPHeader` element of an authentication request: | Authentication Header Token | Expansion | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `AMPS_CLIENT_NAME` | Client name provided in the logon request. | | `AMPS_CONNECTION_NAME` | The connection name for the connection requesting logon, as assigned by the AMPS server (based on the transport name). | | `AMPS_CORRELATION_ID` | The correlation ID provided on the logon request. | | `AMPS_MESSAGE_TYPE` | The message type of the logon request. | | `AMPS_PASSWORD` | The password provided with the logon request. | | `AMPS_REMOTE_ADDRESS` | Remote address from which the logon request was made. | | `AMPS_USER_NAME` | User name for the request. | | `CORRELATION_ID` |

    The correlation ID provided on the logon request.

    Legacy compatibility token

    | | `USER_NAME` |

    User name for the request.

    Legacy compatibility token

    | For `Entitlement` blocks, the module requires one of the following two options. These options are used to specify the entitlements to use for the context. | Option | Description | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ResourceURI` |

    Identifier for the credential store to use for this entitlement context. This is a synonym for the `CredentialStore` parameter. The module accepts this synonym to make it easier to verify that the `Entitlement` context and the `Authentication` context use the same values.

    There is no default for this parameter. Either the `ResourceURI` or `CredentialStore` must be provided. If both are provided, the module uses the value of the `CredentialStore`.

    | | `CredentialStore` |

    Identifier for the entitlement cache. When used for entitlement, this is the name of the entitlement cache that AMPS uses to look up the information.

    There is no default for this parameter. Either the `ResourceURI` or `CredentialStore` must be provided. If both are provided, the module uses the value of the `CredentialStore`.

    | For example, the following configuration loads the module and sets default values that are used for client transports. The module is also used for the admin interface, but that interface uses a separate credential store. Notice that, since the `HTTPHeader` options are set when the module is loaded, the custom headers are provided everywhere the module is used, regardless of the `ResourceURI` or `CredentialStore` values. ```xml showLineNumbers ... web-entitlements libamps_http_entitlement.so http://permissions-server:8080/{{USER_NAME}}.json x-tracking-id: {{CORRELATION_ID}} x-origin: AMPS ... web-entitlements web-entitlements localhost:8085 Basic realm="AMPS Admin" web-entitlements AdminCreds http://permissions-server:8080/admin/{{USER_NAME}}.json web-entitlements http://permissions-server:8080/admin/{{USER_NAME}}.json AdminCreds json-tcp tcp 9007 json amps any-tcp tcp 9090 amps ``` ### Using HTTPS for Authentication and Entitlement Requests The Web Service Authentication and Entitlement Module optionally supports `https` entitlement requests. When the `ResourceURI` uses `https` as the scheme, the module will attempt to use `https` to connect to the web service. By default, the module attempts to verify the identity of the remote web service, which requires a key file containing the key for the certificate authority that signed the certificate for the remote web service. AMPS does not require that the certificate and key provided for outgoing `https` requests be the same certificates used for incoming SSL connections to AMPS. However, if you have configured AMPS to accept SSL connections from AMPS clients, the certificates you use for those connections are often suitable for outgoing web authentication module connections, and the same certificates can be provided in both sections of the configuration file. | Option | Description | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Certificate` |

    The certificate to use for the AMPS connection to the web service. When configured, this certificate can be provided to the web service if the web service requests client authentication for the connection.

    There is no default for this parameter.

    | | `Key` |

    The key file to use for the AMPS connection to the web service. When configured, this key can be used for client authentication if the web service requests client authentication for the connection.

    There is no default for this parameter.

    | | `CAKey` |

    The certificate authority key. When configured, this key can be used for the AMPS server to verify the identity of the web service.

    There is no default for this parameter. The `CAKey` must be provided when `AllowUnverfiedPeer` is set to `false`, the default.

    | | `AllowUnverifiedPeer` |

    Specifies whether AMPS requires the web service to identify itself. When this is set to false, the default, AMPS requires that the web service provides a certificate that can be verified with the configured `CAKey`.

    Default: `false`

    | | `AllowSelfSigned` |

    Specifies whether AMPS will accept self-signed certificates for `https` connections.

    Default: `false`

    | The following configuration file shows a common way to configure the module for testing purposes when working with a server that accepts `https` connections. While the outgoing connection will use SSL, the module will not provide certificates to the server, or request that the server verify its identity. ```xml showLineNumbers web-entitlements libamps_http_entitlement.so https://permissions-server:443/{{USER_NAME}}.json true true ``` The following configuration file demonstrates how to configure the module to verify the identity of the remote server, provide verification of the AMPS server identity to that server, and require that all certificates be signed by a certificate authority. ```xml showLineNumbers web-entitlements libamps_http_entitlement.so https://permissions-server:443/{{USER_NAME}}.json /etc/security/amps-cert.pem /etc/security/amps-key.pem /etc/security/ca.pem ``` ## Permissions Management and Request Flow This section describes the request flow for the Web Service Authentication and Entitlement Module when used in the default mode. Notice that authentication and entitlement are two separate steps. The module obtains the set of permissions during the authentication step, and then provides responses to AMPS entitlement requests during the entitlement step. What this means is that, in the default mode, a user must have authenticated using the module for an entitlement request to be allowed. ### Authentication Step 1. Logon request received from the client. 2. Module requests an entitlement document (via `GET`) from a specified Web Service. The credentials in the logon request are provided as the credentials for the `GET`. The module supports both Basic and Digest authentication to the Web Service, as described in RFC 7616 (HTTP digest access authentication) and RFC 7617 (The 'Basic' HTTP authentication scheme). 3. If AMPS cannot retrieve the entitlement document using the provided credentials, AMPS returns a failure for the logon request. 4. If the Web Service Authentication and Entitlement module already has a parsed entitlement document for this user in the `CredentialStore` for this request, the module simply returns success for the authentication request. 5. Otherwise, the module parses the entitlement document and stores the entitlements in the `CredentialStore`. If the module can't successfully parse the document, it returns a failure for the authentication request. ### Entitlement Step 1. The module looks up the user name in the `CredentialStore` for the request. If the module has no stored entitlements for the user, the module denies the request. 2. The module looks for a matching entitlement. Entitlements for a user are searched in exactly the same order in which they appear in the document. The module uses the first entitlement that matches the request. If no entitlements match, the module denies the request. 3. The module checks the entitlement to see if the entitlement grants access to the user or disallows access to the user. If the entitlement disallows access, the module denies the request. 4. The module allows the request and applies any filter specified in the matching entitlement. ### Entitlement Reset The module caches the set of entitlements for a user while that user is connected. As described in the previous section, the module parses the returned permissions document if the module does not have an existing set of entitlements for that user or if the `EntitlementTimeout` has expired and the document has changed. The module provides two mechanisms for automatically resetting the entitlement cache and updating permissions for the user: 1. The module provides "reset cache on disconnect" entitlement behavior when in authentication and entitlement mode. The module resets both the AMPS entitlement cache and the information on the parsed permissions document for a given user when all of the connections for that user are closed. In practical terms, this means that changing the entitlement document returned by the web service has no immediate effect on the permission set that AMPS enforces. AMPS will continue to use the permissions in the document returned from the first logon request for that user until all connections for that user have been closed. When used in authentication and entitlement mode, this functionality is always active; when the last connection for a given user disconnects, the AMPS entitlement cache and the module entitlements are reset. Notice that this also means that if AMPS entitlements are reset for a user (for example, using the `amps-action-do-reset-entitlement` action), this will also clear the module cache, since all connections for that user will be closed. Likewise, a user can always update permissions by completely logging out of AMPS and then logging back in. 2. Optionally, the module can set an `EntitlementTimeout`. When this is set, a change to the user permissions after the timeout period will reset any current connections for that user, clear the AMPS entitlement cache, and replace existing permissions with the contents of the permissions document returned. Notice that AMPS only has access to the entitlements document when a user is authenticated. AMPS does not retain the logon credentials for a user, and only retrieves documents from the web service during logon. This means that a change in entitlements can only be checked during the logon process for the same user name. :::tip This functionality is not available when the module is in `EntitlementOnly` mode, as described below. When used in `EntitlementOnly` mode, the module does not track connection or disconnection, so entitlements are cached in the module for the lifetime of the instance. ::: ### Entitlement Only Mode Starting in AMPS 5.3.1, the HTTP authentication and entitlements module supports a mode where another module can be used for authentication, and this module can be used for entitlements. To enable this mode, include the following configuration item in the `Modules` block that loads the module: ```xml showLineNumbers ``` When an `EntitlementOnly` element is provided, the module cannot be used to authenticate users. Instead, the module makes an _unauthenticated_ HTTP request the first time that permissions are requested for a given user ID. The permissions document returned is cached, and used for subsequent requests. When using the module in this mode, the module may not be used in an `Authentication` block. The module configuration or `Entitlement` block configuration must provide a `ResourceURI`, and may provide the other options normally accepted in the `Authentication` block (for example, `HTTPHeader`, `RetryCount`, and so on). In this mode, the module does not see connect and logon requests. Therefore, the module does not provide automatic entitlement cache reset in this mode and the module caches the permissions for a given user in a given credential store for the lifetime of the instance. ### Entitlement Only Request Flow This section describes the request flow when the module is operating in entitlement only mode. This is very similar to the flow in the default mode, except that all of the processing happens as a part of the first entitlement request for a given user ID, and no credential information is available for the request to the HTTP server. 1. Entitlement request is received from AMPS. 2. The module looks up the user name in the `CredentialStore` for the request. If the module has stored entitlements for the user, the module proceeds to step 6. 3. Module requests an entitlement document (via `GET`) from a specified Web Service. No credentials are provided on this request. 4. If the module cannot retrieve the entitlement document, the module returns a failure for the entitlement request. 5. The module parses the returned permissions document. If an error occurs while parsing the permissions document, the module returns a failure for the entitlement request. Otherwise, the module loads the parsed permissions into the `CredentialStore`. 6. The module uses the `CredentialStore` to look for a matching entitlement. Entitlements for a user are searched in exactly the same order in which they appear in the document. The module uses the first entitlement that matches the request. If no entitlements match, the module denies the request. 7. The module checks the entitlement to see if the entitlement grants access to the user or disallows access to the user. If the entitlement disallows access, the module denies the request. 8. If the module allows the request, any filters specified in the matching entitlement are applied. --- # Providing an Identity for Outbound Connections For outgoing replication connections, AMPS may need to provide an identity and credentials to the replication destination. AMPS uses a module type called an authenticator to provide those credentials and handle any challenge/response protocol required by the authentication module in the remote system. AMPS provides a default authenticator module, `amps-default-authenticator-module`, that is automatically configured as the Authenticator for the instance if no other instance Authenticator is provided. This module provides a user name with no password. To determine the user provided to AMPS, the module uses the value of the User option to the module if one is provided. Otherwise, the module uses the current user of the AMPS process. If the current user cannot be determined by the system, the module falls back to the value of the `USER` environment variable. The `amps-default-authenticator-module` provides the ability to send a specific password (available in version 5.3.0.0 and higher). To provide a specific password, use one of the following options: | Option | Description | | ----------------------------- | ---------------------------------------------------------- | | `Password` | Provide the contents of this option as the password. | | `PasswordFileName` | Read the password from the specified filename. | | `PasswordEnvironmentVariable` | Read the password from the specified environment variable. | The Authenticator used for a replication Destination must provide credentials that are accepted by the Transport of the remote instance that the Destination is connecting to. See [Configuring Outgoing Replication Destinations](/docs/amps-user-guide/replication/config-outgoing-replication) for information on configuring the Authenticator for a Destination. If an installation uses Kerberos for replication security, the AMPS server must be able to provide a Kerberos token to authenticate itself to a downstream instance. For this situation, the AMPS distribution includes an authenticator that can provide Kerberos tokens, as described in the section on the [Multimethod Authenticator Module](./multi-authenticator-module). The multi-authenticator also provides the ability to provide credentials to an LDAP server, with functionality similar to the `amps-default-authenticator-module`. The AMPS distribution also includes a module that can run an external program to provide an authentication token for an outgoing replication connection. See [Command Execution Authenticator](./exec-authenticator-module) for details. --- # Multimethod Authentication Module AMPS includes a module that supports the commonly used infrastructure for enterprise authentication. In this release, the module includes support for LDAP or Kerberos authentication. In this release, the multimechanism authentication module is provided with AMPS, but is not loaded by default. This module is an optional extension to the AMPS product, and while it is included with the AMPS distribution, the module must be explicitly loaded, enabled, and configured. This module provides authentication, but does not provide an entitlement mechanism. When planning a strategy for securing AMPS using this module, you will also need to plan a strategy to manage entitlements. ## When to Use the Multimechanism Authentication Module 60East recommends using this module when integrating AMPS authentication into an existing infrastructure. If your environment does not have an existing infrastructure that offers one of the authentication methods supported by this module, it is typically easier to use the HTTP authentication and entitlement module than it is to implement or deploy a new authentication system. The AMPS Multimechanism authentication module can be a good option when: * The site has an existing authentication infrastructure for users that need to be authenticated to AMPS, and that infrastructure supports authentication using: * Kerberos, _or_ * LDAP * The authentication infrastructure is relatively stable and well-supported, with support for adding AMPS to the set of applications that use this infrastructure. ## Setting Authentication Mechanisms To enable a particular authentication mechanism in the multimechanism authentication module, you simply provide configuration parameters for that mechanism. For example, if you provide configuration parameters for an LDAP server, the module will enable LDAP. If you provide configuration parameters for Kerberos, the module will enable Kerberos. When more than one authentication mechanism is enabled, the module will attempt to detect the authentication mechanism used for a given logon request based on the credentials provided. If the module cannot determine the mechanism to use for a given request, and there is more than one mechanism configured, the module defaults to the mechanism specified in the `DefaultAuthenticationMechanism` in the module options. Notice, however, that the module does not allow a mechanism that accepts arbitrary passwords (in this release, LDAP) to be configured with a mechanism that accepts passwords of a specific format (in this release, Kerberos). In this release, the practical result is that a given module can be configured to use Kerberos _or_ LDAP for authentication, but cannot be configured to use both. ## Configuring AMPS to use the Multimechanism Authentication Module The multimechanism authentication module is included in the AMPS distribution, but is not loaded in AMPS by default. To load the module, add the following configuration item to the `Modules` block in your AMPS configuration: ```xml showLineNumbers ... multimech-authentication libamps_multi_authentication.so ... ``` This module does not require any options as part of the module configuration and ignores any options provided when the module is loaded. This module supports the following options when used in an `Authentication` block: **Kerberos Options** | Option | Description | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Kerberos.Keytab` |

    Sets a keytab file to use for Kerberos authentication. This option must be set to the path of the file, which can be either an absolute path or a relative path based on the current working directory of the AMPS server process.

    When this option is specified, the module will provide Kerberos authentication and a `Kerberos.SPN` must be specified.

    There is no default for this parameter.

    | | `Kerberos.SPN` |

    Sets the Service Principal Name (SPN) to use for Kerberos authentication.

    When this option is specified, the module will provide Kerberos authentication and a `Kerberos.Keytab` must be specified.

    There is no default for this parameter.

    | **LDAP Options** | Option | Description | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `LDAP.Host` |

    Sets the host name to use for LDAP authentication.

    When this option is specified, the module will provide LDAP authentication. This parameter is required if any other `LDAP` parameter is specified.

    There is no default for this parameter.

    | | `LDAP.Port` |

    Sets the port number to use for LDAP authentication.

    When this option is specified, the module will provide LDAP authentication and an `LDAP.Host` must also be specified.

    Default: `389`

    | | `LDAP.ProtocolVersion` |

    Sets the version of the LDAP protocol to use.

    When this option is specified, the module will provide LDAP authentication and an `LDAP.Host` must also be specified.

    Default: `2`

    | | `LDAP.BaseDN` |

    Sets the base Distinguished Name (DN) to use for LDAP authentication.

    When this option is specified, the module will provide LDAP authentication and an `LDAP.Host` must also be specified.

    This parameter defaults to an empty string.

    | | `LDAP.ServiceAccountDN` |

    Sets the Distinguished Name (DN) for the service account to use for LDAP authentication.

    When this option is specified, the module will provide LDAP authentication and an `LDAP.Host` must also be specified.

    This parameter defaults to an empty string.

    | | `LDAP.ServiceAccountPasswordFile` |

    Sets a file from which to read the password for the service account to use for LDAP authentication.

    When this option is specified, the module will provide LDAP authentication and an `LDAP.Host` must also be specified.

    This parameter defaults to an empty string, which specifies that no password will be provided.

    | **General Options** | Option | Description | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AllowAnonymous` |

    When set to `enabled`, allows users to logon without providing a password. In this case, however, the authenticated username will be set to an empty string.

    Default: `disabled`

    | | `DefaultAuthenticationMechanism` |

    When provided, sets the authentication mechanism to use if AMPS cannot identify the type of authentication token provided by the connection.

    In this release, the value for this parameter can be either `Kerberos` or `LDAP`.

    There is no default for this option. If no `DefaultAuthenticationMechanism` is configured and AMPS cannot identify the type of authentication token provided by a connection, AMPS reports an error for that logon.

    | The module must be configured with at least one authentication method. Otherwise, the module fails to initialize and AMPS will halt the startup process. For example, the following configuration loads the module and configures the module to use LDAP authentication by contacting the server `myenterprise-auth-server` on port `9389`. In this case, the LDAP server does not require authentication. Otherwise, the configuration would provide the service account DN and password for the server. ```xml showLineNumbers ... multi-auth libamps_multi_authentication.so ... multi-auth myenterprise-auth-server 9389 localhost:8085 json-tcp tcp 9007 json amps any-tcp tcp 9090 amps ``` The configuration below loads the module and configures the module to use Kerberos for clients that provide a Kerberos token on logon. For Kerberos, AMPS will use the SPN `AMPS/host.domain.com` and the keytab file at `/path/to/amps.keytab`. ```xml showLineNumbers ... multi-auth libamps_multi_authentication.so ... multi-auth AMPS/host.domain.com /path/to/amps.keytab localhost:8085 json-tcp tcp 9007 json amps any-tcp tcp 9090 amps ``` --- # Multimethod Authenticator ## Providing Replication Credentials with the AMPS Multimechanism Authenticator Module AMPS includes a module that can provide credentials for outgoing replication connections, that is designed for use when the multimechanism authenticator module is in use for the destination AMPS instance. In this release, this module can provide credentials for both LDAP and Kerberos authenticator mechanisms. This module is provided with AMPS, but it is not loaded by default. This module is an optional extension to the AMPS product and while it is included with the AMPS distribution, the module must be explicitly loaded, enabled, and configured. ### When to Use the Multimechanism Authenticator Module 60East recommends using this module when a replication connection is authenticated and uses the AMPS multimechanism module with Kerberos configured. This module can also be useful when LDAP is configured, however, in many environments the password capabilities of the `amps-default-authenticator` module can be sufficient for LDAP authentication. ### Setting Authentication Mechanisms To enable a particular authentication mechanism in the multimechanism authenticator module, you simply provide configuration parameters for that mechanism. For example, if you provide configuration parameters for an LDAP server, the module will enable LDAP. If you provide configuration parameters for Kerberos, the module will enable Kerberos. ### Configuring AMPS to use the Multimechanism Authenticator Module The multimechanism authenticator module is included in the AMPS distribution, but is not loaded in AMPS by default. To load the module, add the following configuration item to the `Modules` block in your AMPS configuration: ```xml showLineNumbers ... multimech-authenticator libamps_multi_authenticator.so ... ``` This module does not require any options as a part of the module configuration and ignores any options provided when the module is loaded. This module supports the following options when used in an `Authenticator` block: **Kerberos Options** | Option | Description | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Kerberos.Keytab` |

    Sets a keytab file to use for Kerberos authentication. This option must be set to the path of the file, which can be either an absolute path or a relative path based on the current working directory of the AMPS server process.

    When this option is specified, the module will provide Kerberos authentication and a `Kerberos.SPN` must be specified.

    There is no default for this parameter.

    | | `Kerberos.SPN` |

    Sets the Service Principal Name (SPN) to use for Kerberos authentication.

    When this option is specified, the module will provide Kerberos authentication and a `Kerberos.Keytab` must be specified.

    There is no default for this parameter.

    | **LDAP Options** | Option | Description | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `LDAP.Username` | Sets the username to provide when authenticating to a destination that uses LDAP authentication.When this option is specified, the module will provide LDAP authentication. This parameter is required if the `LDAP.PasswordFile` parameter is specified.There is no default for this parameter. | | `LDAP.PasswordFile` | Specifies the file name from which to read the password to provide when authenticating to a destination that uses LDAP authentication.When this option is specified, the module will provide LDAP authentication and an `LDAP.Username` must also be specified.There is no default for this parameter. | The module must be configured with at least one method for providing credentials. Otherwise, the module fails to initialize and AMPS will halt the startup process. For example, the following configuration loads the module and configures the module to provide Kerberos tokens to the destination at `my-failover-partner:4000`. ```xml showLineNumbers ... multi-authenticator libamps_multi_authenticator.so ... ... ... amps-replication my-failover-partner:4000 multi-authenticator AMPS/host.domain.com /path/to/amps.keytab ``` --- # OAuth Authentication The AMPS distribution includes a module that provides authentication through OAuth 2.0 to an external authorization server. In this release, the OAuth authentication module is provided with AMPS, but is not loaded by default. This module is an optional extension to the AMPS product, and while it is included with the AMPS distribution, the module must be explicitly loaded, enabled, and configured. When using this module, AMPS contacts an authorization server to confirm that a connection to AMPS has provided valid credentials to access AMPS. This module does not provide entitlements, and would typically be used with another module to enforce entitlements. ## When to Use the OAuth Module The AMPS OAuth Authentication module can be a good option when: * The site does not have an existing authentication and entitlements infrastructure for AMPS. * The site has an existing OAuth authorization server deployed, and wants to use the same mechanism for access to AMPS. ## Configuring AMPS to use OAuth Authentication The OAuth Authentication module is included in the AMPS distribution, but is not loaded by default. To load the module in AMPS, add the following configuration item to the `Modules` block of the AMPS configuration file: ```xml showLineNumbers ... oauth-authentication libamps_oauth_authentication.so ... ``` Options for the module may be set when the module is loaded or when the module is used for `Authentication`. Options set when the module is loaded are inherited as the default values for all uses of the module in the instance. Options specified in an `Authentication` block override options set when the module is loaded. The module supports the following options: | Option | Description | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `TokenEndpoint` (required) |

    URI endpoint of the authorization server. The module will submit authorization requests to this URI.

    There is no default for this option.

    | | `RedirectURI` (required) |

    URI to which the authorization server redirects the user.

    There is no default for this option.

    | | `ClientID` (required) |

    Unique identifier issued by the authorization server for AMPS to identify itself to the authorization server.

    There is no default for this option.

    | | `ClientSecret` (required) |

    Secret used for AMPS to identify itself to the authorization server.

    There is no default for this option.

    | | `GrantType` |

    Type of OAuth grant or flow to request.

    Default: `authorization_code`

    | | `RequestTimeout` |

    The maximum amount of time to wait for the authorization server to return a result each request, in milliseconds. If the server does not return a response within the specified time, the module closes the connection and the request fails.

    Increasing this timeout above the default value may produce potential stuck thread warnings if the authentication service is slow to respond.

    Default: `5000`

    | | `RetryCount` |

    Sets the number of times to retry the request if retrieving the response fails for any reason.

    Default: `0` (only try once)

    | | `HTTPHeader` |

    Sets a header to add to the HTTP request.

    The configuration can specify any number of `HTTPHeader` elements, and the module will provide each of the specified headers with the request.

    There is no default for this option. If no `HTTPHeader` option is included, the module provides a standard set of headers.

    | For example, the following configuration loads the module and sets the module as the default authentication for all transports in the instance. The configuration then explicitly sets the admin interface to use AMPS default authentication (that is, developer mode authentication that allows access to all admin stats). ```xml showLineNumbers ... oauth-authentication libamps_oauth_authentication.so https://oauth.example.com/token app-specific-id-from-server validation-to-server http://localhost:3000 true ... oauth-authentication https://oauth.example.com/token app-specific-id-from-server token-token-token http://localhost:3000 true amps-default-authentication-module json-tcp tcp 9007 json amps any-tcp tcp 9090 amps ``` ### Using HTTPS for OAuth Requests The OAuth Authentication Module optionally supports `https` entitlement requests. When the `TokenEndpoint` uses `https` as the scheme, the module will attempt to use `https` to connect to the web service. By default, the module attempts to verify the identity of the remote web service, which requires a key file containing the key for the certificate authority that signed the certificate for the remote web service. AMPS does not require that the certificate and key provided for outgoing `https` requests be the same certificates used for incoming SSL connections to AMPS. However, if you have configured AMPS to accept SSL connections from AMPS clients, the certificates you use for those connections are often suitable for outgoing web authentication module connections, and the same certificates can be provided in both sections of the configuration file. | Option | Description | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Certificate` |

    The certificate to use for the AMPS connection to the web service. When configured, this certificate can be provided to the web service if the web service requests client authentication for the connection.

    There is no default for this parameter.

    | | `Key` |

    The key file to use for the AMPS connection to the web service. When configured, this key can be used for client authentication if the web service requests client authentication for the connection.

    There is no default for this parameter.

    | | `CAKey` |

    The certificate authority key. When configured, this key can be used for the AMPS server to verify the identity of the web service.

    There is no default for this parameter. The `CAKey` must be provided when `AllowUnverfiedPeer` is set to `false`, the default.

    | | `AllowUnverifiedPeer` |

    Specifies whether AMPS requires the web service to identify itself. When this is set to false, the default, AMPS requires that the web service provides a certificate that can be verified with the configured `CAKey`.

    Default: `false`

    | | `AllowSelfSigned` |

    Specifies whether AMPS will accept self-signed certificates for `https` connections.

    Default: `false`

    | ### Authentication Request Flow This section describes the request flow for the OAuth Authentication module. 1. Logon request received from the client. This logon request includes an OAuth token received from the authorization server. 2. AMPS provides the OAuth token to the authorization server and requests an access token with the parameters configured on the module. The module contacts the authorization server at the `TokenEndpoint` specified, providing the token from the client logon request and the parameters configured for the module. 3. The authorization server returns a reply. * If the reply contains an access token, authentication succeeds. * If the reply is empty, contains an error, or does *not* contain an access token, authentication fails. :::tip Notice that this module only handles authentication. The module does not set any policy for entitlements (resource access). ::: --- # Simple Access Entitlements Module The AMPS distribution includes a module that provides access to resources that meet specific patterns. In this release, the simple access entitlement module is provided with AMPS, but is not loaded by default. This module is an optional extension to the AMPS product, and while it is included with the AMPS distribution, the module must be explicitly loaded, enabled, and configured. When using this module, AMPS grants and denies permissions to resources based on the name of the resource. The name of the user is not considered by this module, so when this module is used every user has the same set of permissions for the transport. ## When to Use the Simple Access Module The AMPS Simple Access module can be a good option when: * There are specific topics for a transport that are allowed or denied, but no other restrictions on the transport. * There is no other entitlement system in use for the installation. Most often, the simple access module is used to allow access to the parts of the Admin console that do not modify the state of an AMPS instance, while refusing access to the parts of the Admin console that affect the instance state. ## Configuring AMPS to use the Simple Access Module The simple access entitlement module is included in the AMPS distribution, but is not loaded in AMPS by default. To load the module, add the following configuration item to the `Modules` block in your AMPS configuration: ```xml showLineNumbers ... simple-access libamps_simple_access_entitlement.so ... ``` Options for the module are set when the module is used for `Entitlement`. When used in an `Entitlement` block, the module requires the `AllowedTopics` and/or `DeniedTopics` options to be specified. | Option | Description | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AllowedTopics` |

    A regular expression that matches the topics that the module will allow access to. The module will grant access only to topics that match this regular expression and do not match the `DeniedTopics` regular expression.

    Defaults to `.*`, matching all topics in the instance.

    | | `DeniedTopics` |

    A regular expression that matches the topics that the module will deny access to. The module will grant access only to topics that do not match this regular expression.

    There is no default for this parameter. If not provided, the module does not consider any topics to be explicitly denied and will grant access to any topic that matches the `AllowedTopics` parameter.

    | | `GrantedPermissions` |

    When set, this option directs the module to grant only read or write permissions when a topic is allowed.

    This option accepts a value of `read` or `write`, specifying which permission the module will grant.

    There is no default for this parameter. If not provided, the module will grant both read and write permissions to any allowed topic.

    When using this option for the `Admin` transport, notice that the `administrator` actions use an HTTP `GET`, so they are considered read operations.

    | For example, the following configuration loads the module, uses the module for entitlements on the administrative console, and explicitly refuses access to paths beneath `/amps/administrator` -- the paths that might modify the state of the instance. Since `AllowedTopics` defaults to `.*`, all other topics are allowed. ```xml showLineNumbers ... simple-access libamps_simple_access_entitlement.so ... localhost:8085 simple-access ^/amps/administrator ``` --- # Protecting Data in Transit Using TLS/SSL AMPS provides the ability to use Secure Sockets Layer (SSL)/ Transport Layer Security (TLS) connections for communication with AMPS clients. See the [Configuring Transports](/docs/amps-user-guide/transports/configuring-transports) section and the documentation for the AMPS clients for details. AMPS uses TLS to encrypt network traffic between clients and servers. No information about the transport is passed to the AMPS authentication and entitlement system. Encryption at the network level is completely independent of the AMPS authentication and entitlement system, and these features can be used independently. Although AMPS ships with libraries that are current at the time of release, it is recommended that an installation of AMPS load the version of OpenSSL and Crypto libraries that are vetted and approved by the site. Using libraries other than the ones that ship with AMPS by default also provides the ability for the site to easily install patched versions of these libraries at whatever cadence is necessary, without changing the AMPS patch level or requiring an AMPS upgrade. See the [Externals](/docs/amps-user-guide/configuring-amps/instance-configuration.md#externals) section of the documentation on [Instance-Level Configuration](/docs/amps-user-guide/configuring-amps/instance-configuration) for details. :::tip In this version of AMPS, a transport configured to use TLS defaults to accepting only TLS version 1.1, TLS version 1.2, and TLS version 1.3 protocols. It is possible to enable older protocols using the `SecureSocketProtocols` configuration directive. 60East recommends using the default settings unless there is a specific reason to enable older protocols and the security implications of enabling older protocols are well understood. ::: ## Verifying Connection Identity using Mutual TLS (mTLS) AMPS supports certificate verification for incoming connections. To add certificate verification for incoming connections, add the `VerifyClient` option to the `Transport` and provide a set of trusted certificates to use for verification using the `CAFile` or `CAPath` parameter. If the certificate provided by an incoming connection has not been signed by one of the trusted certificates, AMPS will refuse the connection. AMPS also supports certificate verification for outgoing replication connections. To require that an outgoing replication connection verify the certificate for the destination, provide the `VerifyClient` option in the `Transport` for the `Replication` `Destination` and provide a set of trusted certificates to use for verification using the `CAFile` or `CAPath` parameter. If the certificate provided by the destination server has not been signed by one of the trusted certificates, AMPS will close the outgoing connection before attempting to log on. --- # Securing AMPS One of the most important considerations when using AMPS in production is keeping your data safe. This means both ensuring that subscribers only have access to the data that they are allowed to have access to and that only authorized publishers are allowed to publish messages into the system. This chapter describes the mechanisms within AMPS to protect access to AMPS resources through client, administrative and replication connections. In this chapter, we describe the AMPS security infrastructure and present general information about securing an AMPS installation. AMPS uses a plugin model for providing authentication and entitlement, and allows a great deal of freedom in how a given module implements security checks. This chapter discusses the concepts, principles, and guarantees that AMPS provides. The specific steps and configuration you use to secure an installation of AMPS depend on the plugin you use to secure AMPS. There are three aspects to securing connections to AMPS: 1. Authentication assigns an identity to a connection and verifies that identity. 2. Entitlement enforces permission to access AMPS and read or write AMPS resources, based on the identity assigned to a connection. 3. The AMPS process may also need to provide credentials to another AMPS instance (for example, to secure outgoing replication). AMPS installations typically create custom plugins for securing AMPS. These plugins integrate with the enterprise authentication and entitlement system, and are designed to enforce the policies for the specific site. For more information on developing modules for use with AMPS, contact 60East support for the AMPS Server SDK. The AMPS distribution includes an auxiliary module that contacts a web service for authentication and entitlement. This module is described in the section [RESTful Authentication and Entitlement](securing/http-auth-module.md). The AMPS distribution also contains an entitlement module that can be used to restrict access to specific topics for all users. This module is described in the section [Simple Access Entitlements](securing/simple-access-module.md). For applications that need to connect with an existing Kerberos or LDAP system for authentication, the AMPS distribution includes an auxiliary module that can use either a Kerberos or LDAP system for authentication. This module is described in the section [Multimethod Authentication](securing/multi-authentication-module.md). If an installation uses Kerberos for replication security, the AMPS server must be able to provide a Kerberos token to authenticate itself to a downstream instance. For this situation, the AMPS distribution includes an authenticator that can provide Kerberos tokens, as described in the section [Multimethod Authenticator](securing/multi-authenticator-module.md). If a site uses OAuth for validating credentials, the AMPS distribution includes an [OAuth Authentication Module](securing/oauth-module.md) for authentication. For a situation where an outgoing replication connection needs to obtain credentials from an external source (such as generating a single use token), AMPS offers a [Command Execution Authenticator](securing/exec-authenticator-module.md) to run a specific external command and read the output. --- # Configuring the State of the World (SOW) The `SOW` section of the configuration file specifies the configuration for the AMPS State of the World (SOW). AMPS supports several different types of `SOW` topics that can be configured as part of the SOW. A `Topic` acts as a last value cache to store data. For more information on SOW `Topic` last value caching, see the sections on [State of the World (SOW) Topics](../sow), [Querying the State of the World (SOW)](../sow-queries), [Out-of-Focus Messages](../oof), [State of the World Message Enrichment](../enrichment), [Incremental Message Updates](../delta-publish), and [Receiving Only Updated Fields](../delta-subscribe). A `Queue`, `LocalQueue`, or `GroupLocalQueue` provides a mechanism for ensuring that messages are processed by an application once, as described in the [Message Queues ](../queues)section of this guide. The differences between the queue types specify how the queue will behave in a replicated set of instances, as described in [Queue Replication Types](../queues/getting-started-with-amps-queues.md#queue-replication-types). The current state of messages that have not been delivered to applications can be queried as described in [Querying the State of the World (SOW)](../sow-queries). `View` topics are configured using one or more of `Topic`, `Queue`, `View`, or `ConflatedTopic` as the underlying source of information. The [Aggregation and Analytics](../views) section of this guide describes how to configure a `View`. A `View` also supports queries and subscriptions as described in [Querying the State of the World (SOW)](../sow-queries), [Out-of-Focus Messages](../oof), and [Receiving Only Updated Fields](../delta-subscribe). A `ConflatedTopic` is a way to mitigate message velocities that are too high for subscribers to efficiently process. It provides a way for those subscribers to consume data from a `Topic`, `View`, or another `ConflatedTopic`, while also supporting queries and subscriptions as described in [Querying the State of the World (SOW)](../sow-queries), [Out-of-Focus Messages](../oof), and [Receiving Only Updated Fields](../delta-subscribe). Described below are the configuration items available for `SOW`. Expand each item for more details.
    `Topic` Specifies that AMPS will record distinct messages for this topic in the SOW. SOW `Topic` definitions are used directly as a last-value cache, and are required for many of the advanced messaging features in AMPS such as out-of-focus notifications and delta messaging. SOW `Topic` definitions can also be used as the `UnderlyingTopic` for views, aggregates, and conflated topics. See the [Configuring Topics in a SOW](/docs/amps-user-guide/sow/configuring-topics-in-sow) section of this guide for information on configuring a `Topic`.
    `Queue` Defines a message queue. Rather than delivering each message to all matching subscriptions, message queues provide features to help ensure that each message is delivered to and processed by a single subscriber. See the [Message Queues](/docs/amps-user-guide/queues) section in this guide for a full description of their functionality. AMPS queues provide a variety of replication models. See the [Configuring Queues in a SOW](/docs/amps-user-guide/queues/configuring-queues-in-sow) section of this guide for information on configuring a `Queue`.
    `View` Defines a view over one or more SOW topics, conflated topics, or other views. A view can perform aggregation and can `JOIN` multiple topics together. It can also be based on a SOW topic of one message type and project results of a different message type. See the [Configuring Views in a SOW](/docs/amps-user-guide/views/configuring-views-in-sow) section of this guide for information on configuring a `View`.
    `ConflatedTopic` Defines a copy of a SOW topic or view that receives current value updates at a specified interval, conflating any changes to values that occur between the scheduled updates. See the [Configuring Conflated Topics in a SOW](/docs/amps-user-guide/conflated-topics/configuring-conflated-topics-in-sow) section of this guide for information on configuring a `ConflatedTopic`.
    --- # Configuring Topics in a SOW This section outlines the configuration options for recording a `Topic` in the State of the World (SOW). All SOW topics require a basic definition of the messages to be recorded. Described below are the required configuration items for a `Topic` in a `SOW`. Expand each item for more details.
    `Name` (required) The name of the SOW topic. By default, unique messages published to this topic will be stored in a topic-specific SOW database. Every SOW requires a method of determining which messages are unique. Several methods are provided within AMPS. See the [Understanding SOW Keys](/docs/amps-user-guide/sow/sow_keys) section for information on generating SOW Keys, and the following sections for relevant configuration items. If no `Name` is provided, AMPS accepts `Topic` as a synonym for `Name` to provide compatibility with versions of AMPS previous to 5.0. Notice that if the topic uses a `Pattern` tag (described below) to record multiple logical topics in a single physical topic, the `Name` defines the physical topic only.
    `MessageType` (required) The type of messages to be stored. To use AMPS generated SOW keys, the message type specified must support content filtering so that AMPS can determine the SOW key for the message. All of the default message types, except binary, support content filtering. Since the binary message type does not support content filtering, that type can only be used for a SOW when publishers use explicit keys. See the [Message Types](/docs/amps-user-guide/message-types) section for a discussion of the message types that AMPS loads by default. Some message types (such as Google Protocol Buffers) require additional configuration, and must be configured before using the message type in a SOW topic.
    Below is an example of a simple SOW topic configuration: ```xml showLineNumbers orders /orderId nvfix ./sow/%n.sow ``` ## Required Considerations A SOW `Topic` _must_ also consider the options outlined in the sections below. ### Storage and Recovery A SOW topic must either specify the `FileName` to be used for persisting the topic, or declare that the topic will be in-memory only by setting the `Durability` to `transient`. Described below are the configuration options for specifying the storage and recovery behavior for the topic, if the topic is not durably persisted or if the recovery file is not present. Expand each item for more details.
    `FileName` (required if `Durability` is `persistent`) The file where the SOW data will be stored. This element is required for SOW topics with a `Durability` of `persistent` (the default) because those topics are persisted to the filesystem. This is not required for SOW topics with a `Durability` of `transient`. This element should contain the path and the file name. The path can be either an absolute path or a path relative to the current working directory of the AMPS process. Within this element, the escape `%n` will be replaced with the `Name` and `MessageType` of the topic. This can be a convenient way to avoid having to retype the topic name in this element. Two different topics must not share the same file. Two instances of AMPS must not share the same file.
    `Durability` Defines the data durability of a SOW topic. SOW databases specified as `persistent` are stored to the file system and retain their data across instance restarts. Those specified as `transient` are not persisted to the file system and are recreated each time the AMPS instance restarts. Notice that when the `Durability` is `transient`, and the topic is recorded in the transaction log, each time AMPS starts, AMPS will recover the state of the topic from the transaction log. The recovery begins at the `RecoveryPoint` specified for the topic, which defaults to `epoch`, the beginning of the transaction log. (For persistent topics, AMPS recovers from the last message written to the SOW topic, or from the `RecoveryPoint` if the SOW file is removed.) Valid values: `persistent` or `transient` When a value of `persistent` is specified, the `FileName` element must be present. Synonyms: `Duration` is also accepted for this parameter for backward compatibility with configuration prior to 4.0.0.1. Default: `persistent`
    `RecoveryPoint` For SOW topics that are covered by the transaction log, the point from which to recover the SOW if the SOW file is removed, or if the SOW topic has `transient` duration. This configuration item allows two values:
    • `epoch` - Recovers the SOW from the beginning of the transaction log.
    • `now` - Recovers the SOW from the current point in the transaction log.
    Default: `epoch`
    `Expiration` Defines the length of time a record should remain in the SOW database for this topic. The expiration time is stored on each message, so changing the expiration time in the configuration file will not affect the expiration of messages currently in the SOW. AMPS accepts interval values for `Expiration`, using the interval format described at the start of this guide in the section on Units, or one of the following special values:
    • A value of `disabled` specifies that AMPS will not process SOW expiration for this topic. In this case, AMPS saves any expiration value set on a message by the publisher, but does not process expiration. This value must be set to `disabled` (the default) if `History` is enabled for this topic.
    • A value of `enabled` specifies that AMPS will process SOW expiration for this topic, with no expiration set by default. Instead, AMPS uses the value set on the individual messages (with no expiration set for messages that do not contain an expiration value).
    `Expiration` must be `disabled` if `History` is enabled. Default: `disabled` (messages never expire)
    ### Record Identity Definition Each SOW topic must define how AMPS will determine which messages are unique. Typically, record identity is based on the content of one or more fields within the message. An application can either have AMPS determine the key by specifying one or more `Key` fields or provide a SOW key with the `publish` command each time a message is published to AMPS. AMPS also provides the ability to provide a custom `SowKey` generator with a plugin module. See the [Understanding SOW Keys](/docs/amps-user-guide/sow/sow\_keys) section for a full discussion. Described below are the configuration options for specifying how AMPS determines the `SowKey` for a message. Expand each item for more details.
    `Key` Specifies an XPath-based identifier within each message that AMPS will use to generate a SOW key, which determines whether a message is unique. This element can be specified multiple times to create a composite key from the combined value of the specified `Key` elements. When one or more `Key` elements is specified for the SOW, AMPS generates the SOW key for each message. When no `Key` fields are specified and no `KeyGenerator` is specified, publishers must explicitly provide the SOW key for each message when the message is published. 60East recommends configuring a `Key` and having AMPS generate the SOW key for a message unless your application has specific needs that make this impractical. AMPS automatically creates a hash index for the set of fields specified in the `Key` elements. There is no default for this element.
    `KeyDomain` The seed value for `SowKeys` used within the topic when AMPS generates the SOW key. The default is the topic name, but it can be changed to a string value to unify `SowKey` values between different topics. For example, if your application has a `ShippingAddress` SOW and a `CreditRating` SOW that both use `/customerID` as the SOW key, you can use a `KeyDomain` to ensure that the generated `SowKey` for a given `/customerId` is identical for both SOW topics. This does not affect how AMPS processes the SOW topics, but can make correlating information from different SOW topics easier in your application. This option can only be specified when one or more `Key` fields are specified. When a SOW key generator module is used, or the publisher must send a SOW key, this option is not valid. Default: `Name` of the SOW topic.
    `KeyGenerator` Specifies the SOW key generator module to use for this topic. When this configuration element is present, AMPS calls the specified module to generate a SOW key for each incoming message. A `KeyGenerator` element contains the following elements:
    • `Module` (required within a `KeyGenerator` element) - The name of the module. This module must be loaded elsewhere in the configuration file.
    • `Options` - Contains one or more XML elements. These elements are provided to the key generator module as options. The options provided depend on the key generator. The creator of the key generator module must document the options for that module.
    Default: Unset (no SOW key generator module). When there is no SOW key generator module specified, AMPS uses the specified `Key` fields if the `Key` fields are provided. If no generator is specified and no `Key` fields are specified, AMPS requires publishers to set a SOW key on each message published.
    ### Memory and File Growth If the SOW topic will contain a large number of records, or if an individual record will exceed the default allocation size, the topic must define the allocation size. The SOW topic configuration also specifies how the SOW file is allowed to grow. See [SOW Parameters](/docs/amps-user-guide/operation/operations-best-practices.md#sow-parameters) in the [Operations Best Practices](/docs/amps-user-guide/operation/operations-best-practices) section for detailed recommendations. Described below are the configuration options for controlling how the file is allocated and how the file grows. Expand each item for more details.
    `SlabSize` The size of each allocation for the SOW file, as a number of bytes. When AMPS needs more space for the SOW, it requests this amount of space from the operating system. This effectively sets the maximum message size that AMPS guarantees can be stored in the SOW. This size includes headers set by AMPS on the message. 60East recommends setting this value only if you will be storing messages larger than the default `SlabSize` or if performance or capacity testing indicates a need to tune SOW performance. If you plan to store messages larger than the default setting, 60East recommends a starting value of several times the maximum message size. For example, if your maximum message size is 2MB, a good starting point for `SlabSize` would be 8MB. If it becomes necessary to tune the `SlabSize`, see [SOW Parameters](/docs/amps-user-guide/operation/operations-best-practices.md#sow-parameters) for a full discussion about tuning this setting. Default: `5MB` Maximum: `1GB`
    `InitialSlabCount` The number of SOW slabs that AMPS will allocate on startup. Default: `1` Maximum: `1024`
    ## Optional Considerations A SOW `Topic` _may_ also provide the following options (notice, though, that there are restrictions on how some of these options are used with other options). ### Indexing Options A SOW topic can declare additional hash indexes or direct AMPS to create memo indexes before a field is queried by an application to improve performance. AMPS automatically creates a memo index for a field within a SOW topic when that field is used (for example, is used in a view or is queried by an application). In addition, AMPS automatically creates a hash index (the primary key index) for the combination of fields used to define the SOW key. Indexing is described in more detail in the [Indexing SOW Topics](/docs/amps-user-guide/sow/sow\_indexing) section. Described below are the configuration options that allow you to manage index creation for a SOW topic. Expand each item for more details.
    `HashIndex` AMPS provides the ability to do fast lookup for SOW records based on specific fields. When one or more `HashIndex` elements are provided, AMPS creates a hash index for the fields specified in the element. These indexes are created on startup and are kept up to date as records are added, removed, and updated. The `HashIndex` element contains a `Key` element for each field in the hash index. AMPS uses a hash index when a query uses an exact string match for all of the fields in the index. AMPS does not use hash indexes for range queries or regular expressions. AMPS automatically creates a hash index for the set of fields specified in the set of `Key` fields for the SOW, if those fields are specified.
    `Index` AMPS automatically creates memo index fields as needed. This can include the first time a particular field is used in a query. AMPS supports the ability to create memo indexes for specific fields during startup using the `Index` configuration option. When one or more `Index` elements are provided, AMPS creates memo indexes for any field specified in an `Index` element on startup, prior to executing a query that uses that field. Otherwise, AMPS indexes each field the first time a query uses the field. Adding one or more `Index` configurations to a `SOW/Topic` can improve retrieval performance the first time a query that contains the indexed fields runs for large SOW topics.
    `ExpectedKeyCountHint` For SOW topics that will contain a large number of distinct keys, providing an expected key count allows AMPS to pre-size the data structure that holds the key. This can provide a performance improvement for publishers by avoiding cases where AMPS has to resize the data structure. On startup, AMPS will size the internal data structures to hold the number of keys provided. AMPS does this by presizing the structure to hold a number of keys that is a power of 2 equal to or greater than the hint provided. This hint does not limit the number of keys in the topic. This hint sets the number of keys that the topic can hold without resizing the data structure. There is no default for this value. When no value is provided, AMPS does not pre-size data structures for the SOW.
    Below is an example of a SOW with hash indexes. ```xml showLineNumbers customers /customerId json ./sow/%n.sow /customerName /zipCode /customerType ``` ### Historical Query A SOW topic can keep message state to allow "point in time" historical queries of current values. Notice that this option is not required for message-by-message replay; recording the topic in a transaction log provides full replay. Instead, this option provides the ability to determine what the current value was for a message at a particular point in time (even if that value was set, and remained unchanged, long before the point in time that is being queried). A SOW topic can, optionally, maintain the ability to query current values at a specific point in time. To specify this, include a `History` element in the topic configuration. Described below are the configuration options the `History` element must include. Expand each item for more details.
    `Window` (required if `History` is present) For a historical SOW, the length of time to store history. For example, when the value is `1w`, AMPS will store one week of history for this SOW. Used within the `History` element.
    `Granularity` (required if `History` is present) For a historical SOW, the granularity of the history to store. For many applications, it is not necessary for AMPS to store all of the updates to the SOW. This parameter sets the resolution at which AMPS will save the state of a message. A value of `0s` or equivalent specifies that AMPS will preserve every update within the `Window`. For example, when you set a granularity of `1m`, AMPS will save the state of the message no more frequently than once per minute, even when the state of the message is updated several times a minute. Used within the `History` element.
    Below is an example of a historical SOW configuration that will store 7 days of history from the catalog topic, with the state of the messages being saved every 15 minutes. ```xml showLineNumbers catalog /sku json ./sow/%n.sow 7d 15m ``` ### Message Enrichment AMPS can modify a message’s content as it is published to a SOW topic from an application. See the [State of the World Message Enrichment](/docs/amps-user-guide/enrichment) section for details. Described below are the configuration options that allow you to perform message enrichment. Expand each item for more details.
    `Preprocessing` When present, specifies the message enrichment to be performed before AMPS determines the SOW key for the message. The `Preprocessing` element must contain one or more `Field` elements that specify the enrichment to perform.
    `Enrichment` When present, specifies the message enrichment to be performed after AMPS determines the SOW key for the message. The `Enrichment` element must contain one or more `Field` elements that specify the enrichment to perform.
    Below is an example of a SOW with enrichment. This configuration adds a `/fullName` field that is constructed from the `/firstName` and `/lastName` fields. ```xml showLineNumbers sales-reps /employeeId bflat CONCAT(/firstName, " ", /lastName) AS /fullName ``` ### Multiple Logical Topics in One Physical Topic A SOW topic can be declared as a regular expression topic, where multiple topic names use the same definition and are stored in the same physical file. This can be useful when converting a system from topic-based routing, or any situation where a set of topics use the same message structure, the same method for determining the key, and each topic has a relatively small set of messages. AMPS can store the last values for a set of topics that match the same naming pattern and use the same configuration in a single set of SOW data structure and physical SOW file. See [Storing Multiple Logical Topics in One Physical Topic](/docs/amps-user-guide/sow/pattern\_topics) for details. Described below is the configuration option used to define a set of related SOW topics. Expand the item for more details.
    `Pattern` When present, declares that this topic will record multiple logical topics into one physical data structure and file, and specifies the pattern to use to determine if the topic that a message is published to will be captured in this topic. Physical topics that include multiple logical topics have the benefits and limitations described in the Storing Multiple Logical Topics in One Physical Topic section of the AMPS User Guide. When this element is present, the `Topic` cannot specify `History`. There is no default for this element. Legacy Protocols Note: The legacy header formats do not include support for subscribing to or querying from topics that use this element.
    --- # How Does the SOW Work? Much like tables in a relational database, topics in the AMPS SOW persist the most recent update for each message. AMPS identifies a message by using a unique key for the message. The SOW key for a given message is similar to the primary key in a relational database: each value of the key is a unique message. The first time a message is received with a particular SOW key, AMPS adds the message to the SOW. Subsequent messages with the same SOW key value update the message. There are several ways to create a SOW key for a message: * Most applications specify that AMPS assigns a SOW key based on the content of the message. The fields to use for the key are specified in the SOW topic definition, and consist of one or more XPath expressions. AMPS finds the specified fields in the message and computes a SOW key based on the name of the topic and the values in these fields. 60East recommends this approach unless an application has a specific need for a different approach. * A topic can also be configured to require that a publisher provide a SOW key for each message when publishing the message to AMPS. * AMPS also supports the ability for custom SOW key generation logic to be defined in an AMPS module, which will be invoked to generate the SOW key for each message. While these SOW keys are generated automatically by AMPS, rather than being provided by the publisher, the logic to generate these keys is provided by the module, and the configuration required (if any) is determined by the module. The following diagrams demonstrate how the SOW works, using a SOW topic that is configured to have AMPS determine the SOW key based on the `/orderId` field within the message. As each message comes in, AMPS uses the contents of the `/orderId` field to generate a SOW key for the message. The SOW key is used to identify unique records in the SOW, so AMPS will store a distinct record for each distinct `/orderId` value published to this topic. The calculated SOW key will be returned in the `SowKey` header of messages received from the topic in the SOW. In the previous diagram, two messages are published where neither of the messages have matching keys existing in the `ORDERS` topic. The messages are both inserted as new messages. Some time after these messages are processed, an update comes in for the order with an `orderId` of `2`. This message changes the price from 120 to 95. Since the incoming message has an `orderId` of 2, this matches an existing record and overwrites the existing message for the same SOW key, as seen in the diagram below. AMPS replaces the entire record with the contents of the update. Although the SOW key is derived from the content of the message in many cases, the SOW key is distinct from the content of the message. Each record in a SOW topic has a distinct SOW key, which is stored with the record. The SOW stores the full message in the message type format for performance. There is no re-serialization required to send a message to subscribers. By default, a topic recorded in the SOW is _persistent_. For these topics, AMPS stores the contents of the SOW for that topic in a dedicated, memory-mapped file. This means that the total SOW does not need to fit into memory, and that the contents of the SOW database are maintained across server restarts. You can also define a _transient_ SOW topic, which does not store the contents of the SOW to a persisted file. The SOW file is separate from the transaction log, and you do not need to configure a transaction log to use a SOW. When a transaction log is present that covers the SOW topic, on restart AMPS uses the transaction log to keep the SOW up to date. When the latest transaction in the SOW is more recent than the last transaction in the transaction log (for example, if the transaction log has been deleted), AMPS takes no action. If the transaction log has newer transactions than the SOW, AMPS replays those transactions into the SOW to bring the SOW file up to date. If the SOW file is missing or damaged, AMPS rebuilds the SOW by replaying the transaction log from the beginning of the log. :::tip When a SOW topic is `persistent`, each Topic must be stored in a separate file. Only one instance of AMPS can access a given file; the same copy of the SOW file _cannot_ be used by multiple instances of AMPS. ::: When the SOW for a Topic is _transient_, AMPS does not store the SOW for this topic across restarts. In this case, AMPS will synchronize the SOW with the transaction log when the server starts to restore the state of the topic. By default, this recovery processes the entire transaction log. You can use the `RecoveryPoint` configuration option to specify that the topic should have only new publishes or should recover from a specific point in time (for example, you could use an environment variable to provide a timestamp to the `RecoveryPoint` so that AMPS recovers only the last 24 hours of messages.) --- # Storing Multiple Logical Topics in One Physical Topic AMPS provides a way to easily define a set of related SOW topics by specifying a `Pattern` element in the `Topic` configuration. When this element is present, AMPS creates a container SOW topic that can include a number of SOW topics in one physical file. A publish to a topic name that matches the `Pattern` will be treated as an individual SOW topic within the container topic that defines the pattern. The definition of each individual topic (for example, the `Key` values defined, the hash indexes defined, and so on) is defined by the container `Topic`, and is the same for every individual topic within the container SOW topic. Using this approach creates a single physical topic (that is, the container is a _single_ in-memory SOW topic and, when the topic is persisted, a single file) that contains records for any number of individual topic names. The topics within the container maintain the last value of each individual record within _each_ of the topics. Publishers and subscribers can use these topics as though the topics were each configured individually as a `Topic` in the AMPS configuration file (with some minor behavioral differences resulting from all of the topics and messages being stored in the same data structure, as described in the following sections). Although AMPS treats every topic within the container SOW topic as a distinct topic for the purposes of publishing and subscribing, AMPS manages those topics as records within a single SOW object. When the overall SOW topic is persisted, every message for an individual topic is stored within the same file. Likewise, the overall SOW topic is treated as a single topic in memory (including for monitoring and statistics purposes). In cases where an application has a large number of topics and each topic has a small number of messages (typically, in cases where each topic has only a single message), using a `Pattern` can use considerably less memory than individual `Topic` entries for the same number of topics and messages.
    Legacy Protocol Note Querying or subscribing to topics that use the `Pattern` element is supported for connections using the `amps` protocol. Legacy protocols do not include support for this feature.
    :::tip 60East recommends using a `Pattern` for a topic in situations where an existing system uses topic names rather than content filtering, and it is not practical to adjust the system. For example, when migrating a legacy system that distinguishes orders for different customers using different topic names rather than using the content of the message (such as using topic names `/orders/customerA` and `/orders/customerB` rather than including a `customer` field on the message), creating a SOW topic using a `Pattern` of `^/orders/` might be the most straightforward way to adapt the system to AMPS. For a small number of topics, or cases where an individual topic would have a large number of entries, 60East recommends using individual topics rather than specifying a `Pattern`. ::: ## When to Use a Pattern in a Topic The `Pattern` element allows you to define a large number of SOW topics that will hold a small number of records (typically, only one record per topic) while minimizing the memory and storage overhead for each topic. This can be especially helpful when migrating a system that uses topic-based routing to AMPS, since you can easily create a large number of topics (hundreds or thousands) without having to explicitly specify each one in the AMPS configuration file in cases where it is important to query the last value of each topic. Consider using the `Pattern` element in cases where: * You need to be able to query the current value of a record (or topic). If you do _not_ need to query current values, there is no need to define topics in the SOW at all (consider using ad hoc topics instead). * The information that determines whether a given message is unique is not contained in the message itself. If that information is already present in the data, it is more efficient to use a `Topic` with the unique property configured as a `Key`. * The messages have the same structure and are the same logical type of message. Messages that are different types, or that have different structures, would typically be represented in different topics. * Your application requires a large number of topics, or you do not know the topics in advance, such that it is impractical to define the topics using individual `Topic` declarations. * Each unique topic will have a small number of messages (ideally, only one message per topic). * Your application does not require historical point in time query. * All of the topics to be managed together have the same general set of permissions. AMPS does not support applying different entitlements to individual topics within an overall SOW topic (some limited workarounds are available through content filters). If any of the above considerations are not true, consider using a set of `Topic` declarations rather than using the `Pattern` element in a single `Topic`. Container topics are most commonly used when adapting a system that did not support content filtering (content-based routing) to an AMPS-based application in cases where the message data itself does not contain enough information to support content-based routing. Applications designed for AMPS most frequently use a `Topic` and content filtering rather specifying a `Pattern` for a `Topic` and providing routing information in the topic name. ## Limitations When Storing Multiple Logical Topics in a Physical Topic For most purposes, topics that use a `Pattern` work just like any other topic defined using the `Topic` directive. However, there are some differences in behavior, as outlined below: * When an application issues a `sow` or `sow_and_subscribe` that uses a regular expression for the topic name, messages from topics within a topic that uses `Pattern` are delivered between a _single_ `group_begin` and `group_end` pair. Messages from any topic name within the topic may be delivered in any order within the query results. Each message will indicate which topic within the topic it originated from. * A topic that uses `Pattern` _cannot_ be the underlying topic for a view. * A topic that uses `Pattern` can be the underlying topic for a conflated topic, but the conflated topic must be configured to use such a topic. * All of the topic names within the topic must have the same permissions. ## Configuration File Precedence AMPS allows you to define a standalone topic, view, queue, or conflated topic with a `Name` that matches the `Pattern` of the `Topic`. To do so, however, that definition must appear in the configuration file **before** the definition of the topic that uses the `Pattern`. The topic, view, queue, or conflated topic will be configured as though the topic that defined the `Pattern` is not present. For example, the following `SOW` configuration creates a `Topic` named `/orders/specialHandling` and a `Topic` with a `Pattern` that matches `^/orders/`. The `/orders/specialHandling` topic adds preprocessing, and could also, in principle, have different permissions than the topic names that are matched by the `Pattern`. ```xml showLineNumbers /orders/specialHandling json /orderId COALESCE(/orderId, CONCAT(/customerName, /customerSerialNumber)) as /orderId ./sow/%n.sow RegexOrders ^/orders/ json /orderId ./sow/%n.sow ``` With these definitions, a publish to the following topic names would produce the following results: | Message Published to Topic | Results | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `/orders/specialHandling` |

    Matches `Topic` definition. Stored in the `Topic`.

    The `Preprocessing` directive runs and creates the `/orderId` from the `/customerName` and `/customerSerialNumber` if there is no `/orderId` already present.

    | | `/orders/RHAT` | Matches the regular expression topic definition, stored in the regular expression topic. | | `/orders/specialHandling/oops` |

    Matches the regular expression topic definition, stored in the regular expression topic.

    Notice that a `Topic` definition is an exact match on the topic name, not a pattern match.

    | | `/customer/orders/timothy_someone` |

    Does not match either the `Topic` or the regular expression topic.

    Not included in the SOW.

    | ## Entitlements for Logical Topics AMPS considers permissions for all of the logical topics within the physical SOW topic to be _identical_. When checking permissions for the topic with an entitlement module, AMPS requests that the module provide permissions for the `Name` specified in the topic. Any topic name included in the container will use the permissions, entitlement filter, and entitlement select list specified by the module for that `Name`. If it becomes necessary to restrict access to individual topics within the physical topic, there are two approaches that you can take: 1. Create a new topic with a `Pattern` that specifies the topics that require different permissions, and apply the permissions to that topic. 2. Provide an entitlement filter that uses the `TOPIC_NAME()` function to restrict access to specific topic names; for example, `TOPIC_NAME() IN ('/orders/RHAT', '/orders/MSFT', '/orders/IBM')`. Using this method is less efficient than providing permissions for those topics (either as standalone topics, or for a regular expression topic containing exactly those three topics), but this approach can be a good option in cases where subscribers typically subscribe only to the topics they are entitled to, different subscribers have substantially different sets of entitlements, or there are no logical or convenient groupings that can be used to separate the topics into several regular expression topic declarations. ## Logical Topics and the Transaction Log The transaction log can be used to record publishes to a physical topic that contains multiple logical topics. To do this, the transaction log specification must contain a `Topic` directive that matches the physical topic `Name`. This will capture all of the logical topics in the transaction log. Notice that *only* the physical topic `Name` is considered in this case. ## Conflated Topics and Logical Topics AMPS provides the ability to create a set of conflated topics to provide conflation over the set of logical topics in a single physical topic. To do this, the `Name` of the logical topic is provided as the underlying topic for the conflated topic, and the conflated topic provides a `TopicFormat` template to use for constructing the conflated topic names. See the configuration reference for [Configuring Conflated Topics in a SOW](/docs/amps-user-guide/conflated-topics/configuring-conflated-topics-in-sow) for details. --- # Programmatically Deleting Records from the Topic State AMPS allows applications to explicitly remove records from a SOW topic using the `sow_delete` command. When removing records from a SOW, there are three different ways to indicate which message, or messages, will be deleted: 1. Using a content filter. AMPS will delete all messages in the SOW that match the content filter. To delete every message in the SOW, use the special filter `1=1` to indicate that the filter is true for every message, regardless of the contents of the message. (In essence, AMPS runs a query to locate the records to be deleted, and then deletes the matching records.) 2. Using the SOW key assigned to the message. AMPS accepts a list of SOW keys, and will remove the messages indicated by those SOW keys. 3. Using message data. The application provides message data with the `sow_delete` command. AMPS parses the message data to determine the SOW key for the record that would be updated if the command were a `publish`, and deletes that record (if one exists). Notice that if the topic is configured so that publishers must provide the SOW key, the key cannot be derived from the data, which means that using message data to delete messages may not produce the expected results. When a record is removed from the SOW, AMPS sends an out-of-focus (OOF) message to any subscriptions that have requested OOF notifications. AMPS also updates any views that use the SOW topic, and the record will be removed from conflated topics at the next conflation interval. When the SOW is configured with the `History` option to enable historical queries, the `sow_delete` command removes the message from the current set of messages in the SOW. The command does not remove previously saved versions of the message: the historical state of the SOW is unaffected by the `sow_delete`. The most efficient way to delete a specific message or specific set of messages is to use the SOW key that AMPS assigns, when that key is available. You can provide these keys in the `SowKeys` header (a delete by keys), or by providing a filter expression that will be evaluated as a query on the primary key or a hash index. See the [Indexing SOW Topics](sow\_indexing) section for details on how AMPS determines if a hash index or primary key can be used for a filter. When the SOW delete provides an example message to be deleted, AMPS parses that message to determine the SOW key and then uses that to key to delete the message, which is also relatively efficient. :::tip Deleting a message from the SOW means that AMPS can reuse the space that the message consumed, but AMPS does not reduce the size of the storage for the topic when a message is removed. Typically, SOW topics in production reach a steady state based on the number of messages that are typically present at any given time: it is most efficient to simply make the space available for new messages. To reduce the size of a file used to persist a topic in the SOW after messages are removed, use the [Compact SOW Topic](/docs/amps-user-guide/actions/do-elements/do-sow-compact) action. Running this operation will typically reduce throughput to the topic being compacted during the process of compacting the topic, so this should only be done during a maintenance window or when reducing (or pausing) throughput to the topic would have less impact on the application than leaving the SOW file at its current size. ::: Removing a message from a `Topic` in the State of the World removes the message from that `Topic` and notifies any `View` or `ConflatedTopic` that depends on this topic that the message has been removed (see [Aggregation and Analytics](../views) for details on creating a `View`, see [Conflated Topics ](../conflated-topics)for details on creating a conflated topic). Removing a message from a `Topic` adds the delete command to the transaction log, but does not remove messages stored in the transaction log (see [Record and Replay Messages](../txlog)). If the `Topic` contains `History`, the `sow_delete` affects the current value of the `Topic` but does not remove previous state. AMPS will remove records that have not been current for longer than the retention `Window`, as described in the [Historical SOW Topic Queries](../sow-queries/historical-queries) section. --- # SOW Maintenance Applications that store topics in the SOW must consider the ongoing storage needs and file management for the SOW. There are two aspects to SOW maintenance: 1. Ensuring that the host system has enough capacity to efficiently store and manage the topics in the SOW. Capacity planning guidelines are discussed in the [Capacity Planning](/docs/amps-user-guide/operation/capacity-planning) section in the operations section of this guide. 2. Setting and implementing a data retention policy for the contents of each topic in the SOW. The data retention policy for a topic in the SOW is determined by the needs of your application. Consider the following questions: * Does the topic have a data set that tends to stay at a consistent size? If so, there may be no need to explicitly manage data retention. Many AMPS applications have topics that fall into this category. For example, an application that uses a SOW topic to track the current price of a specific set of ticker symbols has little need to set a data retention policy. The SOW will always contain the same number of records (one for each ticker symbol), and those records will always contain data of a consistent size. The application may choose to remove a record when a symbol is removed from the set, but otherwise rely on publishers to keep the data current. * Is the data only valid for a fixed duration relative to when the data is published? If so, [Setting Per-Message Lifetime](/docs/amps-user-guide/sow/sow-maintenance/expiration) using message expiration may be a good way to manage the SOW. For example, an application that needs to ensure that quotes are removed from the system after 10 minutes from the time the quote is published could use SOW expiration to remove records after 10 minutes. Managing this expiration using SOW expiration may be more efficient than using an action, since messages may expire at any point in time. * Is the data valid until a certain condition becomes true? If so, having the application remove records from the SOW that are no longer needed or configuring a [Scheduled Maintenance](/docs/amps-user-guide/sow/sow-maintenance/scheduled-maintenance) action may be a good way to manage the SOW. For example, an application that needs to clear the state of the SOW every 24 hours during a maintenance window could use an action to remove those records. An application that can determine when a record is no longer needed can remove the record immediately, which means that the topic only contains data that the application needs at any given time. Regardless of the approach an application takes, 60East recommends that every application that uses a SOW consider capacity and explicitly consider the data retention needs of each topic and the application. --- # Setting Per-Message Lifetime By default, a topic in the SOW stores all distinct records until a record is explicitly deleted. For scenarios where message persistence needs to be limited in duration, AMPS provides the ability to set a time limit on the lifespan of SOW topic messages. This limit on duration is known as message expiration and can be thought of as a "Time to Live" feature for messages stored in a SOW topic. ### Using Expiration Expiration on SOW topics is disabled by default. For AMPS to expire messages in a SOW topic, you must explicitly enable expiration on the SOW topic. There are two ways message expiration time can be set. First, a topic recorded in the SOW can specify a default lifespan for all messages stored for that topic. Second, each message can provide an expiration as part of the message header. AMPS stores the expiration time for each message individually, as a property of the message in the SOW. The expiration for a given message is first determined based on the message expiration specified in the message header. If a message has no expiration specified in the header, then the message will inherit the expiration setting for the topic expiration. If there is no message expiration and no topic expiration, then it is implicit that a SOW topic message will not expire. When an expiration of 0 is provided in the message header, this indicates that AMPS should not provide expiration for this message. ## Enabling Expiration for a Topic AMPS configuration supports the ability to specify a default message expiration for all messages in a single SOW topic. Below is an example of a configuration section for a SOW topic definition with an expiration. The [Configuring Topics in a SOW](/docs/amps-user-guide/sow/configuring-topics-in-sow) section has more detail on how to configure the SOW topic. ```xml showLineNumbers ORDERS sow/%n.sow 30s /55 /109 fix ``` In this case, messages with no lifetime specified on the message have a 30 second lifetime in the SOW. When a message arrives and that message has an expiration set, the message expiration on the publish overrides the default expiration for the topic. Each publish or delta publish that arrives, including an update to an existing message, updates the expiration time. AMPS also allows you to enable expiration on a SOW topic, but to only expire messages that have message-level expiration set: ```XML showLineNumbers ORDERS sow/%n.sow enabled /55 /109 fix ``` With this configuration file, expiration is enabled for the topic. The message lifetime is specified on each individual message. When expiration is disabled for a SOW topic, AMPS preserves any message expiration set on an individual message but does not expire messages. AMPS processes expirations during startup when SOW expiration is enabled. This means that any record in the SOW which needs to be expired will be expired as AMPS starts. Notice that if expiration has been disabled in the configuration file, AMPS will not process expiration for the topic. ## Setting Expiration for a Message When expiration is enabled for a topic in the SOW, each message published to that topic expires at the configured time by default. Individual messages have the ability to specify the expiration for that individual message. When an expiration time is provided on a message, that value overrides the default expiration set for the topic. For example, the SOW configuration for a topic might specify an expiration of 5 minutes for a pending order. For large orders, however, a publisher might explicitly prevent messages from expiring by providing a `0` for the expiration time when publishing the message. AMPS does not process expiration for any messages in a topic recorded in the SOW unless expiration is enabled for the topic. When expiration is not configured for a topic, messages published to that topic do not expire, regardless of the expiration setting on an individual message. When a message arrives, AMPS calculates the expiration time for the message and stores a timestamp at which the message expires in the SOW with the message. When the message contains an expiration time, AMPS uses that time to create the timestamp. When the message does not include an expiration time, but the topic contains an expiration time, AMPS uses the topic expiration for the message. Otherwise, there is no expiration set on the message, and AMPS records a timestamp value that indicates no expiration. Messages in the SOW topic can receive updates before expiration. When a message is updated, the message’s expiration lifespan is reset. For example, a message is first published to a SOW topic with an expiration of 45 seconds. The message is updated 15 seconds after publication of the initial message, and the update resets the expiration to a new 45 second lifespan. This process can continue for the entire lifespan of the message, causing a new 45 second lifespan renewal for the message with every update. If a message expires, then the message is deleted from the SOW topic. This event will trigger delete processing to be executed for the message, similar to the process of executing a `sow_delete` command on a message stored in a SOW topic. ## Recovery and Expiration When using message expiration, one common scenario is that the message has an expiration set, but the AMPS instance is shut down during the lifetime of the message. To handle such a scenario, AMPS calculates and stores a timestamp for the expiration, as described above. Therefore, if the AMPS instance is shutdown, upon recovery the engine will check to see which messages have expired since the occurrence of the shutdown. Any expired messages will be removed from the topic as soon as possible. Notice that, because the timestamp is stored with each message, changing the default expiration of a SOW topic does not affect the lifetime of messages already in the SOW. Those timestamps have already been calculated, and AMPS does not recalculate them when the instance is restarted or when the defaults on the SOW topic change. If expiration is not enabled for the topic after the configuration change, AMPS does not process expirations for that topic and messages will not expire. ## Replication and Expiration Because the expiration time is stored as an attribute of each individual message, that expiration time is replicated with the message. A downstream instance that receives the message via replication does not reset or change the expiration time that is stamped on the message. Expiration processing happens on each individual instance. The fact that a message has expired is not replicated (this is not necessary, since the message expiration is stored as a part of the message, so each individual instance can manage expiration locally). --- # Creating a Maintenance Schedule for a Topic For many applications, messages need to be removed at specific times (for example, at the end of a trading day, or when the message reaches a certain state) rather than having a message-specific time to live. These applications typically configure a scheduled maintenance plan using AMPS actions to manage the SOW and remove unneeded information. For full details on AMPS actions, see the [Configuring AMPS for Automation with Actions](../../actions) section in the _User Guide_. ## Sample Maintenance Plan Below is an example of a configuration section for a SOW topic definition, where records will need to be removed when they have reached a state of `completed` and been inactive for more than 24 hours. Since this is intended to manage the size of the saved state of the topic, it isn't necessary for messages to be removed precisely when they reach that state. Removing messages once a day, before activity begins for that day, is enough. ```xml showLineNumbers ORDERS sow/%n.sow /orderId nvfix ``` To create this maintenance plan, we configure an AMPS action that runs at `02:00` local time and removes the messages that the topic no longer needs. ```xml showLineNumbers amps-action-on-schedule 02:00 Maintenance for ORDERS topic amps-action-do-delete-sow ORDERS nvfix /status = 'closed' AND LAST_UPDATED() < ({{AMPS_UNIX_TIMESTAMP}} - 86400) ``` Notice that there are two parts to this action. The `On` element specifies when the action should run -- in this case, every day at `02:00` local time. The `Do` element directs AMPS to delete messages from the `ORDERS` topic (of message type `nvfix`) where the `/status` is `closed`, and where the last update for the message is earlier than 24 hours (86400 seconds) from the time the action was started. At the scheduled time, AMPS internally runs a `sow_delete` command that removes the specified messages. This command is also written to the transaction log and replicated to other instances. With this configuration, AMPS can efficiently maintain the SOW topic based on the needs of the application. --- # Using the State of the World State of the World topics are used for several different purposes: ### Queries / Point in Time Database At any point in time, applications can issue SOW queries to retrieve all of the messages that match a given topic and content filter. When a query is executed, AMPS will test each message in the SOW against the content filter specified and all messages matching the filter will be returned to the client. The topic can be a literal topic name or a regular expression pattern. For more information on issuing queries, please see the section on [Querying the State of the World (SOW)](../sow-queries). ### Atomic Query and Subscribe If the application needs to receive updates or needs the ongoing state of the topic rather than running a one-time query, the application can query the State of the World _**and simultaneously**_ subscribe to updates to the topic. This is typically much more efficient than running repeated queries of the topic. This command, `sow_and_subscribe`, is described in the [Query and Subscribe](../sow-queries/query-and-subscribe) topic in the [Querying the State of the World (SOW)](../sow-queries) section. ### Enable Advanced Messaging Features Because the State of the World maintains a record of the current state of messages, it enables several of the advanced messaging features provided by AMPS. #### Out-of-Focus Messages A subscription to a topic can optionally request to be notified when a message is removed or no longer matches the subscription when the topic is recorded in the State of the World. See the [Out-of-Focus Messages](../oof) section for details. #### Message Enrichment AMPS can optionally enrich messages when they are published to a topic that is recorded in the State of the World. The enrichment can include logic based on the previous state of the message. See the [State of the World Message Enrichment](../enrichment) section for details. #### Publishing Incremental Updates Because a topic stored in the State of the World maintains the current value of a message, applications do not need to republish the full message when making updates to a message. See the [Incremental Message Updates](../delta-publish) section for details. #### Aggregation and Analysis Since a State of the World topic maintains a complete set of current values for a topic, a State of the World topic is the foundation of analysis and aggregation of the messages published to a topic. See the [Aggregation and Analytics](../views) section for more details. #### Receiving Updated Fields Only When a message in the State of the World is replaced or updated, AMPS can determine which fields (if any) have changed from the previous values. A subscriber can optionally request to be delivered only fields that have changed from the previous values. See [Receiving Only Updated Fields](../delta-subscribe) for details. ### Application Scenarios The topics titled [When Should I Store a Topic in the SOW](/docs/intro-guide/sow/why_use_sow) and [Scenario and Feature Reference](/docs/intro-guide/feature_guide/) in the[ Introduction to AMPS](/docs/intro-guide/intro) provide an introduction to some of the application scenarios that can benefit from using a State of the World topic. --- # Indexing SOW Topics AMPS maintains indices over SOW topics, views, and conflated topics to improve query efficiency. There are two types of indices available: 1. _Memo indices_ are created automatically when AMPS needs to use a particular field for a query. These indices maintain the value of a key, and can be used for any type of query, including regular expression queries, range queries, and comparisons such as less than or greater than. You can also request that AMPS pre-create an index of this type with the `Index` directive of the SOW topic configuration. 2. _Hash indices_ are defined by the configuration for the topic, view or conflated topic. These indices maintain a hash derived from the values provided for the fields in the key. When the topic is configured so that AMPS generates the SOW key, AMPS automatically creates a hash index that contains all of the fields in the SOW Key. You can create any number of hash indexes for a SOW topic, with any combination of fields. Hash index queries are significantly faster than queries using memo indexes. Both types of indices are maintained in memory. The section on [Estimating AMPS Instance Memory Usage](../operation/capacity-planning.md#estimating-amps-instance-memory-usage) has more details. A hash index can be created using any XPath Identifier in the message. For example, if you are using a `composite-local` message type, you can create a hash index using fields from any part of the message. If you are using an `xml` message, you can create a hash index that uses the XML attributes. The values of hash indices are always evaluated as strings. Hash indices are only used for exact matches on the value of the fields or with the `IN` operator, and only for queries that use the exact set of fields in the hash index. Other operators or functions (for example, `LIKE`, `!=`, `BETWEEN`, `IS NULL`, `IS NOT NULL`, and so on) cannot use the hash index. To use a hash index, the comparison must use a literal string for comparison to specify that the comparison uses an exact string comparison and not a numeric comparison. For example, if your configuration specifies a hash index that uses the fields `/address/postalCode` and `/customerType`, a query filter such as `/address/postalCode = '04109' AND /customerType = 'retail'` will use the hash index. A query filter such as `/address/postalCode = '04109' AND /customerType LIKE 'retail|remainder'` will not use the hash index, since this filter uses the `LIKE` operator rather than exact matching. Likewise, a comparison such as `/address/postalCode = 04109` will not use the hash index, since the expression requests a numeric comparison rather than a string comparison. Starting with AMPS 5.3.1.0, AMPS will also use a hash index for a compound filter if the _first_ clause in the filter is an `IN` operator that can use a hash index _and_ the other comparisons in the filter are evaluated using the `AND` operator. In this case, AMPS evaluates the `IN` clause first, and executes the rest of the expression against the results of the `IN` clause. For example, a filter like `/id IN ('jon', 'jim', 'joy') AND /price > 50` will use a hash index to find matching records for `/id` and then compare the matching records to the rest of the filter (in this case, a numeric comparison on `/price`). (Notice that this optimization is not available if the other comparisons use the `OR` operator.) AMPS uses a hash index for filters where possible. If the filter does not meet the requirements for using a hash index, AMPS uses memo indices for the fields in the filter if those are available. If one or more of the required memo indices is not available, AMPS creates the indexes during the query. If your application frequently uses queries for an exact match on a specific set of fields (for example, retrieving a set of customers by the `/address/postalCode` field), creating a hash index can significantly improve the speed of those queries. --- # Understanding SOW Keys This section describes AMPS SOW keys in detail, including information on how AMPS generates SOW keys and considerations for applications that generate SOW keys. An individual SOW topic may use either AMPS-generated SOW keys or user-generated SOW keys. Every message in the SOW must use the same type of key generation. Regardless of how the SOW key is generated, AMPS creates an opaque value from the SOW key and uses this value for efficient lookup internally. For SOW keys that AMPS generates, this opaque value is returned in the message header for SOW messages and is used in commands that reference SOW keys. When the SOW key is provided with a message, AMPS returns the original value in the SOW key header, and the original value is used in commands that reference SOW keys. For topics that have a SOW key (including views and conflated topics), commands that directly use the SOW for a topic (for example, `sow`, `sow_and_subscribe`, `sow_delete`) can provide a SOW key, or a set of SOW keys with the command. When a set of SOW keys is provided with one of these commands, the command will only operate on messages that have a SOW key in the provided set. ### AMPS-Generated SOW Keys AMPS-generated SOW keys are often the easiest and most reliable way to define the SOW key for a message. The advantages of this approach are that AMPS handles all of the mechanics of generating the key, the key will always match the data in the message, and there is no need for a publisher to be concerned with how AMPS assigns the key. The publisher simply publishes messages, and AMPS handles all of the details. AMPS generates SOW keys based on the message content when you define one or more `Key` fields in the SOW configuration. For example, if your SOW tracks unique orders that are identified by an `orderId` field in the message, you could provide the following `Key` element in your SOW configuration: ```xml /orderId ``` This configuration item tells AMPS to use that field of the message to generate SOW keys. AMPS supports composite SOW keys when multiple `Key` elements are provided. For example, the following configuration specifies that every unique combination of `/orderId` and `/customerId` is a unique record in the SOW: ```xml /orderId /customerId ``` When AMPS generates a key, it creates the key based on the _key domain_ (which is the name of the topic by default) and the values of the fields specified as SOW keys. AMPS concatenates these values together with a unique separator and then calculates a checksum over the value. This ensures that different values create different keys, and ensures that records in different topics have different keys. In some cases, you may need AMPS to calculate consistent SOW key values for identical messages even when the messages are published to different topics. The SOW topic definition allows you to set an explicit key domain in the configuration, which AMPS will use instead of the topic name when generating SOW keys. For example, if your application uses the `orderId` field of a message as a SOW key in both a `ShippingStatus` topic and an `OpenOrders` topic, having AMPS generate a consistent key for the same `orderId` value may make it easier to correlate messages from those topics in your application. By setting the same `KeyDomain` value in the Topic configuration for those SOW topics, you can ensure that AMPS generates consistent SOW keys for the same order ID across topics. An application should treat SOW keys generated with the AMPS default SOW key generator as opaque tokens. The value of a generated SOW key is guaranteed to be consistent for the same fields, values, and key domain. However, an application should not make assumptions as to the specific value that the AMPS default key generator will produce from a given set of values. If an application requires a specific value for the SOW key, the application should generate a SOW key, as described in the following section. #### Using Enrichment with SOW Keys The preprocessor phase of AMPS enrichment occurs before AMPS generates SOW keys for a message. You can use this phase of enrichment to construct fields that are then used to generate the SOW key for a message. #### Customizing AMPS-Generated SOW Keys AMPS allows you to customize how the server generates SOW keys for a topic. To customize SOW key generation, you implement a SOW key generator module and specify that the module should be used to generate keys for that SOW topic. To use a custom SOW key generator, you first load the module in the `Modules` section of the configuration file, then specify the module as the `KeyGenerator` for the SOW topic. ```xml showLineNumbers ... key-generator libmy_key_generator.so custom-keyed-sow ./sow/%n.sow key-generator module-specific-option another-specific-option ... ``` For information on implementing a custom SOW key generator, contact 60East support for the AMPS Server SDK. ### User-Generated SOW Keys AMPS allows applications to explicitly generate and assign SOW keys. In this case, the publisher calculates the SOW key for the message and includes that key in the message when it is published. AMPS does not interpret the data in the message to decide whether the message is unique: AMPS uses only the value of the SOW key. When using a user-generated SOW key, applications should consider the following: * All publishers should use a consistent method for generating SOW keys. * SOW keys must contain only characters that are valid in Base64 encoding. * The application must ensure that messages intended to be logically different do not receive the same SOW key. User-generated SOW keys are particularly useful for the `binary` message type. For this message type, AMPS does not parse the message, so providing an explicit SOW key allows you to create a SOW that contains only `binary` messages. To specify that AMPS will require publishers to this topic to submit the SOW key, the `Topic` configuration does not specify any `Key` fields and does not specify a `KeyGenerator` for the topic. --- # Batching Query Results When processing a SOW query, AMPS has the ability to combine messages into batches for more efficient network usage. The maximum number of messages in a batch is determined by the `BatchSize` parameter on the SOW query command. AMPS defaults to a `BatchSize` value of 1, meaning AMPS sends one message per batch in the response. The `BatchSize` is the maximum number of records that will be returned within a single response payload. Each AMPS response for the query contains a `BatchSize` value in its header to indicate the number of messages in the batch. This number will be anywhere from 1 to `BatchSize`. :::info The `BatchSize` parameter only applies to the results of a SOW query. In all other cases, AMPS does not delay a message once it is ready to be sent to a subscriber. ::: Current versions of the AMPS client libraries set a batch size of 10 when no other batch size is specified. Notice that the format of messages returned from AMPS may be different depending on the message type requested. However, the information contained in the messages is the same for all message types. :::info When issuing a `sow_and_subscribe` command AMPS will return a `group_begin` and `group_end` segment of messages before beginning the live subscription sequence of the query. This is also true when a `sow_and_subscribe` command is issued against a non-SOW topic. When the topic is not in the State of the World, no messages will be delivered between the `group_begin` and `group_end` messages. ::: Using a `BatchSize` greater than 1 can yield greater performance, particularly when querying a large number of small records. In general, 60East recommends using a `BatchSize` that provides good network utilization without consuming excessive server memory. Most applications that use small messages set a batch size designed to create batches that fit well into the maximum transmission unit (MTU) for the network. AMPS reports an error if an application requests a batch size larger than 10,000 records (this value is orders of magnitude larger than the typical `BatchSize` used by applications). For applications that return a large number of messages that are larger than the MTU, 60East recommends testing performance with a variety of batch sizes. Because the client libraries parse the AMPS headers common to each message once per batch, a batch size larger than `1` can improve processing performance on the client side, particularly if the client message handling is efficient. Likewise, because the AMPS server only has to serialize the common headers once per batch, a batch size larger than `1` can improve performance at the server side (as well as reduce the overall bandwidth for a group of messages). At the same time, the server will hold a batch of messages until the batch can be transmitted together (or until the query is complete), so providing large values for the batch size can introduce latency in receiving results, and reduce performance if the total size of the batch is very large. In general, the default client value is a good compromise for many application patterns if the messages are larger than will fit into the MTU of the network. For smaller messages, or if it is important to tune performance, 60East recommends testing with a variety of batch sizes. :::info Using an appropriate `BatchSize` parameter can help achieve the maximum query performance with a large number of messages when many messages will fit into the MTU for your network. For larger messages, tune the batch size based on performance testing with a variety of batch sizes. ::: For more information on executing queries, please see the Developer Guide for the AMPS client of your choice, available from the 60East [documentation site](/). --- # Historical SOW Topic Queries Topics in the State of the World can also be configured to include historical snapshots of messages, which allows subscribers to retrieve the contents of the topic at a particular point in time. As with simple queries, a client can issue a query by sending AMPS a `sow` command and specifying an AMPS topic. For a historical query, the client also adds a timestamp that includes the point in time for the query in the `Bookmark` header of the command. A filter can be used to further refine the query results based on the message content. ## When to Use a Historical SOW Query Use a historical SOW query when it is important to get a snapshot of the state of messages in a topic as they existed at a specific point in time (that is, if it is important for an application to be able to query the state of the world at a point in time). If an application needs to replay the exact sequence of messages delivered to a topic, but does not need to be able to query the values that were current at a specific point in time, record the topic in the transaction log and replay from the transaction log. If an application needs to _both_ retrieve a snapshot of the values that were current at a specific point in time and replay the exact sequence of messages from that point forward, use a historical SOW query and record the topic in the transaction log. ## Configuring the Topic: Window and Granularity By default, AMPS does not maintain history for a topic in the State of the World. To enable history (and historical query) for the topic, add the `History` element to the `Topic` configuration. This element configures how much information AMPS stores for enabling historical queries. There are two options that control how AMPS stores data for historical queries: 1. The `Window` option sets the amount of time that AMPS will retain historical versions of messages. AMPS will remove the historical state of the message from the SOW topic once that historical state is older than the specified window. (If the message has been deleted, and the delete command is older than the specified window, AMPS may remove the message from the SOW topic entirely). AMPS always retains the most current state of a message, even if that state was published earlier than the specified `Window`. In other words, a given version of a message is eligible for removal after it has no longer been the most current update to that message for _longer than_ the specified `Window`. 2. The `Granularity` option sets the interval at which AMPS retains a historical copy of a message in the SOW. For example, if the `Granularity` is set to `10m`, AMPS stores a historical copy of the message no more frequently than every 10 minutes, regardless of how many times the message is updated in that 10 minute interval. AMPS stores the copies when a new message arrives to update the SOW. This means that AMPS always returns a valid SOW state that reflects a published message, but -- as with a conflated topic -- the SOW may not reflect all of the states that a message passes through. This also means that AMPS uses SOW space efficiently. If no updates have arrived for a message, since the last time a historical message was saved, AMPS has no need to save another copy of the message. When a message is deleted from a topic that maintains history, AMPS saves the fact that the message has been deleted, and queries as of that point in time will not return the message. However, previously saved states of the message within the `Window` are still present and can still be queried. Likewise, if an application queries at a point in time earlier than the `Window`, AMPS will return an empty result set (even if messages had actually been present in the topic at that point), since the SOW state is only retained for the period in the `Window`. :::tip The `Granularity` for a topic is always specified as a duration. If your application requires that a query be able to return the exact state of the SOW exactly as AMPS would have represented it at that time (with no tolerance for the granularity), you can specify that AMPS keep every message during the `Window` by specifying a `Granularity` of `0s`. Notice that this is _not_ required to replay every message after a point-in-time query, since replay is delivered from the transaction log rather than the stored State of the World. ::: When a historical SOW and Subscribe query is entered, and the topic is covered by a transaction log, AMPS returns the state of the SOW adjusted to the next oldest granularity, then replays messages from that point. In other words, AMPS returns the same results as a historical SOW query, then replays the full sequence of messages from that point forward. The transaction log and the SOW topic are maintained separately and have separate views of history. When a version of the message is removed from the SOW topic (because it is older than the specified `Window`), the message remains in the transaction log, but will not be returned by a SOW query. :::info The length of time that messages remain in the topic is specified by the `Window`. A SOW topic that retains history does not support sow expiration. If it is necessary to delete messages after they have been active for a certain period of time in a topic that maintains history, use an explicit delete from an application or use a scheduled action, as described in [Creating a Maintenance Schedule for a Topic](../sow/sow-maintenance/scheduled-maintenance). ::: ## Message Sequence Flow The message sequence flow is the same as a simple SOW query flow. Once AMPS has transmitted the messages that were in the SOW as of the timestamp of the query, the query ends. Notice that the query will include messages that have been subsequently deleted from the SOW, but which were the current state of the message as of that timestamp. ## Pagination with Historical SOW Queries Topics that maintain `History` in the SOW support paginated queries from a point in time. When the topic is also covered by the transaction log, the `sow_and_subscribe` command also supports paginated subscriptions from a point in time. See [Paginated SOW and Subscribe](managing-result-sets.md#paginated-sow-and-subscribe) for details. --- # Managing Result Sets AMPS allows you to control the results returned by a SOW query by including the following options and header on the query: | Option / Header | Result | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `top_n` (option) |

    Limits the results returned to the number of messages specified.

    When a `skip_n` option is also provided for a subscription, AMPS creates a paginated subscription. Otherwise, this option applies only to the SOW query part of a `sow_and_subscribe` or `sow_and_delta_subscribe` command.

    | | `skip_n` (option) |

    Skips the number of messages specified before returning results.

    A command that provides this option must also provide a `top_n` option.

    | | `OrderBy` (command header) |

    Orders the results returned as specified.

    Requires a comma-separated list of identifiers of the form:

    `/field [ASC \| DESC] [ TEXT ]`

    The `ASC` directive specifies that AMPS sort the results in ascending order (the default). `DESC` specifies that AMPS sort the results in descending order.

    The `TEXT` hint specifies that AMPS will sort the column according to the textual representation of the column. This can be helpful in cases where the column represents a string value, but where some values could be interpreted as numeric values.

    For example, to sort in descending order by `orderDate` so that the most recent orders are first, and ascending order by `customerName` for orders with the same date, you might use a specifier such as:

    `/orderDate DESC, /customerName ASC `

    As another example, the following specifier will sort the `orderId` field as a string, with the `updateTimestamp` sorted in descending order for orders with the same `orderId`.

    `/orderId TEXT, /updateTimestamp DESC`

    If no sort order is specified for an identifier, AMPS defaults to ascending order. If no type hint is specified for an identifier, AMPS defaults to using a mixed-type sort.

    | For details on how to submit these options with a SOW query, see the documentation for the AMPS client library your application uses. When replacing a subscription that uses `top_n`, `skip_n`, or `OrderBy`, any of these options specified on the original command must be provided on the replacement command. In other words, `sow_and_subscribe` command that specifies `top_n=10,skip_n=20` must provide both `top_n` and `skip_n` on a replacement command. ## Paginated SOW and Subscribe When `top_n` and `skip_n` are specified on a `sow_and_subscribe` command, AMPS creates a _paginated subscription_. (Both `top_n` and `skip_n` must be provided to create a paginated subscription.) With a paginated subscription, AMPS maintains a list of the set of results for the SOW query, and delivers only results that fall between the first record after the `skip_n` number and within the number of records specified by the `top_n` number. This allows applications that only need a subset of the results returned by a filter to work with only those results. This is commonly used for interactive applications, where a user interface shows a small number of records at a time in the interface. When the subscription specifies an `OrderBy`, that header specifies the order in which records are sorted within the paginated subscription. If no `OrderBy` is specified, the results are sorted by the `SowKey` generated by AMPS (effectively, an arbitrary but stable order). From a subscriber point of view, paginated subscriptions behave as though only the messages in the pagination window are present in AMPS. For example, when out-of-focus notifications are enabled and a message in the topic is deleted, subscribers receive an `oof` notification _only_ if the deleted message was in the pagination window. Likewise, if a message that was previously in the pagination window falls outside of the window due to an insert or delete, the message that is now outside of the window will be out of focus, and will generate an `oof` notification. For example, consider the following topic in the SOW, where the topic uses the `/id` field as a key. With a `top_n` of `2`, a `skip_n` of `1`, and an `OrderBy` of `/id`, the results for the subscription will include the records with `id` of `2` and `id` of `5`. Now a new message is published with an `id` of `4`, as shown below: Since the new message falls within the pagination window, the message is published to the subscriber. Given that the message with the `id` of `5` is no longer within the pagination window, the subscriber will receive an `oof` message for the message with an `id` of `5` if the subscriber has requested out-of-focus notifications. While a paginated subscription is active, AMPS maintains a list of the messages that match the subscription in memory (but does not, as of version 5.3.2, maintain the entire sorted result set in memory). For efficiency, when more than one subscription uses the same topic, these subscriptions will use the same result set in memory. The memory used counts as part of the configured `MessageMemoryLimit`. Each connection that uses the result set is counted as consuming a portion of the memory retained. For example, if 5 connections use the same result set, each of those connections is counted as using 1/5 of the memory for the result set. In addition, each paginated subscription requires that AMPS maintain state for the window for that subscription: this memory is not shared and is counted for that client. ## Aggregated SOW Queries AMPS provides the ability to aggregate the results of a SOW query. The results of an aggregated SOW query are the same as the results of querying a `View` with the same definition. To request an aggregated SOW query, provide the `grouping` and `projection` options with the `sow` query. | Option | Description | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `grouping=[keys]` |

    For use with aggregated SOW queries.

    The format of this option is a comma-delimited list of XPath identifiers within brackets. For example, to aggregate entries based on their `/description` (producing one record in the aggregation for each distinct value in `/description`), you would use the following option:

    ` grouping=[/description] `

    When this option is provided, a `projection` must also be provided.

    When the topic has `History` enabled, this option can be used with a bookmark to aggregate the historical state of the SOW.

    | | `projection=[fields]` |

    For use with aggregated SOW queries.

    Specifies a comma-delimited set of fields to project, within brackets. Each entry has the format described in the AMPS User Guide.

    This option must contain an entry for every field in the aggregated message. If there is no entry for a field in this option, that field will not appear in the aggregated message, even if the field is in the underlying message.

    There is no default for this option. When this option is provided, a `grouping` must also be provided.

    When the topic has `History` enabled, this option can be used with a bookmark to aggregate the historical state of the SOW.

    | --- # Overview of SOW Queries A client can issue a query by sending AMPS a `sow` command and specifying an AMPS topic. Optionally a filter can be used to further refine the query results. AMPS also allows you to restrict the query to a specific set of messages identified by a set of SowKeys. When AMPS receives the `sow` command request, it will validate the filter and start executing the query. When returning a query result back to the client, AMPS will package the `sow` results into a `sow` record group by first sending a `group_begin` message followed by the matching SOW records, if any, and finally indicating that all records have been sent by terminating with a `group_end` message. AMPS returns the results for a SOW query in a single, atomic operation. Any messages for the client that arrive during the SOW query are delivered after the SOW results. :::danger AMPS treats queries as a single, atomic operation. All results from a query are sent to a client before the results of any subsequent commands. Use care when issuing queries that return a result set large enough to take several seconds or more to transmit over the network. When planning for large queries, please see the information on how AMPS handles a situation where messages are produced faster than the client or network can consume them. This is discussed in the section on [Slow Client](../ha/slow-client-management-and-capacity-limits) mitigation. ::: The sequence diagram below illustrates the message flow for a SOW query. For purposes of correlating a query request to its result, each query command can specify a `QueryId`. The `QueryId` specified will be returned as part of the response that is delivered back to the client. The `group_begin` and `group_end` messages will have the `QueryId` attribute set to the value provided by the client. The client specified `QueryId` is what the client can use to correlate query commands and responses coming from the AMPS engine. AMPS does not allow a `sow` command on topics that do not have a SOW enabled. If a client queries a topic that does not have a SOW enabled, AMPS returns an error. :::tip The ordering of records returned by a SOW query is undefined by default. You can include an `OrderBy` parameter on the query to specify a particular ordering based on the contents of the messages. ::: --- # Query and Subscribe AMPS has a special command that will execute a query and place a subscription at the same time to prevent a gap between the query and subscription where messages can be lost. Without a command like this, it is difficult to reproduce the SOW state locally on a client without creating complex code to reconcile incoming messages and state. For example, this command is useful for recreating part of the SOW in a local cache and keeping it up to date. Without a special command to place the query and subscription at the same moment, a client is left with two options: 1. Issue the query request, process the query results, and then place the subscription, which misses any records published between the time when the query and subscription were placed; _or_ 2. Place the subscription and then issue the query request, which could send messages placed between the subscription and query twice. Instead of requiring every program to work around these options, the AMPS `sow_and_subscribe` command allows clients to place a query and get the streaming updates to matching messages in a single command. In a `sow_and_subscribe` command, AMPS behaves as if the SOW command and subscription are placed at the exact same moment. The SOW query will be sent before any messages from the subscription are sent to the client. Additionally, any new publishes that come into AMPS that match the `sow_and_subscribe` filtering criteria and come in after the query started will be sent after the query finishes (and the query will not include those messages.) As with a simple SOW query, any other messages that arrive for the client while the SOW query is running will also be delivered after the query results. AMPS allows a `sow_and_subscribe` command on topics that do not have a SOW enabled. In this case, AMPS simply returns no messages between `group_begin` and `group_end`. The sequence diagram below illustrates the message flow for `sow_and_subscribe` commands: ## Historical SOW Query and Subscribe For topics that have `History` configured, AMPS SOW Query and Subscribe also allows you to begin the subscription with a historical SOW query. For historical SOW queries, the subscription begins at the point of the query with the results of the SOW query. The subscription then replays messages from the transaction log. Once messages from the transaction log have been replayed, the subscription then provides messages as AMPS publishes them. In effect, a SOW Query and Subscribe with a historical query allows you to recreate the client state and processing as though the client had issued a SOW Query and Subscribe at the point in time of the historical query. A historical SOW and Subscribe requires that the SOW topic is recorded in the transaction log and that history is enabled on the SOW. If history is not enabled for the topic, a `sow_and_subscribe` command returns the current state of the SOW and the subscription begins atomically at the point in time when AMPS processes the command. ## Conflated Subscriptions with SOW and Subscribe A `sow_and_subscribe` command can include the conflation interval and conflation key options for server side conflation (as described in [Conflated Subscriptions](../pub-sub/conflation)), just as a regular subscription can. When the command requests conflation, the results of the SOW query are not conflated. Conflation only applies to the subscription. ## Replacing Subscriptions with SOW and Subscribe As described in [Replacing Subscriptions](../pub-sub/replace), AMPS allows you to replace an existing subscription. When the subscription is entered with the `sow_and_subscribe` command, AMPS will re-run the SOW query delivering the messages that are in scope with the new filter but which were not previously delivered. If the subscription requests out-of-focus (OOF) messages, AMPS will deliver out of focus messages for messages that matched the previous filter but do not match the new filter. As with the initial Query and Subscribe, AMPS guarantees to deliver any changes to the SOW that match the filter and occur after the point of the query. --- # Querying the State of the World (SOW) When SOW topics are configured inside an AMPS instance, clients can issue SOW queries to AMPS to retrieve all of the messages matching a given topic and content filter. When a query is executed, AMPS will test each message in the SOW against the content filter specified and all messages matching the filter will be returned to the client. The topic can be a straight topic or a regular expression pattern. --- # State of the World (SOW) One of the core features of AMPS is the ability to persist the most recent update for each distinct message published to a given topic. The State of the World (SOW) can be thought of as a database where messages published to AMPS are filtered into topics, and where the topics store the latest update to each distinct message. The SOW gives subscribers the ability to quickly resolve any differences between their data and updated data in the SOW by querying the current state of a topic or any set of messages inside a topic. Topics recorded in the SOW are also used for caching data, providing "point in time" snapshots of active data flows, providing key/value stores over data flows, and so on. Topics recorded in the SOW are the underlying sources for AMPS aggregation and analytics capabilities, and the ability to store the previous state of a message is the foundation of advanced messaging features such as delta messaging and out of focus notifications. AMPS also provides the ability to keep historical snapshots of the contents of the SOW, which allows subscribers to query the contents of the SOW at a particular point in time and replay changes from that point in time. AMPS can maintain the SOW for a topic in a persistent file, which will be available across restarts of the AMPS server. The SOW can also be *transient*, in which case the state of the SOW does not persist across server restarts. Topics do not keep the current values in the SOW by default. To provide this capability for a topic, you must configure AMPS to maintain the topic in the SOW by adding a definition for the `Topic` to the `SOW` section of the AMPS configuration file. --- # Client Connections To accept connections from publishers or subscribers, an AMPS instance must have at least one `Transport` configured for client connections. The `Transport` must specify: * The network protocol used for the transport, called the transport _type_. * The network address, such as IP address and port, that the AMPS server will listen to for incoming connections. * Optionally, the AMPS command header format, called the _protocol_. The `amps` protocol is the default if no protocol is provided and is the protocol used by most applications. Websocket connections use the `websocket` protocol. A transport can _optionally_ set other parameters on the transport. This includes setting the authentication and entitlements that apply to connections for this transport, setting slow client parameters for the transport, and so forth. ## TCP Connections The transport type `tcp` specifies communication between applications and AMPS uses a standard TCP/IP connection. AMPS supports optional compression to and from applications that use TCP/IP. This compression is enabled per connection when an incoming connection is compressed. The AMPS Client Libraries enable compression on a connection when the `compression` type is specified as part of the connection options. See the Developer Guide for the client library for details. :::info If the Transport configuration includes TLS/SSL options such as `PrivateKey` and `Certificate`, AMPS will require TLS/SSL for incoming clients even if the transport type is `tcp`. ::: ## TLS/SSL Connections AMPS supports TLS/SSL connections between applications and AMPS. To enable SSL on a transport, you must: * Specify a transport `Type` of `tcps`, _and_ * Provide a certificate and private key for the connection You can optionally set other parameters for SSL connections, as described in the [Configuring Transports](configuring-transports) section. :::info 60East recommends using the `tcps` transport `Type` for SSL connections for clarity. However, AMPS uses SSL connections for a `tcp` connection whenever a `PrivateKey` and `Certificate` are provided for a `Transport`, regardless of whether the transport `Type` is specified as `tcp` or `tcps`. AMPS clients require that the connection string use `tcps` for SSL connections, even if the AMPS `Transport` configuration uses `tcp`. ::: AMPS supports optional compression to and from client applications that use TLS/SSL. This compression is enabled per connection when an incoming connection is compressed. The AMPS Client Libraries enable compression on a connection when the `compression` type is specified as part of the connection options. See the Developer Guide for the client library for details. ## WebSocket Protocol Connections To allow AMPS to be used directly from within browser-based applications, the AMPS server supports communication over WebSockets. An AMPS client `Transport` that uses `tcp` or `tcps` as the underlying `Type` can be configured to use the websocket `Protocol` for communication. The AMPS JavaScript client library supports `websocket` protocol connections to AMPS (while the other client libraries support `amps` protocol connections). When a `Transport` is configured to use the `websocket` protocol, a client that connects to that transport sends an HTTP upgrade header indicating that further communication should use the WebSocket protocol. The AMPS server returns a response indicating that it is switching protocols to WebSocket, and further communication between the client and server uses the proprietary AMPS protocol exchanged within WebSocket frames. The contents of the frames are AMPS messages, as described in the [AMPS Command Reference](../../amps-command-reference/). There is no difference in capability between connections that use the `websocket` protocol and connections that use the AMPS protocol. The commands exchanged between the AMPS server and client are the same, although the "on wire" format within which the commands are exchanged is different. A WebSocket connection to AMPS is a persistent, full-duplex connection to the server. A WebSocket connection is not a RESTful interface to AMPS, nor does the connection support HTTP beyond the initial upgrade request. ## Using IPv6 For Connections Starting with version 5.3.3, AMPS supports connections over both IPv4 and IPv6 protocols if the host has IPv6 support enabled. Both IPv4 and IPv6 address formats are fully supported for use with specifying the network address of a transport. The IP protocol version used for a transport is determined by the network address format specified for the transport. If only a port is specified and the host supports IPv6, AMPS will listen for incoming connections over both IPv4 and IPv6 protocols. For outgoing connections, such as Replication Connections, AMPS fully supports both IPv4 and IPv6 address formats. AMPS will prefer to resolve a URI to IPv4 addresses by default. The DNS resolution behavior can be specified as part of the configuration of the Replication Destination. Specific parameters and behavior related to IPv4 vs IPv6 connections are described in the [Configuring Transports](configuring-transports) section. ## Unix Domain Sockets AMPS provides transports that use unix domain sockets for applications that run on the same system as the AMPS server and require extremely low-latency messaging. Unix domain sockets are not supported by all AMPS clients, since some programming environments do not support these sockets. With this transport type, many of the configuration settings that apply to TCP/IP sockets are not relevant. Instead, the transport requires the name of a file on the local filesystem as the location at which to create the socket. --- # Configuring Protocols In AMPS, `Protocols` define the format of the commands that clients use to communicate with the server. AMPS offers a range of preconfigured protocols. Described below are the recommended protocols for application connections. Expand each item for more details.
    `amps` Standard protocol for AMPS clients.
    `websocket` Standard protocol for websocket connections using the AMPS Javascript library.
    The `Protocols` element configures options for a given protocol. The element is a container for one or more `Protocol` elements. Each `Protocol` is a combination of a `Module`, that defines the basic protocol, and a set of options to configure the behavior of that basic protocol. Many installations of AMPS have no need to configure a `Protocols` block. When this is required, it is most common to need to specify the options used by the `websocket` protocol to match the needs of the operating environment. See the _Websocket Options_ section below for more details. Described below are the configuration items for defining a `Protocol`. Expand each item for more details.
    `Name` The name to use for the customized `Protocol`. This name is used in the `Transport` element to refer to the customized `Protocol`. 60East recommends that the `Name` of the customized `Protocol` contain the `Name` of the `Module`. For example, a customized protocol that adjusts the behavior of the `websocket` protocol might be named `websocket-custom`. There is no default for this value.
    `Module` This element defines the protocol module to customize. The `Module` must be the name of a protocol module loaded into AMPS. By default, AMPS loads the following protocols:
    • `amps` - Standard AMPS messaging, using compact headers in JSON-based format. AMPS accepts `json` as a synonym for `amps` in a protocol declaration.
    • `fix-session` - FIX session protocol, for use with systems that publish FIX messages using this format.
    • `websocket` - Websocket protocol, using JSON format headers.
    AMPS provides the following protocols for backward compatibility. These protocols are supported for existing usage, but will not be enhanced with future protocol changes and do not support all features of the above protocols.
    • `fix` - Standard AMPS messaging, using headers in FIX format.
    • `nvfix` - Standard AMPS messaging, using headers in NVFIX format.
    • `soap` - Standard AMPS messaging, using headers in SOAP format.
    • `xml` - Standard AMPS messaging, using headers in XML format.
    ### Websocket Options The `websocket` protocol accepts the following additional configuration options. Expand each item for more details.
    `WWWAuthenticate` The type of authentication used for this protocol. This specifies how the connection will receive credentials from the connection. The `websocket` protocol accepts two options for this element:
    • Negotiate - Use the "negotiate" scheme for HTTP authentication.
    • Basic - Use basic authentication. When basic authentication is specified, the `realm` must also be set. The syntax for setting the `realm` value is `realm=`.
    For example, to use basic authentication and set the `realm` to a value of `HTTP Special`, you would use the following `WWWAuthenticate` element: ```xml showLineNumbers Basic realm="HTTP Special" ```
    `TrustedAdmin` Specify whether connections that use this protocol should accept connections from clients that have successfully authenticated to the administrative interface. Default: `false`
    `HTTPHeader` Specify that the server will return the custom header specified in response to a websocket request. A protocol of `websocket` type can have any number of these elements configured, and AMPS will return all of the headers defined. There is no default for this element.
    The following configuration snippet shows one way to define a customized `websocket` protocol: ```xml showLineNumbers ... websocket-portal websocket Basic realm="AMPS Admin" enabled ... ``` --- # Configuring Transports The `Transports` element configures how AMPS communicates with publishers and subscribers, as well as how AMPS accepts connections for replication. The `Transports` element is a container for one or more `Transport` elements. Each `Transport` is a combination of a network transport, an AMPS header protocol, and a message type. A `Transport` also specifies the `Authentication` used to validate the users that connect, and the `Entitlement` used to enforce permissions for users that connect over that transport. AMPS supports a variety of network transports, header protocols and message formats for communication between publishers and subscribers. For more information on Transports, see [Transports](/docs/amps-user-guide/transports) in this section of the guide. Described below are the configuration items available for a `Transport`. Expand each item for more details.
    `Name` (required) The name to use for this `Transport`. This name appears in the AMPS log for messages related to the transport. When the `Type` of the `Transport` is `amps-replication` or `amps-replication-secure`, 60East recommends that the `Name` of the `Transport` match the value of the `Type`. There is no default for this value.
    `Protocol` (required) This element defines the protocol to use for sending and receiving messages. The protocol is typically `amps`, the name of a specific protocol for interoperability with another system or a legacy application, or the name of a custom protocol module specified in the `Modules` element. AMPS provides support for the following protocols:
    • `amps`: Standard AMPS messaging, using compact headers in a JSON-based format.
    • `websocket`: Websocket protocol, using JSON format headers.
    AMPS accepts `json` as a synonym for `amps` in a protocol declaration. AMPS also loads the following legacy protocols. These protocols are supported for backward compatibility. They will not be enhanced with new functionality, and do not provide all of the features of the above protocols.
    • `fix`: Standard AMPS messaging, using headers in FIX format.
    • `fix-session`: FIX session protocol, for use with systems that publish FIX.
    • `nvfix`: Standard AMPS messaging, using headers in NVFIX format.
    • `soap`: Standard AMPS messaging, using headers in SOAP format.
    • `xml`: Standard AMPS messaging, using headers in XML format.
    60East recommends using the `amps` protocol for general purpose AMPS messaging. When your application uses Javacript and web sockets, use the `websocket` protocol. Older versions of AMPS used message headers in the same format as the message type: if your instance supports applications that expect to use a specific message type protocol, use that protocol in your `Transport` configuration.
    `Type` (required) The type of `Transport`. Valid values include:
  • `tcp` - The standard TCP transport.
  • `tcps` - Secure TCP transport: this transport type uses SSL and requires a certificate and private key to be set.
  • `amps-replication` - For inbound replication connections. Notice that AMPS replication does not use the same transport type as other applications.
  • `amps-replication-secure` - For inbound replication connections that use SSL. Notice that AMPS replication does not use the same transport type as other applications. This transport type requires a certificate and private key to be set.
  • `amps-unix` - For incoming client connections over Unix domain sockets. This transport type requires a `FileName`, which is the location on the file system where the Unix domain socket will be created.
    • For details on using replication transports, see the [Configuring Incoming Replication Transports](/docs/amps-user-guide/replication/config-incoming-replication) section.
    `InetAddr` The port on which AMPS will listen for this transport. This element can also specify an IP address, in which case AMPS listens only on that address. If no IP address is specified, AMPS listens on all available addresses. Starting with version 5.3.3, both IPv4 and IPv6 address formats are fully supported for use with specifying the network address of a transport. If no address is specified and the host supports IPv6, AMPS will listen for incoming connections on both IPv4 and IPv6 protocols. If you wish to limit AMPS to listen for addresses of only a specific IP protocol you may specify the `ANY` address for that protocol. For example: `0.0.0.0:9007` will cause AMPS to listen on port `9007` for only IPv4 addresses. `[::]:9007` will cause AMPS to listen on port `9007` for only IPv6 addresses. This element is not required for transports of the `amps-unix` `Type` but is required for all other `Type` values.
    `MessageType` Restricts a transport to a single message type. When provided, AMPS assumes that all connections to this transport use the specified `MessageType`. If a different `MessageType` is provided in the connection string, AMPS refuses the connection. If no `MessageType` is provided in the connection string, AMPS accepts the connection and assumes that the connection uses the specified `MessageType`. When the `Transport` `Type` is `amps-replication` or `amps-replication-secure`, this element is ignored and the `Transport` accepts all message types configured in the instance. When present, this is a reference to the name of a specific message type defined in the `MessageTypes` section or one of the message types that AMPS loads by default. In this release, AMPS loads the following message types by default: `fix`, `nvfix`, `xml`, `json`, `msgpack`, `bson`, `bflat` and `binary`. Composite message types, message types based on C structs, and message types based on Google Protocol Buffers, must be defined in the `MessageTypes` element before they can be used. A `Transport` that uses the `amps` `Protocol` defaults to accepting all message types defined by the instance, and does not require setting a `MessageType` element. When the `Transport` does not specify a `MessageType`, the connecting client must declare the message type it will use when logging on. A `Transport` that uses the `amps` protocol can specify a single `MessageType` to accept by including this element. When a single `MessageType` is specified, AMPS does not require that the message type is specified by the client. Important: A `Transport` that uses one of the legacy `Protocol` values (`fix`, `nvfix`, or `xml`) must specify a `MessageType.` Default: When the `Protocol` is `amps`, defaults to accepting all message types defined by the instance. When the `Protocol` is one of the legacy values, there is no default and a `MessageType` must be provided.
    `InitialState` Defines whether, when AMPS starts, the transport is enabled or disabled. When the transport is disabled, AMPS does not listen for or accept connections on the transport. When `InitialState` is `disabled`, the transport must be explicitly enabled after startup (for example, through an `Action` or the administrative console) for AMPS to listen for and accept connections on the transport. This configuration option can be useful for defining a `Transport` that is only available when certain conditions are true: for example, an instance might start with the connection used by clients disabled and let an external monitoring system enable the connection during business hours and disable the connection outside of business hours. Default: `enabled`
    `Entitlement` Specifies the entitlement module to use for this transport. If no entitlement module is provided, the transport uses the default entitlement module for the instance. This element must contain a `Module` element with the `Name` of an entitlement module. If the module requires options, those options are provided in an `Options` element within the `Entitlement` element. Default: The module specified in the `Entitlement` element for the instance (defaults to `amps-default-entitlement-module` if not provided).
    `Authentication` Specifies the authentication module to use for this transport. If no authentication module is provided, the transport uses the authentication module for the instance. This element must contain a `Module` element with the `Name` of an authentication module. If the module requires options, those options are provided in an `Options` element within the `Authentication` element. Default: The module specified in the `Authentication` element for the instance (defaults to `amps-default-authentication-module` if not provided).
    `MessageMemoryLimit` The total amount of memory to allocate to messages before offlining clients for this transport. If this value is specified for the transport, AMPS will allow the specified amount of memory for connections to this transport, independent of the limits set for any other transports or the instance as a whole. This option is specified in bytes, and accepts the standard AMPS notation (for example, `10GB` or `250MB`). Default: The setting configured at the instance level. If this option is not specifically set at the instance level, the instance defaults to 10% of total host memory or 10% of the amount of host memory AMPS is allowed to consume (as reported by `ulimit -m` ), whichever is lowest.
    `MessageDiskLimit` The total amount of disk space to allocate to messages before disconnecting clients. If this value is specified for the transport, AMPS will allow the specified amount of memory for connections to this transport, independent of the limits set for any other transports or the instance as a whole. This option is specified in bytes, and accepts the standard AMPS notation (for example, `10GB` or `250MB`). Default: The setting configured at the instance level. If this option is not specifically set at the instance level, the instance defaults to `1GB` or the amount specified in the `MessageMemoryLimit`, whichever is highest.
    `MessageDiskPath` The path to use to write offline files. Default: `/var/tmp`, or the setting configured at the instance level.
    `TransportFilter` A transport filter to use for the transport. When specified, each command received over this transport is provided to the filter. This element requires a `Module` element, which contains the name of the module that provides the filter. This element may contain an `Options` element, which contains the set of options to provide to the module. The options required by a specific module depend on the module: see the documentation for the module for details. A transport can specify multiple filters. When multiple filters are specified, AMPS provides the command to each specified filter, in the order in which the filters appear in the transaction log. There is no default for this element. If no `TransportFilter` is specified, AMPS processes commands exactly as they are received.
    `ClientMessageAgeLimit` The maximum amount of time for the client to lag behind. If a message for the client has been held longer than this time, the client will be disconnected. This parameter is an AMPS time interval (for example, `30s` for 30 seconds, or `1h` for 1 hour). Default: No age limit, or the setting configured at the instance level.
    `ClientMaxCapacity` The amount of available capacity a single client can consume. Before a client is offlined, this limit applies to the `MessageMemoryLimit`. After a client is offlined, this limit includes the `MessageDiskLimit`. This parameter is a percentage of the total. Default: `50%` (previous versions defaulted to`100%`).
    ## TLS/SSL Transports Starting with 5.1, AMPS supports encrypting client connections using the SSL (Secure Sockets Layer) network protocol. AMPS performs additional configuration validation when the transport is configured with a `Type` of `tcps`. However, if a `Certificate` and `PrivateKey` are specified for a `Transport` of type `tcp`, AMPS will use SSL for that `Transport`. These options also apply to transports of type `amps-replication-secure`. Described below are the configuration items available for setting up a `Transport` to use SSL. Expand each item for more details.
    `Certificate` (required) The certificate file to use for the server. This element is required for TLS/SSL connections. Default: There is no default for this option.
    `PrivateKey` (required) The private key to use for the server. This element is required for TLS/SSL connections. Default: There is no default for this option.
    `Ciphers` The cipher list to use for this transport. The cipher list is passed to the OpenSSL implementation without being interpreted by the AMPS server. For OpenSSL, details on the format of the cipher list are available at: [https://www.openssl.org/docs/man1.1.1/man1/ciphers.html](https://www.openssl.org/docs/man1.1.1/man1/ciphers.html) Default: There is no default for this option.
    `SecureSocketProtocols` The SSL/TLS protocols accepted by this transport. This parameter accepts a space delimited list of values from the following list: `SSLv2`, `SSLv3`, `TLSv1`, `TLSv1.1`, `TLSv1.2`, `TLSv1.3`. Default: `TLSv1.1 TLSv1.2 TLSv1.3` NOTE: 60East recommends using the default protocols unless there is a specific reason to enable earlier versions of the protocol and the security implications of the earlier protocols are well understood.
    `VerifyClient` Specifies whether the client is required to provide a certificate to be verified by the server. When this parameter set to `True`, at least one of `CAFile` or `CAPath` must be specified. Default: `False`
    `CAFile` Specifies a `.pem` file containing trusted certificates to be used to verify client certificates. There is no default for this parameter.
    `CAPath` Specifies a path to a directory containing `.pem` files containing trusted certificates to be used to verify client certificates. When this parameter is provided, and `VerifyClient` is set to `True`, AMPS will use every `.pem` file in that directory for verification. There is no default for this parameter.
    ## Unix Domain Socket Transports For protocols of `Type` `amps-unix`, AMPS supports the following additional configuration options. Expand each item for more details.
    `FileName` (required) Specifies the location on the filesystem where the Unix-domain socket will be created. This location is the path that will be provided to clients that need to connect using this transport. This element is required for transports of type `amps-unix`. There is no default for this parameter.
    `FileMask` Specifies the file mask to use when creating the Unix-domain socket. The value of the mask is an octal number (by convention, four digits) in the same format as the standard chmod command, and AMPS applies this mask exactly as the chmod command would. The file is created with the user and group that the AMPS server process runs under.
    • `0444` File is readable by owner, group, and any user.
    • `0440` File is readable by owner and members of the owner's group.
    • `0400` File is readable by owner only.
    • `0664` File is readable and writable by owner and members of the owner's group. File is readable by any user.
    • `0644` File is readable and writable by owner. File is readable by members of the owner's group and any user.
    ## Websocket Transports For a `Protocol` of `websocket`, AMPS supports the following additional configuration options. Expand each item for more details.
    `PerMessageDeflate` Controls per-message deflation for websocket connections. When this value is `disabled`, AMPS does not perform per-message deflation. If this option is not present, or is set to any other value, AMPS performs per-message deflation. Per-message deflation is normally negotiated between an application and AMPS during the opening handshake of a websocket connection. 60East recommends leaving this option enabled unless you have a specific reason for disabling it. Default: `enabled`
    `HTTPHeader` Specifies a header to be returned during the websocket handshake. This element can be specified any number of times. Every specified header will be returned. There is no default for this parameter.
    ## Example Transport Configurations This section shows examples of `Transport` configuration. ### Transports with Slow Client Management In this example, an individual transport sets a different slow client management policy than the overall default policy. ```xml showLineNumbers showLineNumbers ... 10GB /mnt/fastio/AMPS/offline 30s ... regular-tcp tcp 9007 amps regular-websocket tcp 9008 websocket highpri-tcp tcp 9995 35GB 70GB amps ``` ### Transports with Transport Filter This example includes a transport filter that translates the names of legacy topics to current topics for commands received on this transport. Transport filters are described in the [Transport Filters](transport-filters) section of this guide. ```xml showLineNumbers translate-legacy-topics tcp 9017 amps amps-topic-translator orders_for_northamerica:NAOrders catalog_items:Catalog customer_.*:Customers ``` ### TLS/SSL Transport This example configures TLS/SSL for a transport. The transport explicitly sets the type to `tcps` to indicate an encrypted connection. The transport also provides the `Certificate` and `PrivateKey` file to use for encryption. This configuration also includes an optional directive to AMPS on which ciphers to enable. ```xml showLineNumbers ssl-all-message-types tcps 9007 amps ${AMPS_INSTALL}/cert.pem ${AMPS_INSTALL}/key.pem HIGH:!aNULL ``` --- # HTTP Preflight ## Streamlining AMPS Connections: Enabling TCP Clients Over HTTP Proxy with HTTP Preflight The HTTP preflight feature allows TCP clients (Python, Java, etc.) to initiate a connection through an HTTP proxy in a way similar to WebSocket clients. This enables a proxied connection for tcp/amps transport, just as AMPS already supports for tcp/websocket transport and the Admin interface. By using HTTP preflight, TCP clients can connect to AMPS via an HTTP proxy, optionally including custom HTTP headers for proxy authentication or routing. **Key Considerations:** _AMPS Still Requires Separate Ports_: This feature does not allow a single transport in AMPS to handle both WebSocket and raw TCP connections on the same port. AMPS still requires distinct ports for tcp/amps, tcp/websocket, and Admin. _Proxy Compatibility_: The goal is to allow tcp/amps clients to connect through an HTTP proxy, making it easier to route traffic through infrastructure that already supports WebSockets. For guidance on configuring a proxy for use with AMPS, refer to the [NGINX Proxy Configuration](/docs/amps-integrations/nginx-proxy/intro) or [Apache Proxy Configuration](/docs/amps-integrations/apache-proxy/intro) guides. For additional information on accessing AMPS through a proxy, refer to the [Using AMPS with a Proxy](/docs/amps-user-guide/operation/proxy) section. ## Usage There are no server-side changes required to use it. Simply add ?http_preflight=true in the connection string in order to enable HTTP Upgrade handshake on the client: ```java showLineNumbers Client client = new Client("example"); client.connect("tcps://localhost:443/amps/json?&http_preflight=true"); ``` ## HTTP Headers A client can optionally set additional HTTP headers. This allows it to provide header data required by an HTTP proxy, for example, an authentication token. Here's an example of how to do it in the Java client: ```java showLineNumbers Client client = new Client("example"); try { client.addHttpPreflightHeader("Cookie: a=1; b=2"); // raw form client.addHttpPreflightHeader("Token", "token_value"); // key-value form client.connect("tcps://localhost:443/amps/json?&http_preflight=true"); } ``` Other clients have a similar API available -- please refer to each client's API documentation for more details. **When to Use:** Use HTTP Preflight when you need to limit the number of exposed ports while supporting both WebSocket and TCP/TCPS connections through a reverse proxy like Nginx. **Why Use It:** - _Reduces Open Ports_: Allows TCP/TCPS clients to connect via a single exposed port instead of requiring separate ports for different transport types. - _Proxy-Friendly Routing_: Enables proxies like Nginx to handle WebSocket-like traffic and route TCP connections efficiently. - _Simplifies Network Management_: Minimizes firewall and security rule complexity by consolidating traffic through a single entry point. - _Avoids Additional Configuration in AMPS_: No changes are needed on the AMPS server; only the proxy (Nginx) requires setup. - This approach is ideal for environments with strict port usage policies or those looking to simplify their network infrastructure while maintaining flexible client connectivity. --- # Replication Connections To receive replicated messages from other AMPS instances, an AMPS instance must have a transport configured as `Type` `amps-replication`or `amps-replication-secure`. A replication transport cannot be used to communicate with applications (and vice-versa). Replication connections accept any message type, can service multiple upstream AMPS instances and are configured as part of an overall High Availability plan. See [Replicating Messages Between Instances](../replication), [Highly Available AMPS Installations](../ha) and the [Configuring Replication](../replication/configuring-replication) section of this guide. Replication connections are typically authenticated as described in the section on [Securing AMPS](../securing) and can also use TLS/SSL, including mTLS, as detailed in [Protecting Data in Transit Using TLS/SSL](../securing/tls). --- # Transport Filters AMPS provides the ability for incoming commands to be modified, or _filtered_. When one or more transport filters are specified, AMPS provides each incoming command to those filters as soon as the header for the message is parsed. Each filter can modify the message data or a subset of the headers, and can choose to have AMPS stop processing the command (or can request that AMPS disconnect the connection that submitted the command). The filters for a `Transport`, if any, are defined in the configuration for the transport. When more than one filter is specified, AMPS runs each filter in the order in which they appear in the configuration file. Transport filters are implemented as extension modules. To create an extension module, contact AMPS support for the server SDK. AMPS loads the following transport filters by default. Expand each item for more details.
    `amps-topic-translator` Translates topic names on incoming commands. This module requires one or more of the following options: `Topic` Specifies the translation to use. This option takes the following format: original `:` translated The original parameter can be a literal topic name or a PCRE regex. Any topic on any command that matches that parameter will be converted to the translated topic. For example, to convert the topic `legacy` to the topic `new`, you would specify the following option: ```xml legacy:new ``` To translate any topic beginning with `/orders/northamerica` to `NAorders`, you would specify the following option: ```xml ^/orders/northamerica:NAorders ```
    `amps-conflated-topic-translator` Translates an incoming `subscribe`, `sow_and_subscribe`, `delta_subscribe`, or `sow_and_delta_subscribe` command for a specific topic name as follows:
    • Translates the topic name on the command to a different topic name.
    • Adds a conflation interval to the command, if there is no conflation interval specified on the incoming command.
    This module can be useful for removing a conflated topic that is infrequently-used, or for which subscribers only monitor a small number of messages out of the overall topic. This module requires one or more of the following options: `Topic` Specifies the translation to use. This option takes the following format: original `:` translated `:` interval The interval specifies the conflation interval to apply to the translated commands if one is not provided. For example, to convert all subscriptions to the topic `orders-C` to the topic `orders`, and guarantee that each translated subscription has a conflation specified, with a 500 millisecond default for conflation, you would use the following options: ```xml orders-C:orders:500ms ``` To convert all subscriptions to the topics `slowUpdates` and `verySlowUpdates` to the topic `updates` and guarantee that each translated subscription has a conflation specified, with a 2 second default for conflation, you would use the following options: ```xml slowUpdates:updates:2s verySlowUpdates:updates:2s ```
    The following transport filter is included in the AMPS distribution, but is not loaded by default. Expand the item for more details.
    Correlation ID Timestamper Writes an ISO 8601 format timestamp to the correlation ID for `publish` and `delta_publish` messages received by AMPS. This module can be useful for gaining a better understanding of AMPS message latency by facilitating the latency measurement of a publish message from the point that AMPS processes the message to the point that it is received by a consumer. It’s important to note that the latency measurement is not end to end latency as it doesn’t include the latency between the publisher and AMPS (since the publisher does not timestamp the message). To load the module in AMPS, add the following configuration item to the `Modules` block of the AMPS configuration file. Then, to use the module, add it to the `TransportFilter` section of the `Transports` block, as shown below. ```xml transport-filter-correlation-id-timestamper libamps_transport_filter_correlation_id_timestamper.so primary tcp ... transport-filter-correlation-id-timestamper True ``` This module supports a single option - `Override`. If specified and set to `True`, the module will overwrite the correlation ID value if it is set. Otherwise, the value will not be overwritten. Note: For certain features such as bookmark subscriptions, SOW topics and views, you may see unexpected results. For example, in the case of a bookmark subscription, messages could be replayed that were received by AMPS hours previous to the bookmark subscription and therefore not result in a meaningful latency measurement if compared with the current time. This is not caused by any delay in AMPS, but rather because the received messages are from historical data (this scenario would be similar for a SOW query). In the case of views, the messages are generated internally by AMPS. Given this, transport filters do not apply and therefore timestamps are not added to the correlation ID field. In the case of replication, replicated messages will not be restamped and therefore the timestamp in the correlation ID header field is not overwritten when a message is replicated. Additionally, when calculating latency, it is critical that the clocks on the hosts being utilized are in sync. For example, if AMPS is running on host A and timestamping publishes, then messages are received on host B and the latency calculation is performed, if the clocks on host A and B are not in sync the latency measurement could be invalid and possibly even be a negative value.
    --- # Transports In order to send and receive messages, an AMPS server must allow incoming connections. *Transports* configure incoming connections to AMPS and are defined in the `Transports` element of the AMPS configuration file. AMPS provides two distinct kinds of incoming connections: - *Client Connections* - for use by the AMPS clients to support external applications - *Replication Connections* - to receive replicated messages from other AMPS instances Each transport controls how authentication and entitlements are enforced for that transport. The transport can either accept the defaults for the instance as a whole, or choose settings unique to that transport. AMPS also provides the ability to filter incoming commands using the Transport Filters capability. --- # Diagnostic Utilities The AMPS distribution provides the following utilities for analyzing and displaying data from the files used by the AMPS server. | Utility | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `amps-grep` |

    Utility for searching AMPS error and event logs or AMPS journal files.

    For error and event logs, the utility is aware of the structure of AMPS messages, and searches message by message rather than line by line.

    For journal files, the utility is able to interpret the journal file format and is aware of the structure of AMPS bookmarks and is also able to correlate client names to the hash values in journal files, making it much easier to locate specific messages or the activity of a specific publisher.

    | | `amps_journal_dump` |

    Utility for extracting information from the journal files that comprise the AMPS transaction log.

    This utility fully supports compressed journal files.

    | | `amps_journal_search` |

    Utility for searching for a specific message within the journal files that comprise the AMPS transaction log.

    This utility is optimized for locating a specific message by exact bookmark.

    For other searches, use `amps-grep`.

    | | `amps_sow_dump` | Utility for extracting information from the file that holds SOW topic data. | | `amps_sqlite3` | Utility for running SQL queries over the SQLite3 database that holds AMPS statistics. | | `amps_file` | Utility for providing file type and version information for AMPS files. | | `amps_clients_ack_dump` | Utility for displaying the contents of the file that contains information on the last message persisted for a given publisher. | | `amps_queues_ack_dump` | Utility for displaying the contents of the file that contains information on the last acknowledged message for a given queue. | | `ampserr` | Utility that provides information about AMPS errors and events. | --- # Finding Information in the Log The AMPS log is one of the most useful places to find information when there's a problem with your application. Here are some techniques to use for finding relevant information in the log: * Ensure the log is capturing information that will be useful for diagnosing the problem. In general production use, 60East recommends logging at `info` level (or more verbose). To fully troubleshoot an error, it may be necessary to log at `trace` level to see the exact behavior in AMPS. Some deployments may find it useful to keep a separate log at `warning` level and above to more easily detect errors and potential problems, while maintaining an `info` level log for being able to troubleshoot problems. * To find log messages that may indicate a problem, use `amps-grep` or the Linux `grep` tool to find log messages at warning, error, critical, or emergency levels. For example, you might use the following command line: ```bash amps-grep -E 'warning|error|critical|emergency' log_file ``` or ```bash grep -E 'warning|error|critical|emergency' log_file ``` This will show lines from the log that contain messages logged at those levels. The text that AMPS uses for log messages is guaranteed not to include strings that duplicate one of the log levels, although information that you configure (such as client names, topic names, and so on) may contain those strings. * If you know the name of the client that experienced the problem, you can use that name to get information about the client. It's often helpful to get log messages that include the client name and several lines of output after the client name to help you understand the context in which AMPS produced the message for the client name. To do this, you might you use the following command line: ```bash amps-grep client_name log_file ``` or ```bash grep –B2 –A10 client_name log_file ``` This command line looks for all occurrences of the _client\_name_ in the log file. The `amps-grep` version prints the full messages where the _client\_name_ appears. The `grep` version attempts to approximate this by printing two lines of context before the line that contains the client name, and ten lines of context after the line that contains the client name. Once you've found the information you're looking for, the `ampserr` utility can help you look up more information on messages, as described in [Looking up Errors with ampserr](../logging/error-discovery-ampserr). --- # Planning for Troubleshooting There are several steps you can take to make troubleshooting easier before encountering a problem. 60East recommends that you consider taking the following actions for a production instance of AMPS: 1. Configure the instance to log messages of `info` level or more verbose, if possible, or a minimum of `warning` level. Some problems require more information, so increasing the amount of logging typically makes troubleshooting easier if your instance has storage available. If space is extremely restricted, `warning` level will provide some information, although more logging may be required to completely troubleshoot a problem. 60East recommends that production instances log at `info` level or more verbose. 2. Ensure that client applications use unique names. Wherever possible, ensure that those names can easily be traced back to the instance of the application. For example, you might use the name of the application combined with the name of the logged on user as a unique name. This will help you to more quickly find log messages related to a problem. 3. Enable the administrative server. The administrative console is a good way to get a snapshot of the current state of a running instance, and the Galvanometer provides graphing and historical analysis capabilities. 4. If you are using replication, ensure that your AMPS instances have unique names. Where possible, use names that make it easy to relate replication messages to the servers that process the message. For example, you might relate the AMPS instance name to the purpose that the instance serves, the physical server that the instance runs on, or both. 5. Learn what normal operation looks like for your application. If possible, take the time to inspect the AMPS logs and the output of the administrator console when everything is working as expected. Applications vary in how they use AMPS, and what is normal for your application might indicate a problem in a different application. For example, if your application normally has a few publishers and many subscribers, seeing dozens of publishers come online may indicate that an application has unexpectedly started more publishers. Likewise, if no publishers are online, that may indicate an issue with connectivity to the AMPS server. Understanding normal behavior will help you to more easily and accurately spot problems. --- # Reading Replication Log Messages For replication connections, the replication source creates a client name that it uses to connect to the downstream instance. This client name contains the source, destination, sync setting, and protocol for the connection. The client name uses the following format: ```bash source!destination!sync_setting!protocol ``` Notice, however, that this is a _client name_. The client name is the name used for the connection, but it does not indicate the direction of any particular message. As an example, consider a client name of: ```bash OrderServer!HotBackup!sync!amps-replication ``` This client name is used for a connection that the AMPS instance named _OrderServer_ has made to the AMPS instance named _HotBackup_. The connection uses the amps-replication protocol, and was configured for synchronous replication at the time the client connected. In this case, a message like the following: ```bash 12-1002 client[OrderServer!HotBackup!sync!amps-replication] replication ack received: publish ack [txid=35922] ``` means that a publish acknowledgment was received on the connection that _OrderServer_ made to _HotBackup_. --- # Troubleshooting Regular Expression Subscriptions When a regular expression subscription does not receive the messages that it expects, the two most common reasons are: - The regular expression does not match the topic, or topics, expected; *or* - The user submitting the subscription is not entitled to one or more of the expected topics As with any other subscription, it would also be important to verify that the messages match any content or entitlement filters in place, and that messages are actively being published to the topic in question. ## Troubleshooting Whether Topics Match To determine whether the regular expression matches the topic in question, use an external regex tester to verify that the regular expression matches the topic names that the subscription intends to subscribe to. Keep in mind that AMPS regular expression matches are not limited to the beginning of the topic name unless the expression contains an explicit anchor (that is, `^`). ## Troubleshooting Entitlement Issues There are two ways to troubleshoot entitlement issues: 1. An entitlement module will typically log whether it allows or denies an entitlement request. Check the AMPS logs for logging indicating whether the entitlement request was allowed or denied. Since AMPS caches the results of entitlement requests, using `amps-grep` to quickly scan the logs for the name of the topic is recommended. 2. With a connection logged on with the same credentials as the user, enter a subscription with the **exact** name of the topic in question. That is, rather than using a regular expression subscription, supply the literal topic name on the subscription request. In this case, AMPS will refuse the subscription and log an error if the user is not entitled. In either case, if the user is not entitled to a topic that matches a regular expression subscription, no messages from that topic will be delivered to the user. --- # Troubleshooting Disconnected Clients One common symptom of problems in an AMPS application is that AMPS disconnects clients unexpectedly. AMPS disconnects clients in the following situations: * When transaction logging is configured for the instance and a client with a duplicate name logs on * When heartbeating is enabled and the client misses a heartbeat * When a slow client falls behind by more than the configured threshold * When the entitlement cache for an instance is reset * When the administration console disconnects a client * When the transport is disabled * When an error that would cause incorrect data to be returned to the client is detected This section presents techniques to help you identify why clients are disconnected and correct any problems that may exist. ## Locating the Reason for Disconnection To discover the reason that a client was disconnected, use the following command to find the client name in the logs: ``` amps-grep client_name log_file ``` The results of this can provide information as to why the client was disconnected. AMPS logs a reason for the disconnection if the disconnection was the result of an internal action by AMPS. If the client chose to disconnect, or the disconnection is the result of network instability, the disconnection is logged but no further information is available to AMPS. The reasons that AMPS will log for a client disconnection are shown below: | Reason | Explanation | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `connection closed` | AMPS has detected that the socket has been closed. | | `entitlement reset` | Entitlements have been reset for this user (or the full transport). | | `name in use` | A client session (as identified by the client name) has reconnected, and the redundant connection has been closed. | | `heartbeat` | The connection failed to provide a heartbeat within the period of time set by the connection, so AMPS has closed the connection. | | `slow client` | The connection was closed due to the client capacity settings being exceeded. | | `unknown command` | The client sent an unknown or malformed command, so AMPS closed the connection. | | `auth` | The connection failed to authenticate. | | `entitlement` | The connection successfully authenticated, but the user does not have permission to log in. | | `internal error` | AMPS has detected an error that would cause incorrect data to be returned to the client and disconnected the affected client to prevent further processing. | These reasons are included in the `07-0013` log message that records that the connection has closed. ## Duplicate Client Name Disconnection When a client is disconnected due to another client with the same name logging on, the messages produced might look like: ```bash 2019-11-20T16:26:59.6408410-08:00 [5] warning: 02-0025 A client logon with an 'in use' client name for the same user id forced a disconnect of client: client[my-name] with user id: ``` To resolve this issue, ensure that clients use unique names when connecting to instances that configure a transaction log. For more details, see the discussion of message persistence and the client naming requirements when a transaction log is configured at [Client Names and the Transaction Log](../txlog/understanding-message-persistence.md#client-names-and-the-transaction-log). ## Missed Heartbeat Disconnection When AMPS disconnects a client due to the client failing to heartbeat, the log messages produced look like the following: ```bash 2019-11-20T16:35:23.9185690-08:00 [6] error: 07-0042 AMPS heartbeat manager is disconnecting an unresponsive client: no-heartbeat-client ``` This error most often arises from severe network congestion, a deadlock or similar problem in the application that is preventing the AMPS client library from producing heartbeats, or a problem in AMPS that prevents AMPS from servicing heartbeat requests. ## Slow Client Disconnection The following shows sample log entries for slow client disconnection. If a client named `sleepy-client` was disconnected for being a slow client, the relevant entries in the transaction log might look like: ```bash 2019-11-20T15:33:06.8496430-08:00 [7] warning: 70-0011 client[sleepy-client] slow consumption detected, offline messages. 2019-11-20T15:33:06.8498130-08:00 [7] error: 70-0004 client[sleepy-client] is not consuming messages, disconnecting slow client ``` Notice that there may be a considerable period of time between the client being offlined and the client being disconnected. There are several approaches to solving the problem: * _Reduce the number of messages returned_. Clients most often fall behind when a SOW query or a replay from the transaction log returns a large number of messages. If possible, use content filtering to return a more precise set of messages. * _Improve the rate at which the client handles messages_. If the client message handler takes a relatively long time to process the message, moving message processing onto a different thread or streamlining the processing may improve the speed of the client and allow the client to keep up. * _Adjust the client offlining threshold_. You can also increase the capacity of messages that AMPS will buffer for clients connected to a specific transport, as described in [Slow Client Management](../ha/slow-client-management-and-capacity-limits). ## Admin Console Client Disconnection Disconnection from the admin console provides no additional information, and produces a log message like the following: ```bash 2019-11-20T15:33:06.8502350-08:00 [4] info: 07-0013 client[sleepy-client] disconnected. ``` ## Admin Console Transport Disabled A transport being disabled through the admin console produces messages like the following: ```bash 2019-11-20T16:04:00.9548130-08:00 [10] info: 07-0047 Transport[json-tcp] being disabled. 2019-11-20T16:04:00.9550150-08:00 [4] info: 07-0013 client[amps-json-tcp-18] disconnected. ``` --- # Troubleshooting AMPS The topics in this section discuss best practices for troubleshooting AMPS. --- # Troubleshooting AMPS This chapter presents common techniques for troubleshooting AMPS. Additional troubleshooting information and answers to common questions about AMPS are included on our support site at [https://crankuptheamps.com/support](/support). --- # Configuring a Transaction Log The AMPS transaction log supports durable subscriptions, reliable publish, and historical replay, while also serving as the foundation for high availability features in AMPS. To enable message recording and replay, configure a `TransactionLog` to keep a journal of messages published to an AMPS instance. For further details on using the transaction log, refer to other parts of this section - [Record and Replay Messages](/docs/amps-user-guide/txlog). The sections on [Replicating Messages Between Instances](/docs/amps-user-guide/replication) and [Highly Available AMPS Installations](/docs/amps-user-guide/ha) cover how AMPS uses the `TransactionLog` (and other features) in high availability. Described below are the configuration items for defining a `TransactionLog`. Expand each item for more details.
    `JournalDirectory` (required) Filesystem location where journal files will be stored. This is the directory where AMPS will create new journal files as messages are recorded to the transaction log. A journal directory should be dedicated to a single instance of AMPS.
    `JournalArchiveDirectory` Filesystem location where journal files are archived. The archive directory is intended to allow older files to be stored on a higher-capacity (but potentially slower) device. Journal files in this directory are part of the transaction log. AMPS can replicate from these files, provide bookmark replay from these files, distribute queue messages from these files, and so on. Specifying that the `JournalArchiveDirectory` and the `JournalDirectory` are on the same storage device is not recommended. A journal archive directory should be dedicated to a single instance of AMPS. Use AMPS actions to move files from the `JournalDirectory` to the `JournalArchiveDirectory`.
    `PreallocatedJournalFiles` The number of journal files AMPS will create as part of the server startup. Default: `2` Minimum: `1`
    `JournalSize` Sets the target size for AMPS to use when calculating the size of journal files. AMPS allocates journal files based on the size of an internal buffer. This option sets the target size for the journal file: AMPS will use the smallest file size that is an even multiple of the internal buffer without going under the specified `JournalSize`. Notice that AMPS does not grow journal files once they are allocated. When a journal file is full, AMPS uses the next journal file. AMPS accepts `MinJournalSize` as a synonym for `JournalSize`. Default: `1GB` Minimum: `10M`
    `Topic` The topic to include in the transaction log. When no `Topic` is specified, AMPS initializes transaction log management for the instance, but does not persist messages. If a `Topic` is specified, all messages that exactly match the specified topic or regular expression will be included in the transaction log. If you want all topics of a specific message type to be persisted, use the regular expression `.*` for the name of the topic. Multiple `Topic` elements can be included in a `TransactionLog` element. See the following section for the elements included within a `Topic` element. To capture logical topics stored in a physical SOW topic (that is, to capture all of the topics within a State of the World `Topic` that uses a `Pattern` element), the `Topic` directive should match the `Name` of the physical topic. (The names of the logical topics within the physical topic do not matter in this case.) There is no default for this element. If no `Topic` elements are configured, the transaction log will not record any messages.
    `FlushInterval` AMPS batches writes to the transaction log to optimize for maximum sustained throughput. If a batch is not full within a certain period of time, AMPS will write the partially-filled batch to the transaction log so that the messages can be replicated, delivered to subscribers, and so on. The interval at which messages will be flushed to the journal file during periods of slow activity is the `FlushInterval`. Setting this explicitly has the potential to reduce latency during periods of low traffic, at the risk of somewhat lower performance during periods of higher traffic. Reducing this interval below the default may produce larger journal files during periods of low traffic, since AMPS may write more partial batches to the transaction log. 60East recommends leaving this option at the default setting unless the application is intended to optimize for low data rates and testing at production volumes on the storage that will be used for production demonstrates a performance advantage from reducing the interval. Default: `100ms` Maximum: `1000ms` Minimum: `1us`
    `O_DIRECT` Where supported, `O_DIRECT` will perform DMA directly from/to physical memory to a user space buffer. Having this enabled can improve AMPS performance, however not all devices support `O_DIRECT`. When `O_DIRECT` is disabled, data loss can occur because the operating system can acknowledge that data has been written to the device before the actual write has happened. 60East does not recommend disabling `O_DIRECT` unless the device that holds the transaction log does not support `O_DIRECT`. Default: `enabled`
    `InactiveClientAckExpiration` Sets the amount of time to retain records for an inactive publisher. On recovery, AMPS will remove the `clients.ack` entry for any publisher that has not published a message for longer than the interval set in this option. This option can help reduce `clients.ack` growth for installations that have large numbers of short-lived publishers. If this option is specified, the interval should be longer than the expected time that a given publisher would be inactive and longer than the time that messages from that publisher would be retained. For example, if an application has publishers that are active at the end of every week and the installation retains journals for 14 days, the interval should be longer than 14 days. Default: Retain records indefinitely.
    `CompressedJournalCacheMemoryLimit` Sets the maximum amount of server memory to use for caching compressed journal files when AMPS needs to read data from a compressed journal file. This setting applies to any situation where AMPS needs to read message data from a compressed journal file, including bookmark replay, replication, or reading messages for queue delivery. Increasing this setting can improve performance when replaying messages from compressed journals, in exchange for temporarily consuming more memory for those replays. An uncompressed journal file will consume cache equivalent to the uncompressed size of the journal. Default: 10% of server memory or 10GB, whichever is lower.
    If `Topic` is included in the `TransactionLog` configuration, it must contain the following elements. Expand each item for more details.
    `Name` (required) The name of the topic to record. This element can be a literal name or a regular expression. This element is required and there is no default.
    `MessageType` (required) The message type of the topic. This must be one of the message types loaded by default or a message type declared in the configuration file. This element is required and there is no default.
    The following example demonstrates a transaction log where the journal file will be written to `./amps/journal`. When AMPS starts, a single journal file will be pre-allocated as noted by the `PreallocatedJournalFiles` setting; and when the first journal file is completely full, a new journal file will be created. This journal will contain those messages which match the topic `orders` and have a message type of `nvfix` and also, any messages published to a topic that contains the string `LOGGED_` of a message type of `json`. The journal will contain messages published to the virtual topics within the physical SOW topic named `bucket`, even though the topics for those messages do not contain the word `bucket`. ```xml showLineNumbers ./amps/journal/ /mnt/somedev0/amps/journal 1 10MB orders nvfix LOGGED_.* json bucket json ... other configuration here ... bucket json .*-cached-values /msgId ./sow/%n.sow ``` --- # Managing Journal Files The design of the journal files for the transaction log are such that AMPS can archive, compress and remove these files while AMPS is running. AMPS actions provide integrated administration for journal files, as described in [Configuring AMPS for Automation with Actions](/docs/amps-user-guide/actions). Archiving a file copies the file to an archival directory, typically located on higher-capacity but higher-latency storage. Compressing a file compresses the file in place. Archived and compressed journal files are still accessible to clients for replay and for AMPS to use in rebuilding any SOW files that are damaged or removed. When defining a policy for archiving, compressing or removing files, keep in mind the amount of time for which clients will need to replay data. Once journal files have been deleted, the messages in those files are no longer available for clients to replay or for AMPS to use in recreating a SOW file. If journal files are removed, and a SOW file is retained, this means that the SOW may have data that is not in the transaction log. :::danger While AMPS is running, the `amps-action-do-remove-journal` action is the only way to safely remove a journal file. This action correctly updates the internal AMPS data structures that refer to the journal file. Likewise, the `amps-action-do-archive-journal` action is the only way to safely move a journal file to the archive directory while AMPS is running, and the `amps-action-do-compress-journal` action is the only way to safely compress journals while AMPS is running. ::: To determine how best to manage your journal files, consider your application's access pattern to the recorded messages. Most applications have a period of time (often a day or a week) where historical data is in heavy use, and a period of time (often a week, or a month) where data is infrequently used. One common strategy is to create the journal files on high-throughput storage. The files are archived to slower, higher-capacity storage after a short period of time, compressed, and then removed after a longer period of time. This strategy preserves space on high-throughput storage, while still allowing the journals to be used. For example, if your applications frequently replay data for the last day, occasionally replay data older than the last week, and never request data older than one month, a management strategy that meets these needs would be to archive files after one day, compress them after a week, and remove them after one month. Archival, compression, and removal should be done using AMPS actions. :::danger If you remove journal files when AMPS is shut down, keep in mind that the removal of journal files must be sequential and _cannot_ leave gaps in the remaining files. For example, say there are three journal files, `001`, `002` and `003`. If only `002` is removed, then the next AMPS restart could potentially overwrite the journal file `003`, causing an unrecoverable problem. ::: When using AMPS actions to manage journal files, AMPS ensures that all replays from a journal file are complete, all queue messages in that journal file have been delivered (and acknowledged, if required), and all messages from a journal file have been successfully replicated before removing the file. ## Reference to File Types AMPS creates the following types of files as part of creating and managing the transaction log. Notice that this includes both files that contain messages (_journal files_) and a set of files created by AMPS to improve efficiency when the instance is restarting and recovering the state of the transaction log. The files for a specific instance are prefixed with the instance `Name`. An AMPS instance will only create files that are prefixed with the `Name` of the instance, and on startup will only recover files that are prefixed with the `Name` of the instance. | Extension | File Type | Description | | -------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `.journal` | Journal file |

    These files contain the messages that comprise the transaction log.

    AMPS always writes new messages to an uncompressed journal file.

    | | `.journal.gz` | Compressed journal file |

    These files contain messages that comprise the transaction log.

    These files have been compressed by AMPS as a result of the `amps-action-do-compress-journal` action.

    Other than being compressed, they are treated identically to uncompressed journal files.

    | | `.index.gz` | Journal index file |

    These files are used during recovery to help AMPS quickly rebuild its references to the content of the transaction log without having to completely reprocess each file.

    Each index file contains index information for the corresponding journal file.

    These files do not contain messages.

    | | `.topic.index` | Topic index file |

    These files are used during replay to help AMPS quickly locate messages for a given topic.

    Creating a topic index is optional, and is enabled through instance level `Tuning` configuration.

    These files do not contain messages.

    | | `.clients.ack` | Clients acknowledgment cache |

    Used during recovery to help AMPS quickly identify the last message persisted from each publisher without having to reprocess each journal file.

    These files do not contain messages.

    When present, AMPS will use this file to determine the last message received from each publisher rather than scanning the journals to determine the last message received.

    The publishers tracked in this file include replication sources.

    | | `.queues.ack` | Queue acknowledgment cache |

    For each queue, this file stores the point in the transaction log for which that queue has been completely processed (that is, all messages prior to that point in the transaction log have been acknowledged or expired).

    On recovery, AMPS can begin restoring the state of the queue from that point rather than reprocessing the entire transaction log.

    These files do not contain messages.

    When present, AMPS will use this file to determine the last point in the journal for a given queue where all queue messages were fully consumed (acknowledged or expired). Queue recovery will begin at that point for each queue. If this file is not present, AMPS will scan journals from the beginning to recover the queue.

    | | `.queue.cache` | Queue metadata cache |

    For a queue that specifies that metadata be cached in a file, this is the file that contains the cache.

    If this file is removed, the queue state will be restored from the transaction log (using the recovery point stored in the `queues.ack` file).

    These files do not contain messages or message headers. They contain delivery state for messages in the queue and the location of those messages in the transaction log.

    | --- # Replaying Messages with Bookmark Subscription One of the most useful and powerful features in AMPS is _bookmark subscription_, which is enabled by the transaction log. With bookmark subscription, an application requests a subscription that starts at a specific point in the transaction log. AMPS begins the subscription at the specified point, and provides messages from the transaction log. Each message in the transaction log has a _bookmark_. A _bookmark_ is an opaque, unique identifier that is added by AMPS to each message recorded in the transaction log. For messages provided from a transaction log, the field is included in the `Bookmark` header of the message. AMPS guarantees that bookmarks for the instance are monotonically increasing, which enables AMPS to rapidly find an individual bookmark within the transaction log. A bookmark subscription simply requests that AMPS begin the subscription with the first message following the bookmark provided with the subscription. AMPS locates the bookmark in the transaction log, and begins the subscription at that point in time. One way to think about a bookmark subscription is that AMPS publishes to the subscribing client only those messages that: 1. Have bookmarks after the provided bookmark 2. Match the subscription's `Topic` and `Filter` 3. Have been written to the transaction log AMPS provides these messages in the order in which they were recorded to the transaction log. Since a bookmark subscription requires a transaction log, when a client requests a bookmark subscription for a topic that is not being recorded in the transaction log, AMPS returns an error. If the subscription requests a `completed` acknowledgment, that acknowledgment will be delivered to the subscription once replay has completed. Messages delivered to the subscription after the acknowledgment is delivered are from new publishes. By default, those messages are delivered once they are written to the transaction log. AMPS allows an application to submit a comma-delimited list of bookmark values as the bookmark for a subscription request. In this case, AMPS begins replay at the oldest bookmark in the list. The client controls the bookmark provided on the subscription request. For a bookmark subscription, the AMPS server does not keep a persistent record of which bookmarks a specific client or subscription has processed. The AMPS client libraries provide facilities for easily tracking the messages which an application has processed so the application can resume at the appropriate point in the transaction log. Requesting replay from the transaction log is how AMPS applications manage _resumable subscriptions_. The application keeps track of which messages have been processed, and requests replay from the appropriate point in the transaction log to resume the message stream. This record-keeping is built into the AMPS client libraries, and most often handled transparently when a _bookmark store_ is set for the client. Notice that this means that the AMPS server itself does not track the progress of individual subscriptions, nor does the application need to inform the server of how far the subscription has progressed. The application state needed to resume the subscription is entirely handled on the application side, with no involvement by the AMPS server. (For details on how specific client libraries manage the application state, see the _Developer Guide_ for that client library.) :::tip Bookmark subscriptions are provided from the transaction log rather than the live publish stream. This lets AMPS adapt the pace of replay to the pace at which the subscriber is consuming replayed messages without triggering slow client offlining. ::: :::info While there are similarities between a bookmark subscription used for replay and a State of the World (SOW) query, the transaction log and SOW are independent features that can be used separately. The SOW gives a snapshot of the current view of the latest data, while the journal is capable of playback of previous messages. Historical SOW queries provide a snapshot of the SOW at a defined point in the past, and are provided by the SOW database rather than the transaction log. ::: There are different ways that a client can request a bookmark replay from the transaction log. Each of these bookmark types meets a different need and enables a different recovery strategy that an application can use. The sections below describe the recovery types, the cases in which they can be used, and how the 60East clients implement them. ## Replay of Full Transaction Log The epoch bookmark, when requested on a subscription, will replay the transaction log starting at the very beginning. Once the transaction log has been replayed in its entirety, then the subscriber will begin receiving messages on the live incoming stream of messages. A subscriber does this by requesting a `0` in the `bookmark` header field of their subscription. The AMPS clients provide a constant for epoch, typically represented as `EPOCH`. This type of bookmark can be used in a case where the subscriber has begun after the start of an event, and needs to catch up on all of the messages that have been published to the topic. To ensure that no messages from the subscription are lost during the replay, AMPS replays messages from the transaction log until the client reaches the last message in the transaction log. Once all of the existing messages in the transaction log have been sent to the client, AMPS will cut over to the live subscription stream and provide messages to the client as soon as they are persisted. ## Bookmark Replay from NOW The NOW bookmark, when requested on a subscription, declines to replay any messages from the transaction log, and instead begins streaming messages from the live stream - returning any messages that would be published to the transaction log that match the subscription's `Topic` and `Filter`. This type of bookmark is used when a client is concerned with messages that will be published to the transaction log, but is unconcerned with replaying the historical messages in the transaction log. This strategy is often used for applications that want to ensure that they do not miss messages, even if the application temporarily loses connectivity, but are not concerned with older messages. For this case, the application subscribes with NOW when the application starts, and then re-establishes the subscription with the most recently-processed bookmark if connectivity is lost. This resubscription behavior is typically handled by the client reconnection logic (as in the 60East `HAClient` implementations). The NOW bookmark is performed using a subscribe query with "0|1|" as the `bookmark` field. The AMPS clients provide a constant for this value, typically represented as `NOW`. ## Bookmark Replay with a Bookmark Clients that store the bookmarks from published messages can use those bookmarks to recover from an interruption in service. By placing a subscribe query with the last bookmark recorded, a client will get a replay of all messages persisted to the transaction log after that bookmark. Once the replay has completed, the subscription will then cut over to the live stream of messages. A bookmark subscription must always start at a specific point in the transaction log. The AMPS server uses a bookmark to locate that point in the transaction log and begin replay at that point. To perform a bookmark replay, the client places a bookmark subscription with the bookmark at which to start the subscription. AMPS will also accept a list of bookmarks, delimited by commas, and replay from the earliest bookmark in the list. If a bookmark is unknown (that is, the transaction log does not contain that bookmark), the AMPS server does not have a specific point at which to begin the replay and will assume that the subscriber has failed over from another instance that has not yet replicated publishes to this instance, and begin replay at NOW (the end of the transaction log) by default. Starting with AMPS 5.3.5, an application can optionally specify that if no bookmark in the list of starting points is found, the subscription should restart from EPOCH or fail rather than using the default NOW behavior. See the following section on _Managing Replay Restart Behavior_ for details. #### Developer Note: the MOST\_RECENT value The AMPS client libraries provide a special constant value that requests that the library look up the appropriate recovery point in the bookmark store and then provide that recovery point in the subscription request. This special value is typically represented as `MOST_RECENT`, `RECENT`, or `recent`. When the application requests a bookmark subscription with a bookmark of `MOST_RECENT`, the client library looks for the most recent bookmarks processed for that subscription, then provides the appropriate bookmark or list of bookmarks when resubscribing. This process helps to ensure that the subscription begins at the last processed message, and the application receives the next unprocessed message for the subscription. If there is no record of a subscription, the AMPS clients will start with `EPOCH`, so that the first time a subscription is entered, the application gets the full record of available messages. It's important to remember that the AMPS server has no knowledge of the `MOST_RECENT` value. `MOST_RECENT` itself is never sent to AMPS and never appears in the AMPS log. `MOST_RECENT` is simply a request to the AMPS client library to look up the exact bookmark value to provide to AMPS. The AMPS client libraries always translate a request for `MOST_RECENT` into either a specific value (typically a list of bookmarks) or `EPOCH`. ## Bookmark Replay from a Moment in Time The final type of bookmark supported is the ASCII-formatted timestamp. When using a timestamp as the bookmark value, the transaction log replays all messages that occurred after the timestamp, and then cuts over to the live subscription once the replay stream has been consumed. This bookmark has the format of `YYYYmmddTHHMMSS[Z]` where: * `YYYY` is the four digit year. * `mm` is the two digit month. * `dd` is the two digit day. * `T` is the character separator between the date and time. * `HH` is the two digit hour. * `MM` is the two digit minute. * `SS` is the two digit second. * `Z` is an optional timezone specifier. AMPS timestamps are always in UTC, regardless of whether the timezone is included. AMPS only accepts a literal value of `Z` for a timezone specifier. For example, a timestamp for January 2nd, 2015, at 12:35: ``` 20150102T123500Z ``` With a timestamp, the AMPS server begins replay at the closest point in the current transaction log, even if there is no message recorded at that exact moment. This means that a timestamp prior to the beginning of the journal will start a replay at EPOCH. ## Bookmark Replay with a Starting and Stopping Point As of version 5.3.2, AMPS allows a subscriber to specify the point at which a bookmark replay should stop. When specified, a subscriber will not receive further messages after the replay reaches the stopping point. Also, when a stopping point is specified and a `completed` acknowledgment is requested for the subscription, AMPS will return the `completed` acknowledgment (if requested) when the stopping point is reached, rather than when replay reaches the point in the transaction log at which the subscription was entered. To set a starting and stopping point, a subscription provides a subscription range specifier with the bookmark. The format of the subscription range specifier is as follows: ```xml : ``` The _begin\_interval\_specifier_ is one of: | Specifier | Behavior | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `(` | _Exclusive replay._ Begin immediately _after_ the bookmark or range specified. The specified bookmark will _not_ be present in the replay. | | `[` | _Inclusive replay._ Begin immediately _before_ the bookmark or range specified. The specified bookmark will be present in the replay. | The _end\_interval\_specifier_ is one of: | Specifier | Behavior | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `)` | _Exclusive replay._ End immediately _before_ the bookmark or range specified. The specified bookmark will _not_ be present in the replay. | | `]` | _Inclusive replay._ End immediately _after_ the bookmark or range specified. The specified bookmark will be present in the replay. | Bookmarks provided for a subscription range specifier can be either timestamps (as described above) or literal bookmarks provided by AMPS. For example, to replay messages received by the instance on June 4, 2020 we could construct an interval beginning at midnight on June 4 UTC (inclusive) and ending at midnight on June 5 UTC (exclusive), as follows: ``` [20200604T000000:20200605T000000) ``` AMPS allows future timestamps in a range specifier. For example, it would be valid for an application to enter a subscription that collects messages for a full business day and then completes, even if the application is started before business hours. AMPS will begin delivering messages at the start time specified, and deliver a `completed` ack (if requested) and stop delivering messages at the end time specified. The starting and stopping points are most often provided as timestamps. AMPS also allows an application to specify starting and stopping points using message bookmarks. In this case, rather than finding the specific time specified, AMPS finds the point in the log at which the bookmark was recorded. An application can choose to submit a list of bookmarks as the starting or stopping point. In this case, AMPS will find the _earliest_ bookmark in the starting point list (as recorded in the local transaction log) and the _latest_ bookmark in the stopping point list (as recorded in the local transaction log), and replay as though those two bookmarks had been provided to the replay command. :::tip An inclusive bookmark with the NOW bookmark `0|1` as the starting point will evaulate the subscription against the last entry in the transaction log. ::: ## Content and Topic Filtering As with all other subscriptions, bookmark subscriptions support content filtering. Bookmark subscriptions provide only messages from topics that are recorded in the transaction log. In other words, when a bookmark subscription uses a topic regular expression, only messages from topics that are recorded in the transaction log are provided to the subscription. This ensures that a bookmark subscription provides a consistent, repeatable stream of messages. The topics provided to the subscription are the same during replay, when only messages recorded in the transaction log are available, and after replay completes, when every publish to AMPS is available. This also ensures that a bookmark subscription that replays messages for a specific timeframe gets the same messages as bookmark subscribers that had active subscriptions during that timeframe. Content filtering is covered in greater detail in [AMPS Expressions](../amps-expressions). ## Delivery Rate Control for Bookmark Subscriptions AMPS allows subscribers to specify the maximum delivery rate for messages delivered from a bookmark subscription. A subscriber specifies the maximum rate at which AMPS should deliver messages to the subscription. AMPS then limits the rate at which replay from the transaction log occurs so that the overall rate does not exceed the specified maximum. Rate control is not available for subscriptions that use the `live` option. To request rate control, a subscriber provides the `rate` option on the subscription. A rate can be specified in either messages per second, number of bytes delivered per second, or a multiple of the original delivery rate. For example, the following subscription option limits delivery to 1000 messages per second: ``` rate=1000 ``` To limit delivery to 500KB per second, a subscriber would provide this option: ``` rate=500KB ``` To limit delivery to double the speed at which messages were originally published, a subscriber would provide this option: ``` rate=2X ``` To limit delivery to half the speed at which messages were originally published, a subscriber would provide this option: ``` rate=.5X ``` When using a `rate` that is a factor of the original replay speed, you may want AMPS to skip over long gaps. For example, you may want to do load testing by replaying several days' worth of operations at a `5x` multiplier. In that case, however, your load test does not need to be idle when there are gaps during which no messages are produced (for example, outside of trading hours or during holidays). For this situation, AMPS provides a `rate_max_gap` option that sets the maximum amount of time for a replay to wait to produce a message. For example, with an option string like: ``` rate=5X,rate_max_gap=10s ``` AMPS will attempt to produce messages at 5 times the original publish rate. In the event that there is a gap between messages of more than 50 seconds in the original publish stream (that is, 10 seconds in the replay), AMPS will wait for 10 seconds and then "skip ahead" to the next message in the replay. ## Pausing and Resuming Bookmark Subscriptions As of version 5.0, AMPS offers the ability to pause a bookmark subscription. When a subscriber requests that AMPS pause the subscription, AMPS stops providing messages from the bookmark subscription, but does not remove the subscription. The subscriber can then resume the subscription, and AMPS will again begin providing messages from the subscription. While the subscription is paused, AMPS maintains a record of the current position in the transaction log, and begins replay from that point. This feature is most useful for starting replay of a number of subscriptions at the same point in the transaction log, and ensuring that those subscriptions are resumed together and progress at the same rate. This can be useful for testing purposes, or for reconstructing a sequence of events that involve multiple subscriptions. Notice that sending a `pause` command for a subscription does not affect messages that have already been sent to a client, and that bookmark replays will automatically pace themselves to the rate that a client is consuming messages. This means that, while `pause` can be used to request that AMPS temporarily halt a subscription, this option is not recommended to control message rate for a client that is oversubscribed. An application may create a subscription in the paused state by including `pause` as an option on the initial `subscribe` command. To pause an active subscription, a subscriber sends a `subscribe` command with the existing subscription ID and the `pause` option. To resume a subscription, a subscriber sends a `subscribe` command with the subscription ID (or a comma-separated list of subscription IDs) and the `resume` option. The AMPS clients provide convenience functions or constants for the `pause` and `resume` options. AMPS allows a given client to pause or resume multiple subscriptions at once. When multiple bookmark subscriptions are resumed at the same time, AMPS will attempt to combine replay for the subscriptions. When AMPS can combine replay, AMPS will guarantee that messages across subscriptions are delivered from the same replay, which can help to preserve order across subscriptions. AMPS can combine subscriptions when they are delivered to the same client connection, were paused at the same bookmark, delivered at the same rate and are resumed with the same command. This feature can be useful for synchronizing message delivery across a number of subscriptions. When using `pause` and `resume` for this purpose, an application typically includes the `pause` option on a number of subscriptions when the subscriptions are created, and then resumes the subscriptions when the application is ready to begin the replay. Pausing a subscription stops AMPS from sending messages to the client once the pause command is processed. However, any messages already on the network, or in a network buffer on the client or the server will be delivered to the client. AMPS allows you to begin a subscription in the paused state by providing the `pause` option when creating the subscription. AMPS removes a paused subscription if the subscriber disconnects: for restarting a subscription across subscriber restarts, use the basic bookmark subscription features as described above. ## Conflation and Bookmark Subscriptions AMPS supports subscription conflation for bookmark subscriptions, as described in [Conflated Subscriptions](../pub-sub/conflation). Conflation for bookmark subscriptions works the same way that conflation for regular subscriptions works. Messages from the replay are held by AMPS for the conflation interval. If during that interval the replay finds a message with the same `conflation_key` value, AMPS replaces the held message with the message from the replay. At the end of the conflation interval, AMPS provides the currently held message to the subscriber. During replay from the transaction log, the conflation interval refers to the timeline of the messages being replayed. That is, a conflation interval of `1s` will provide conflated messages with the same `conflation_key` value published during the same second, even if during transaction log replay the messages are replayed at a much higher rate. When using conflation, the bookmark provided on a message post conflation, is the bookmark for the _first conflated message during the interval_ rather than the message that AMPS delivers at the end of the conflation interval. ## Requesting Message Timestamps Messages that are replayed from the transaction log will not have the timestamp field of the header populated by default. In order to request timestamps, provide the `timestamp` option when creating the subscription. ## Selecting Message Durability Options AMPS supports two distinct options for specifying message durability. By default, messages are provided to a bookmark subscription when they are persisted to the local transaction log. Once replay from the transaction log is finished, AMPS sends messages to subscribers as the messages are processed. By default, AMPS waits until a message is persisted to the local transaction log before sending the message to subscribers. Since each message delivered is persisted, this approach ensures that the sequence of messages is consistent for this instance across client and server restarts, and that messages that are received by a subscriber will be available after a restart. AMPS provides options that a subscriber can use to change the point at which AMPS delivers messages once replay from the transaction log has finished. ### Using the 'fully\_durable' Option for Bookmark Subscriptions With the `fully_durable` option, once replay from the transaction log is finished, AMPS sends a message to the subscriber only when the message has been persisted in the local transaction log and all synchronous downstream replication destinations have acknowledged the message. This option is useful for applications where processing of a message should not begin until more than one AMPS instance has persisted the message. This option will typically introduce more latency for incoming messages when those messages must be replicated. When this option is used and one or more of the synchronous downstream replication destinations that receives messages for this topic is offline, the instance will not deliver incoming messages until that destination comes back online or is downgraded to asynchronous replication. ### Using the 'live' Option for Bookmark Subscriptions In some cases, reducing latency may be more important than consistency. To support these cases, AMPS provides a `live` option on bookmark subscriptions. For bookmark subscriptions that use the `live` option, once replay has finished, AMPS sends messages to subscribers _before_ the message has been persisted. This can provide a small reduction in latency at the expense of increasing the risk of inconsistency upon failover. For example, if a publisher does not republish a message after failover, your application may receive a message that is not stored in the transaction log and that other applications have not received. :::info The `live` option increases the risk of inconsistent data between your application and AMPS in the event of a failover. 60East recommends using this option only if the risk is acceptable and your application requires the small latency reduction this option provides. ::: Since the `live` option does not wait for messages to be persisted, subscriptions that use this option are subject to slow client offlining after replay from the transaction log is complete. The `rate`, `pause`, and `resume` options are not supported with the `live` option. ## Managing Replay Restart Behavior By default, if none of the bookmarks provided in the bookmark option for the subscription are present in the transaction log, AMPS assumes that failover has happened and those messages will arrive (either over replication, or when the publishers republish the message). AMPS begins the subscription at the end of the transaction log. Starting with the 5.3.5 feature release, AMPS includes an option, `bookmark_not_found`, that can be used to control the point where a bookmark subscription restarts in the event that no bookmarks included in the starting point are not found in the transaction log. For example, an application that has a short retention time for data (that is, uses an AMPS configuration that clears journals frequently) may want to replay all data that's present if the application has been offline longer than the retention time. Or the application may want a subscription to fail, to indicate that it may have missed data. AMPS accepts the following three values for the `bookmark_not_found` option: | value | result if no provided bookmark is found | |------ | -------------------------------- | | `epoch` | Begin replay at the start of the transaction log. | | `now` | Begin replay at the end of the transaction log. (Default for bookmark subscriptions if this option is not provided.) | | `fail` | Report a failure if no bookmark in the bookmark string is available. | For example, if a bookmark subscription were to provide a bookmark of `934814|44321|,1234567|123583190|` and an option string of `bookmark_not_found=fail`, AMPS would attempt to begin the subscription at the message with bookmark `934184|44321|` *or* at the message with the bookmark `1234567|123583190|`, whichever is earlier in the transaction log. If neither bookmark is present, AMPS will return an error for the subscription request. :::tip When using `bookmark_not_found=fail`, the application is explicitly requesting that AMPS fail to establish a subscription if the bookmark requested is not present. Applications that use this option and an `HAClient` should set a `FailedResubscribeHandler` on the `SubscriptionManager` that the `HAClient` uses. This handler will receive failure notification if a subscription cannot be made during failover. ::: --- # Using the Transaction Log and Bookmark Subscriptions AMPS includes the ability to record messages in a *transaction log*, and replay those messages at a later time. This capability is key for high availability, since it gives subscribers the ability to resume a subscription at a point in time without missing messages. This capability is also the foundation of replication, since it gives AMPS the ability to preserve message streams to be synchronized to an instance that has gone offline. The transaction log in AMPS contains a sequential, historical record of messages. Each message is identified by a `bookmark`, a unique identifier that AMPS uses to locate the message within the overall set of recorded messages. The transaction log can record messages for a topic, a set of topics, or for filtered content on one or more topics. An application can request a subscription that replays messages from the transaction log. Subscriptions that replay from the transaction log are called *bookmark subscriptions*, since the subscription begins at a specific point in the transaction log identified by a specific bookmark. Bookmark subscriptions provide topic and content filtering in the same way that normal subscriptions do, and provide a set of unique capabilities (such as the ability to pause and resume the subscription) that are made possible because the subscription is provided from a persistent record of the message stream. Bookmark subscriptions are also key to high availability with AMPS. When a client is recovering from a restart or failure, this ability to replay allows a client to fill gaps in received messages and resume subscriptions without missing a message. This feature also allows new clients to receive an exact replay of a message stream. Replay from the transaction log is also useful for auditing, quality assurance, and backtesting. The transaction log is used in AMPS replication to ensure that all servers in a replication group are continually synchronized should one of them experience an interruption in service. For example, say an AMPS instance, as a member of a replication group, goes down. When it comes back up, it can query another AMPS instance for all of the messages it did not receive, thereby catching up to a point of synchronization with the other instances. This feature, when coupled with AMPS replication, ensures that message subscriptions are always available and up-to-date. The AMPS transaction log records messages that are received from a publisher and events that affect those messages such as `sow_delete` commands. AMPS does not record messages that are created through a view, out-of-focus messages, or event status messages created by AMPS. When a subscriber requests a replay from the transaction log, AMPS will deliver `publish` messages from the transaction log in exactly the order in which the instance recorded them. Messages that are not stored in the transaction log (for example, out-of-focus messages or event messages, messages that would have been produced by a view, acknowledgment messages to clients) are not delivered as part of the replay. --- # Understanding Transaction Log Message Persistence To take advantage of transactional messaging, the publisher and the AMPS instance work together to ensure that messages are written to persistent storage. AMPS lets the publisher know when the message is persisted, so that the publisher knows that it no longer needs to track the message. When a publisher publishes a message to AMPS, the publisher assigns each message a unique sequence number. Once the message has been written to persistent storage, AMPS uses the sequence number to acknowledge the message and let the publisher know that the message is persisted. Once AMPS has acknowledged the message, the publisher considers the message published. For safety, AMPS always writes a message to the local transaction log before acknowledging that the message is persisted. If the topic is configured for synchronous replication, all replication destinations have to persist the message before AMPS will acknowledge that the message is persisted. For efficiency, AMPS may not acknowledge each individual message. Instead, AMPS acknowledges the most recent persisted message to indicate that all previous messages have also been persisted, as described in [Acknowledgment Conflation and Publish Acknowledgments](../acks/publish-acks). Publishers that need reliable publishing do not wait for acknowledgment to publish more messages. Instead, publishers retain messages that haven't been acknowledged, and republish messages that haven't been acknowledged if failover occurs. The AMPS client libraries include this functionality for persistent messaging (see descriptions of the _publish store_ in client library documentation). See the [Guaranteed Publishing](../ha/ha-details.md#guaranteed-publishing) section of this guide for further details. ## Client Names and the Transaction Log When a transaction log is configured, AMPS needs to be able to tell the difference between different publishers to be able to reliably persist and replay the message stream. AMPS uses the client name as a unique application identifier to be able to tell when a connection is a connection from a different client as compared to a new connection from the same client. The contract between AMPS and the application is that the application must provide a client name that will uniquely and consistently identify a particular instance of the client application. The same instance of the same application should use the client name each time that instance connects, and should not use the same client name as another instance. An individual instance of AMPS enforces this contract when the transaction log is configured. An individual instance will only allow one connection at a time with a given client name when the transaction log is configured. To enforce this, if two clients attempt to connect with the same client name: * If the clients have the same authenticated user ID (or no user ID is set, in cases where default authentication is used), AMPS will consider this a case where the same program is attempting to reconnect. AMPS will consider the existing connection to be out of date and disconnect the existing connection. * If the clients have different authenticated user IDs, AMPS will consider these to be different applications attempting to use the same client name, and refuse to allow the new connection. AMPS will disconnect the new connection. In either case, AMPS logs the disconnect reason in the event and error log. The disconnect reason for the connection that is removed will be logged as "name in use". ## Message Sequence Numbers Every message stored in the transaction log can be referred to by a bookmark, which is a combination of an identifier for the publisher and the sequence number of the message. AMPS uses the sequence number to identify duplicate messages. If a message arrives at AMPS (either from a publisher or over replication) with a sequence number that is equal to or lower than the highest sequence number seen for that publisher, the message is considered to be a duplicate and discarded. There is no other significance to sequence numbers. The sequence numbers simply represent where, in the sequence of messages sent by that publisher, the current message falls. In most applications, message sequence numbers are automatically managed as part of the store-and-forward mechanism of the AMPS client libraries (implemented as the `PublishStore` for the client). The `PublishStore` assigns sequence numbers and manages reliable publication to AMPS. When a message without a sequence number is received for a topic in the transaction log, AMPS creates a different publisher identifier for these publishes, based on the publisher identifier and the name of the AMPS instance. AMPS uses that identifier for the origin of the message, and assigns a sequence number. This approach provides unique identifiers for the messages while preventing conflicts with sequence numbers that the publisher might use. --- # Using amps-grep to Search the Journal ## Using amps-grep to Find Messages in the Transaction Log The transaction log maintains a full record of the messages published to the topics recorded in the transaction log. Finding an exact sequence of messages recorded by an instance can be useful in troubleshooting issues with applications or AMPS itself. The simplest way to find a sequence of messages in the AMPS transaction log is by using the `amps-grep` utility. When used to find records in a transaction log, `amps-grep` also includes the header information for each journal file in the output, to help in cases where the intent of searching the journal is to be able to preserve a sequence of messages or analyze the journals further. ### Finding Messages from a Specific Client When troubleshooting a publisher, it's often useful to be able to see the set of messages published by a particular publisher. To find messages published from a specific publisher, use the following general pattern: ```bash $amps-grep --client=client_name journal_directory/*.journal > out.txt ``` The `amps-grep` command calculates the set of possible client name hashes for the given client name and returns the transaction log entries for those messages. The Linux shell then writes those entries to the `out.txt` file. ### Finding Messages for a Specific Topic When troubleshooting message flow on a given topic, it's often useful to be able to see the full set of messages published to that topic. To find messages published to a specific topic, provide the topic name to the `amps-grep` utility as follows: ```bash $ amps-grep topic_name journal_directory/*.journal > out.txt ``` The `amps-grep` command searches the journal files for all occurrences of the _topic\_name_ provided and returns the transaction log entries for those messages. ### Finding Messages with Specific Data In some cases, it can also be useful to find records with specific data. For example, this might be helpful to see the set of publishes to a specific key within a SOW topic, regardless of the publisher. To find messages with specific data, provide the data to the `amps-grep` utility as follows: ```bash $amps-grep "data" journal_directory/*.journal > out.txt ``` The `amps-grep` command searches the journal files for all occurrences of the _data_ provided and returns the transaction log entries for those messages. --- # Record and Replay Messages AMPS includes support for transactional messaging, which includes persistence, consistency across restarts, and replay of messages published to AMPS. AMPS message queues use the transaction log to hold the messages in the queue. Transactional messaging is also the basis for replication, a key component of the high-availability capability in AMPS (as described in [Replicating Messages Between Instances](replication) and [Highly Available AMPS Installations](ha). AMPS message queues use the transaction log as a persistent record of the messages that have entered the queue, the order of those messages, and which messages have been acknowledged and removed from the queue. All of these capabilities rely on the AMPS _transaction log_. The transaction log maintains a record of messages. You can choose which messages are included in the transaction log by specifying the message types and topics you want to record. The AMPS transaction log differs from transaction logging in a conventional relational database system. Unlike transaction logs that are intended solely to maintain the consistency of data in the system, the AMPS transaction log is fully queryable through the AMPS client APIs. For applications that need access to historical information, or applications that need to be able to recover state in the event of a client restart, the transaction log allows you to do this, relying on AMPS as the definitive single version of the state of the application. There is no need for complex logic to handle reconciliation or state restoration in the client. AMPS handles the difficult parts of this process, and the transaction log guarantees consistency. Topics covered by a transaction log are able to provide reliable messaging with strict consistency guarantees. When a transaction log is enabled, topics covered by the transaction log provide _atomic broadcast_ from that instance. This means that the instance enforces a repeatable ordering on the messages, and guarantees that all subscribers receive messages reliably, in a consistent order, and with no gaps or duplicates. Enabling a transaction log in an instance also enables the following behaviors: * When a transaction log is enabled, AMPS uses the client name provided by each client to manage the sequencing and integrity of the message stream, and to quickly detect and respond to reconnection. To allow this, each connection to AMPS must have a unique client name. If a duplicate client name is detected, one of the clients is assumed to be defunct and is disconnected. * When a transaction log is enabled, persisted acknowledgments to a publish are conflated, as described in [Acknowledgment Conflation and Publish Acknowledgments](acks/publish-acks). --- # Storage Performance Testing ## amps\_bio\_perf\_test AMPS contains a utility, `amps_bio_perf_test`, that measures the raw performance of storage devices for the type of sequential writes (and, optionally, reads) that are used by the AMPS transaction log. The utility can be used to get a sense of the maximum throughput that a given device offers, and to compare one device to another. ### Options and Parameters | Option | Description | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-b,--batch_size` | The number of messages to be batched (1 to 256). | | `-m,--message_size` | The message size in bytes (256 to 128K, must be a multiple of 256). | | `-f,--file_size` | Size of the data file to write to, in MB (max 8192). | | `-r,--read_mix` | Read mix specified as a percentage (0 - 99). | | `-f,--file` |

    Name of the data file that is created (this should be on the device to be tested).

    The file will be allocated before the test begins running, and will contain nonsense data.

    | 60East recommends running the `amps_bio_perf_test` tool on a system that is otherwise quiet. The `--file` specified is the file that the tool creates for writing data, so that file must be on the device to be measured. The `amps_bio_perf_test` uses the `O_DIRECT` flag for writes (as does the AMPS transaction log writer), bypassing caching at the operating system level. :::danger The `amps_bio_perf_test` tool attempts to fully use the I/O capacity of the target device. Do not run this tool on a system that is being used for another purpose, since the tool may impact the performance of any other process using the device being tested. ::: ### Output The `amps_bio_perf_test` tool produces the following metrics: | Metric | Explanation | | ------------------- | --------------------------------------------------------------------------------------------------------------- | | elapsed time | Time during which the tool was actively writing/reading data. | | writer thread count | Number of threads writing data (always 1). | | batch size | Number of messages (submissions) in a batch. | | submit size | Size of each message (submission), in bytes. | | write size | Total size of each batch written, in bytes. | | submit count | Number of messages (submissions) during the test. | | read count | Number of read operations during the test. | | write count | Number of write operations during the test. | | submit rate | Average number of messages (submissions) per second while the tool was actively writing/reading data. | | write rate | Average number of writes (batches written) per second while the tool was actively writing/reading data. | | actual read mix | Proportion of reads to writes during the test (this may differ from the target due to thread scheduling, etc.). | | mean write latency | Mean latency for a write, in microseconds. | The tool also writes csv files with the raw data used to calculate these metrics, which can be helpful for more in-depth analysis (for example, calculating the 99th percentile latency rather than the mean). One file contains the full set of latency samples collected, the other file contains the full set of throughput samples collected. If the files do not exist when the tool is run, it will create them. Otherwise, the tool will append to the files so that results can be tracked over multiple runs. ### Tips on Evaluating Overall Performance The `amps_bio_perf_test` tool tries to simulate the AMPS transaction log workload under ideal conditions for maximum throughput. The tool preallocates the file that it writes to, and also pregenerates all of the data it will write and the data structures that it will use to record results. During the active part of the test, the tool itself does not allocate memory, but instead measures the time taken for writes and records that data into the preallocated records for the results. The submit rate is the absolute maximum number of messages that could be written per second by that device. To achieve these rates, the payload and metadata size for every message would have to exactly equal the submit size, and the ingress rate from the network would need to exactly equal the rate at which the device can write messages (that is, no gaps in message flow or pushback from the device). The mean write latency (measured in microseconds) is the average amount of time that each write takes to complete. This affects the overall latency of the system, and is strongly correlated with performance (as reflected in the write rate). For example, consider a result that shows a "submit rate" of 2500000 for 512 byte messages. This means that this device could (in principle, under absolutely ideal conditions), support a maximum write throughput of: | 512 bytes | 2,500,000 per second | | ---------------- | -------------------- | | 1024 bytes (1KB) | 1,250,000 per second | | 10KB | 125,000 per second | | 100KB | 12,500 per second | These are, of course, approximations of the expected maximum throughput. In practice, message sizes rarely fit perfectly into a batch, application activity sees bursts and lulls, the number of threads reading from the transaction log varies based on activity (AMPS tries to consolidate replays from the transaction log into the minimum number of threads to save CPU and I/O bandwidth), and the transaction log device may be used for other traffic (for example, SOW topic storage, statistics, or error and event logging) that may share I/O bandwidth. These factors (and others) can affect the actual performance in deployment. ### Usage 60East typically runs the `amps_bio_perf_test` tool with a command line similar to the following: ```bash $ amps_bio_perf_test -b 128 -m 512 -s 8192 -r 0 -f /mnt/fastdrive/ampsdir/bio.data ``` This provides a good baseline for the maximum write performance of the system. --- # Dump clients.ack File `amps_clients_ack_dump` is a utility used to inspect and print the contents of a `clients.ack` file. ## Options and Parameters | Option | Description | | ---------------------------------------------- | ------------------------------------------------ | |

    `filename`

    (required)

    | Filename of the acks file. | | `--version` | Show the version number of the program and exit. | | `-h, --help` | Show the help message and exit. | | `-n LIMIT, --limit` | Maximum number of records to print per file. | :::tip `amps_clients_ack_dump` expects a filename at a minimum in order to complete the `clients.ack` store dump process. ::: --- # Identify Type of AMPS File `amps_file` is a utility that identifies the file type and version number of AMPS files. ## Options and Parameters | Option | Description | | ----------------------------------------------- | -------------------------------------------------------------------------------------------- | |

    `file_name`

    (required)

    |

    The file name to report on.

    This argument supports UNIX shell globbing.

    | ## Usage The following example shows the output of running `amps_file` on a SOW file: ```bash %> ./amps_file /amps_dir/sow/mytopic.sow mytopic.sow: AMPS sow 4.0 ``` In this case, the file is recognized as an AMPS SOW file that uses the version 4.0 of the AMPS SOW file format. --- # Find Information in Error Log or Transaction Log ## amps-grep `amps-grep` is a utility used to search AMPS error logs and journal files. The utility supports both literal search terms and regular expressions. The utility is aware of the structure of AMPS error messages, and returns the full text of a matching error rather than simply the line that matches. The utility can search for a literal match, or use regular expressions. The regular expression dialect supported is the full set of regular expressions supported by the Python `re` module. ### Options and Parameters | Option | Description | | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `files`

    (required)

    | The file or files to search. | | `--version` | Show the version number of the program and exit. | | `--help` | Show the help message and exit. | |

    `-e`,
    `--search_term=TERM`

    | An exact match string to search. | |

    `-f`,
    `--file=LITERAL_TERMS_FILE`

    | Obtain exact matches from `LITERAL_TERMS_FILE`, one per line. | |

    `-E`,
    `--extended_regex=REGEX`

    | Regular expression to search. | |

    `-n`,
    `--line-number`

    | Include line numbers in results. | |

    `-H`,
    `--with-filename`

    | Include filename in results. | |

    `-h`,
    `--no-filename`

    | Do not include filename in results. | |

    `-i`,
    `--ignore-case`

    | Use case-insensitive matching. | |

    `-v`,
    `--invert-match`

    | Invert the sense of matching, to select non-matching lines. | | `--no-data` | Do not display data field in journal results. | | `--include-noops` | Include noops in the journal dump. | | `--client=CLIENTS` | Search for records published by a given client. This argument accepts either a client name or a client name hash. This argument can be specified multiple times to search for multiple clients. | :::info `amps-grep` reads from the standard input if no filenames are provided. ::: ### Usage The following examples shows the output of `amps-grep`. This example simply searches for any occurrence of the term `pubber`, the name of a client, in the AMPS log. ```bash $ amps-grep -e pubber trace.out 2017-09-28T16:57:49.8267470-07:00 [26] trace: 12-0010 client[AMPS-Sample-any-tcp-1-212373446269826439] logon command received: {"c":"logon","cid":"0","client_name":"pubber","mt":"nvfix","a":"processed","version":"5.2.1.1.ff11e5c.358302:python"} 2017-09-28T16:57:49.8269820-07:00 [45] info: 1F-0004 [AMPS-Sample-any-tcp-1-212373446269826439] AMPS client session logon for: pubber client session info: client authid = '' client name hash = 7094790850998930143 client version = 5.2.1.1.ff11e5c.358302:python last acked client seq = 0 last txn log client seq = 0 correlation id = 2017-09-28T16:57:49.8270020-07:00 [45] trace: 17-0002 client[pubber] ack sent: {"c":"ack","cid":"0","s":0,"bm":"7094790850998930143|0|","client_name":"pubber","a":"processed","status":"success","reason":"authentication disabled","version":"5.2.1.17.717892.e742467"} 2017-09-28T16:57:49.8272340-07:00 [26] trace: 12-0001 client[pubber] publish command received: {"c":"p","t":"test"}Data=0^A 2017-09-28T16:57:49.8281260-07:00 [26] trace: 12-0001 client[pubber] publish command received: {"c":"p","t":"test"}Data=5^A 2017-09-28T16:57:49.8281840-07:00 [26] info: 07-0013 client[pubber] disconnected. 2017-09-28T16:57:49.8282820-07:00 [26] info: 16-0004 client[pubber] 0 queued sow requests canceled. ``` --- # Dump journal File The AMPS journal dump utility is used in examining the contents of an AMPS journal file for debugging and program tuning. The `amps_journal_dump` utility is most commonly used as a tool to debug the forensic lifespan of messages that have previously been published to AMPS. The `amps_journal_dump` tool is used to show that messages exist in a journaled topic, and to show the order the message was received in, and the timestamp associated with the message. ### Command Line Options The `amps_journal_dump` program has the following options available. These can also be printed to the screen by typing `amps_journal_dump -help`. | Option | Description | | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `filename`

    (required)

    | Filename of the AMPS journal file. | | `-h, --help` | Show the program help message and quit. | | `-l LIMIT` |

    Limit range of output to entries N:M where N is the first entry and M is the last entry.

    Passing in a single value, M, will return the first M results.

    | | `--localtime` | Display ISO 8601 timestamp in local time. | | `--extents` | Add local and replication extents information at the end of the journal dump. | | `--no-data` | Do not display data values in the journal dump output. | | `--include-noops` | Include noops in the journal dump. | ### Looking at the Output In this section will examine some sample output from running `amps_journal_dump`. We will then go over what each of the entries emitted by the program means. ```bash File Name : AMPS-Sample.0000000000.journal File Size : 10485760 Version : amps::txlog/v8 Extents : [1623783047000000005:1623783047000002563] First Timestamp : 20210615T185233.126619Z Last Timestamp : 20210615T185234.032520Z ______________________________________________________________ entry : 0 crc32 : 2427361060 type : publish flags : none file offset : 4096 tx byte count : 256 msg byte count : 21 msg type : json local txid : 1711058949000000003 previous local txid : 0 source txid : 14124379235191657289 source name hash : 0 client name hash : 15749960451245477525 client seq : 1711058949000000003 topic hash : 10912813577604266571 sow expiration time : 0 iso8601 timestamp : 20240321T220952.535075Z timestamp : 212577862192535075 previous byte count : 0 topic byte count : 13 topic : [sample-record] data : [{"message":"example"}] correlation id byte count : 0 correlation id : [] auth id byte count : 0 auth id : [] rep path byte count : 0 rep path : [] _____________________________________________________ ... Total Entries : 2559 Total Bytes : 10485760 Remaining Bytes : 0 ``` As is apparent in the listing, the output from `amps_journal_dump` is split into three sections, a header, a listing of the contents of the journal file and a footer. The header contains general information about the journal file as it is represented in the filesystem, and state data about the journal file. | Field | Description | | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | File Name | The name of the file as it appears on the local filesystem. | | File Size | Number of total bytes allocated by the journal file. | | Version | This is the version of the formatting used to write the data in the journal file. | | Extents | A pair of numbers where the first is the number of extents used by the journal file, and the second is the number of blocks allocated for the journal file. | The second section of the `amps_journal_dump` lists each of the entries contained in the journal file, along with all of the meta-data used to track and describe the entry. For the sake of simplicity, the example only shows a single journal entry. In an actual installation, `amps_journal_dump` will produce a record for every message in the journal. The exact options present depend on the version of the journal file. | Field | Description | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | entry | A monotonically increasing value representing the order in which the record was inserted into the transaction log file. | | crc32 | The cyclic redundancy check used for error checking the transaction log entry. | | type | The AMPS command used in the original message or the type of entry in the journal file. | | file offset | Offset for this entry within the journal file. | | tx byte count | Size of this entry within the journal file. | | msg byte count | Number of bytes in the message payload. | | msg type | Message type of the message. | | local txid | The monotonically increasing identifier used across all records local transaction log journal files. | | previous local txid | The monotonically increasing identifier used across all records local transaction log journal files. | | source txid |

    The monotonically increasing identifier used across all records local transaction log journal files.

    If a publish from a client does not include a sequence number, and the publish was made directly to this instance, this field can be used to hold the original client name hash for a publish or delete. In that case, the client name hash used for message identification purposes is generated by AMPS (to avoid message identity conflicts).

    | | source name hash | For replicated messages, the hash of the instance from which the message was received. | | client name hash |

    The hash of the publisher client name. The combination of the client name hash and the client seq are used as the identity of the message.

    If a publish or delete does not include a sequence number, and the publish was made directly to this instance, this field may be a hash generated by AMPS while the original client name hash is stored in the source txid field.

    | | client seq | The sequence number of the message from this publisher. | | topic hash | The identifier for the topic name. | | sow expiration time | The expiration set on this individual message when it is used in a SOW topic or queue. | | iso8601 timestamp | The time the message was processed, represented as an ISO-8601 timestamp. | | timestamp | The raw timestamp stored in the AMPS transaction log. This is a microsecond-precision timestamp. | | previous byte count | The size, in bytes, of the previous record in the transaction log. | | topic byte count | The length, in bytes, of the topic name. | | topic | The name of the topic this command applies to. | | data |

    The data for this entry in the transaction log.

    For published messages, this will be the message as published. For other commands, this could be data used internally by AMPS.

    | | correlation id byte count | The number of bytes in the correlation ID for the command. | | correlation id | The correlation ID for the command, if one was set on the command. | | auth id byte count | The length of the authenticated ID that submitted the command. | | auth id | The authenticated ID that submitted the command. | | rep path byte count | The length of the replication path. This path records the route that this message took to reach this instance of AMPS. | | rep path | The replication path. This path records the route that the message took to reach this instance of AMPS. | As seen in the listing, the final section contains general usage information about the data contained in the journal file. | Field | Description | | --------------- | ------------------------------------------------------------------------- | | Total Entries | Total number of journal entries entered into the journal file. | | Total Bytes | The number of reserved bytes consumed by the journal file. | | Remaining Bytes | The number of unused bytes available out of the total reserved file size. | ### Timestamp Formatting The timestamp format used in `amps_journal_dump` is formatted by default using the system timezone for its location. To display the time in another timezone, the `TZ` environment variable can be configured to modify the output. ```bash %> TZ='America/New_York' ./amps_journal_dump A.000000000.journal ``` ```bash %> TZ='Asia/Tokyo' ./amps_journal_dump A.000000000.journal ``` ```bash %> TZ='Europe/London' ./amps_journal_dump A.000000000.journal ``` --- # Find Bookmark or Transaction ID in Transaction Log ## amps\_journal\_search The AMPS journal search utility is used to locate entries in an AMPS journal file. The `amps_journal_search` utility is most commonly used as a tool to debug issues with applications that use AMPS, particularly in cases where it is unclear at what point in time a given message was received by AMPS. The `amps_journal_search` tool can be used to find the precise journal file that contains a given message. ### Command Line Options The `amps_journal_search` program includes the following options. The options available can be printed to the screen by typing `amps_journal_search --help`. | Option | Description | | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `filename`

    (required)

    | Filename of the AMPS journal file. | | `-h, --help` | Show the program help message and quit. | | `` |

    The bookmark or transaction id to locate.

    When a `` is provided, the utility searches the record metadata, but does not search the message data.

    | | `-d , --data ` |

    Locate messages that contain the data in ``.

    If `` is a bookmark, also includes records matching the bookmark.

    | | `-topic=TOPIC_NAME` | Search metadata for the provided topic term. | | `--no-data` | Do not provide message data in the output. | | `--client=CLIENT_HASH` | Search client hash(es) metadata for all matching records. This argument may be provided multiple times to search for multiple clients. | --- # Dump queues.ack File `amps_queues_ack_dump` is a utility used to inspect the contents of a `queues.ack` file. ## Options and Parameters | Option | Description | | ---------------------------------------------- | ------------------------------------------------ | |

    `filename`

    (required)

    | Filename of the acks file. | | `--version` | Show the version number of the program and exit. | | `-h, --help` | Show the help message and exit. | | `-n LIMIT, --limit` | Maximum number of records to print per file. | --- # Submit Minidump to 60East `amps-report-minidump` is a utility used to submit minidumps for analysis by 60East Technologies. This utility will require you to provide your email address, a subject, and the minidumps that you wish to submit. Minidumps may be specified individually, or as a glob. ## Options and Parameters | Option | Description | | -------------------------------------------------------------- | ---------------------------------------------------- | |

    `minidumps`

    (required)

    | The minidump file or files to be submitted. | |

    `-e, --email=SENDER_EMAIL`

    (required)

    | Email address to be used as the sender. | |

    `-s, --subject=SUBJECT`

    (required)

    | The subject field of the email. | | `-b, --body=BODY` | The body of the email. | | `-c, --compress` | Create tar.gz before sending multiple minidumps. | | `-t, --ticket=TICKET` | The ticket number you are submitting a minidump for. | | `--help` | Show the help message and exit. | ## Usage The following examples shows the output of `amps-report-minidump`. This example will compress and submit all minidumps in the current directory. ```bash $ amps-report-minidump -e "your@email.com" -s "Ticket 12345" \ -b "This is the dump we discussed" -c "./*.dmp" Thank you for submitting your minidump(s). If you have not already, please open a ticket with 60East Technologies support. ``` --- # Dump SOW File `amps_sow_dump` is a utility used to inspect the contents of a SOW topic store. Additionally, it can be used to gather summary statistics on a SOW file. ## Options and Parameters | Option | Description | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | |

    `filename`

    (required)

    | Filename of the SOW file. | | `-n LIMIT` | Maximum number of records to print per file. | | `-v, --verbose` | Print record metadata for records and file summary. | | `--sizing-chart` | Print memory sizing chart for efficiency comparison (experimental). | | `-d DELIMITER` | Prints only the record data using the provided ASCII character value as the record delimiter \[default: 10 for newline]. | | `--version` | Show the version number of the program and exit. | | `-h, --help` | Show the help message and exit. | ## Usage The following listing shows a simple sow dump of a `json` format topic. Each key which exists in the `order.sow` file is dumped out to stdout. This output can easily be redirected to a new file, or piped into another program for further analysis. :::info `amps_sow_dump` expects a filename at a minimum in order to complete the SOW topic store dump process. ::: ```bash %> ./amps_sow_dump ./order.sow { "id": 0, "value": 1743 } { "id": 1, "value": 6554 } { "id": 2, "value": 3243 } { "id": 3, "value": 5332 } { "id": 4, "value": 3725 } { "id": 5, "value": 1598 } { "id": 6, "value": 6094 } { "id": 7, "value": 7524 } { "id": 8, "value": 2432 } { "id": 9, "value": 9669 } { "id": 10, "value": 140 } ``` ## Verbose Output The `amps_sow_dump` utility also provides for verbose output, which will display more information about the file and its structure in addition to the records contained in the file. ```bash # This is the last record reported by amps_sow_dump for this sample SOW file. key = 5617746317001299819 crc = 1746452067 flags = slab offset = 4096 allocated = 128 data size = 26 expiration = 0 iso8601 timestamp = 20240819T222516.604685Z local txid = 1724106311000000013 string key = [] correlation id = [] data = [{ "id": 10, "value": 140 }] File : sow/order.sow Version : amps-sow-v3.0 Valid Keys : 11 Record Size : 128 Maximum Records : 11 Multirecords : 0 Maximum record size : 26 Average record size : 26.00 Slab Count : 1 Slab Detail size : 5259264 file offset : 4096 valid count : 11 invalid count : 1 stored bytes : 1408 data bytes : 286 deleted bytes : 5257728 ``` ## Sizing Chart The example below shows the output from the `--sizing-chart` flag. This is feature can be useful in tuning AMPS memory usage and performance. The `Record Size` with the asterisk shows the current `Record Size` setting and allows an AMPS administrator to compare memory usage efficiency along with the potential for a multi-record penalty. :::danger This feature is currently considered to be experimental, so changing AMPS record size configuration based on the results may not necessarily help performance, and could hurt performance in some cases. ::: ```bash %> ./amps_sow_dump --sizing-chart ./order.sow ============================================================= Record Size Store Efficiency Multirecords ============================================================= 128 128 B 100.00% 0 256 256 B 50.00% 0 384 384 B 33.33% 0 512* 512 B 25.00% 0 640 640 B 20.00% 0 768 768 B 16.67% 0 896 896 B 14.29% 0 1024 1024 B 12.50% 0 1152 1.12 KB 11.11% 0 1280 1.25 KB 10.00% 0 1408 1.38 KB 9.09% 0 1536 1.50 KB 8.33% 0 1664 1.62 KB 7.69% 0 1792 1.75 KB 7.14% 0 1920 1.88 KB 6.67% 0 ``` --- # Query Statistics Database For more information on working with the AMPS statistics database, see the section on [AMPS Statistics](../amps-statistics) and the descriptions in the [AMPS Monitoring Guide](../../amps-monitoring-guide/). The AMPS distribution includes a convenience utility, `amps-sqlite3`, for easily running queries against a statistics database. ### Query Mode The `amps-sqlite3` utility query mode takes two parameters, as shown below: #### Options and Parameters | Parameter | Description | | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | |

    `database`

    (required)

    | The sqlite3 database file to query. | |

    `query`

    (required)

    |

    The query to run.

    Notice that the query must be enclosed in quotes, since this is a command-line program run by the Linux shell.

    | The `amps-sqlite3` script joins the `STATIC` and `DYNAMIC` tables together, making a single table that is easier to query on. For example, the script joins the `ICLIENTS_DYNAMIC` and `ICLIENTS_STATIC` tables together into a single `ICLIENTS` table. The `amps-sqlite3` utility also provides a set of convenience functions that can be included in the query. #### Convenience Functions | Option | Description | | -------------------------- | ----------------------------------------------------------------------------------------- | | `ISO8601(timestamp)` | Convert an AMPS statistics `timestamp` to an ISO8601 format string. | | `ISO8601_local(timestamp)` | Convert an AMPS statistics `timestamp` to an ISO8601 format string in the local timezone. | | `timestamp(string)` | Convert the provided ISO8601 format `string` to an AMPS statistics timestamp. | ### Size Mode The `amps-sqlite3` utility size mode takes two parameters, as shown below: #### Options and Parameters | Parameter | Description | | ---------------------------------------------- | --------------------------------------- | |

    `database`

    (required)

    | The sqlite3 database file to query. | |

    `size`

    (required)

    | Print the size of the statistics table. | ### Usage #### Query Mode To use the `amps-sqlite3` utility in query mode, simply provide the file name of the database to query and the query to run. For example, the following query returns the set of samples AMPS has recorded for the `system_percent` consumed on each CPU while the instance has been running: ```bash $ amps-sqlite3 stats.db "select iso8601(timestamp),system_percent from hcpus order by timestamp" ``` #### Size Mode To use the `amps-sqlite3` utility in size mode, simply provide the file name of the database to query and the size command. For example, the following command returns the size of the statistics table: ```bash $ amps-sqlite3 stats.db --size ``` --- # Statistics Database Report `amps_sqlite3_report` is a utility used to extract a specific time range from stats.db for reporting or diagnostic purposes. ## Options and Parameters | Option | Description | | ----------------------------------------------- | ------------------------------------------------------------- | |

    `file`

    (required)

    | The file to search. | |

    `timestamp`

    (required)

    | The timestamp to extract, in the format _YYYYMMDD**T**HHmmSS_ | ## Usage The following examples shows how to extract statistics starting at May 1, 2018 at midnight UTC from `amps-sqlite3_report`. The output of the command is the information from the tables starting from the specified time . ```bash $ amps_sqlite3_report stats.db 20180501T000000 amps-sqlite3-report on stats.db since 20180501T0000 TABLE HCPUS timestamp,static_id,idle_percent,iowait_percent,system_percent,user_percent,static_id:1,oid 212391936000000,2,0,0,0,0,2,all 212391936000000,3,0,0,0,0,3,cpu0 212391936000000,4,0,0,0,0,4,cpu1 212391936000000,5,0,0,0,0,5,cpu2 212391936000000,6,0,0,0,0,6,cpu3 ... etc ... ``` --- # Dump Journal Topic Index File The `amps_tx_topic_index_dump` utility is used to display the contents of an AMPS transaction log topic index file for debugging. ### Command Line Options The `amps_tx_topic_index_dump` utility has the following options available. These can also be printed to the screen by providing the `-help` option to the utility. The utility requires one of the options (`topic`, `topic-hash`, or `message-type`) to produce output. | Option | Description | | ---------------------------------------------- | ------------------------------------------------------ | |

    `filename`

    (required)

    | Filename of the AMPS transaction log topic index file. | | `-h, --help` | Show the program help message and quit. | | `--topic-hash` | Display offset list for a given topic. | | `--topic` | Display offset list for topic. | | `--message-type` | Message type of topic used to produce topic hash. | ### Looking at the Output The output of the utility is a representation of the index. Notice that, since the index is optimized for speed and is only intended to be used to reference journal files, the index contains minimal information. For example, the following dump shows the results from a topic where only three messages are present in the transaction log: ```bash AMPS tx topic index dump: topic hash: 12596264898761041958 transition[0]: 18446744073709551615 txid[1]: 1711058031000000105 journal[2]: 0 offset[3]: 29696 offset[4]: 30976 transition[5]: 18446744073709551615 txid[6]: 1711058031000999830 journal[7]: 24 offset[8]: 4411904 ``` This shows the topic hash used in the transaction log, and information about the journals. In this case, messages are found in journal 0 and journal 24. The first message in journal 0 has local transaction id `1711058031000000105`. The messages in that journal file are at offset `29596` and offset `30976` within the file. The other message is found in journal 24, with local transaction id `1711058031000999830` and is at offset `4411904` in the file. Notice that the "transition" indicators are output to indicate the start of an index record for a different journal file. Also notice that the index contains no information about journals that do not contain messages for that topic, nor do they contain any information about the messages other than that needed to quickly locate messages for a given topic. --- # Obsolete Utility: Upgrade File Formats This utility is included for backward compatibility. The utility can upgrade the format of files created in AMPS version 3.0.3 to 4.3.2 to the formats used in AMPS 5.0 and later versions. Files created or updated by AMPS 5.0 can be successfully used in all subsequent versions of AMPS without modification. This utility is unnecessary in those cases, and should not be used for files that have already been used with 5.0 or later versions of AMPS. :::tip For upgrading from versions 5.0.0.0 or later to this release of AMPS, `amps_upgrade` is not necessary. This utility should not be used for files that have been created or modified by AMPS version 5.0.0.0 or later instances. ::: ## Options and Parameters | Option | Description | | ---------- | ------------------------------------------------------------ | | --verbose | Print additional details on each operation to stdout. | | --trace | Print the operations that `amps_upgrade` performs to stdout. | | Option | Description | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | --from=BASE |

    The root directory of the AMPS installation being migrated.

    This is the directory in which you usually start the `ampServer` process. Any relative paths in the config file will be evaluated relative to this directory.

    | | --config=CONFIG | The xml configuration file for the AMPS server being migrated. | | --work-dir=WORK\_DIR | The working directory from which the `ampServer` is invoked. | | --tmp-dir=TMP\_DIR |

    The temporary directory where upgrade files are written while the upgrade process is underway.

    If this directory does not exist, it will be created. `amps_upgrade` will fail without changing any existing files if this directory already exists and contains files from a previous migration.

    | | Option | Description | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | --check-current | Returns true if the instance is the same version as the `amps_migrate` utility, with no upgrade needed. | | --dry-run |

    Step through the entire upgrade process, printing activity, without making changes.

    Returns `false` if errors are encountered or upgrade is impossible.

    | | --upgrade | Upgrade the instance, returning false if the upgrade is impossible or the upgrade process fails. | | -h, --help | Show usage information and exit. | | --version | Show the program's version number and exit. | ## Usage :::tip This utility should not be used for files that have been created or modified by AMPS version 5.0.0.0 or later instances. ::: Simply upgrade an AMPS instance that's executed in the /amps/server directory from another version to this version, storing temporary files in the /amps/tmp directory: ```bash $ amps_upgrade --from=/amps --config=/amps/config.xml --work-dir=/amps/server --tmp-dir=/amps/tmp --upgrade ``` Try out a migration without actually committing the changes to your AMPS instance: ```bash $ amps_upgrade --from=/amps --config=/amps/config.xml --work-dir=/amps/server --tmp-dir=/amps/tmp --dry-run ``` Check to see if your AMPS instance is current: ```bash $ amps_upgrade --from=/amps --config=/amps/config.xml --work-dir=/amps/server --tmp-dir=/amps/tmp --check-current ``` --- # List/Explain Error Codes ## ampserr AMPS contains a utility to expand and examine error messages which may be observed in the logs. The `ampserr` utility allows a user to input a specific error code, or a class of error codes, examine the error message in more detail and, where applicable, view known solutions to similar issues. ### Options and Parameters | Option | Description | | ------------------------------------------- | ------------------------------------------------------------------------------------- | |

    `error`

    (required)

    |

    The error code to lookup.

    This can also be a regular expression.

    | ### Usage The following example shows the output of the “00-0001” error message: ```bash %> ./ampserr 01-0001 AMPS Message 00-0001 [level = info] DESCRIPTION: AMPS Copyright message. ACTION: No recommended action available. Found 1 error matching '00-0001'. ``` The following example will return all messages that begin with “00-”. NOTE: For the sake of brevity, this example does not include all messages that match this query. ```bash %> ./ampserr 00- AMPS Message 00-0000 [level = trace] DESCRIPTION: Internal log message used by AMPS development team. If you see this message logged, please notify AMPS support. ACTION: No recommended action available. AMPS Message 30-0000 [level = warning] DESCRIPTION : AMPS internal thread monitoring has detected a thread that hasn’t made progress and appears 'stuck'. This can happen with long operations or a bug within AMPS. ACTION : Monitor AMPS and if these 'stuck' messages continue, then a restart of the engine could be the only way to resolve it. If it appears busy (high CPU utilization) then it could be a long operation (large query filter.) ``` The following example will return all error messages. NOTE: For the sake of brevity, this example does not include all messages that match this query. ```bash %> ./ampserr . AMPS Message 00-0000 [level = trace] DESCRIPTION: Internal log message used by AMPS development team. If you see this message logged, please notify AMPS support. ACTION No recommended action available. AMPS Message 30-0000 [level = warning] DESCRIPTION : AMPS internal thread monitoring has detected a thread that hasn’t made progress and appears 'stuck'. This can happen with long operations or a bug within AMPS. ACTION : Monitor AMPS and if these 'stuck' messages continue, then a restart of the engine could be the only way to resolve it. If it appears busy (high CPU utilization) then it could be a long operation (large query filter.) ``` The following example will store information on all error messages. into a file named `current-events.txt`. ```bash %> ./ampserr . > current-events.txt ``` This can be convenient for browsing or further search on the errors and events produced by this version of AMPS. --- # Command-Line Basic Client AMPS contains a command-line client `spark`, which can be used to run queries, place subscriptions, and publish data. While it can be used for each of these purposes, `spark` is provided as a useful tool for informal testing and troubleshooting of AMPS instances. For example, you can use `spark` to test whether an AMPS instance is reachable from a particular system, or use `spark` to perform _ad hoc_ queries to inspect the data in AMPS. This chapter describes the commands available in the `spark` utility. For more information on the features available in AMPS, see the relevant chapters in this guide. The `spark` utility is included in the `bin` directory of the AMPS install location. The `spark` client is written in Java, so running `spark` requires a Java Virtual Machine for Java 8 or later. To run this client, simply type `./bin/spark` at the command line from the AMPS installation directory. It will output its help screen as shown below, with a brief description of the `spark` client features. ```bash %> ./bin/spark =============================== - Spark - AMPS client utility - =============================== Usage: spark help [command] Supported Commands: help ping publish sow sow_and_subscribe sow_delete subscribe Example: %> ./spark help sow Returns the help and usage information for the 'sow' command. ``` ## Getting Help with Spark `spark` requires that a supported command is passed as an argument. Within each supported command, there are additional unique requirements and options available to change the behavior of `spark` and how it interacts with the AMPS engine. For example, if more information was needed to run a `publish` command in `spark`, the following would display the help screen for the `spark` client's `publish` feature. ```bash %>./spark help publish =============================== - Spark - AMPS client utility - =============================== Usage: spark publish [options] Required Parameters: server -- AMPS server to connect to topic -- topic to publish to Options: authenticator -- Custom AMPS authenticator factory to use delimiter -- decimal value of message separator character (default 10) delta -- use delta publish file -- file to publish records from, standard in when omitted proto -- protocol to use (amps, fix, nvfix, xml) (type, prot are synonyms for backward compatibility) (default: amps) rate -- decimal value used to send messages at a fixed rate. '.25' implies 1 message every 4 seconds. '1000' implies 1000 messages per second. Example: % ./spark publish -server localhost:9003 -topic Trades -file data.fix Connects to the AMPS instance listening on port 9003 and publishes records found in the 'data.fix' file to topic 'Trades'. ``` ## Spark Commands Below, the commands supported by `spark` will be shown, along with some examples of how to use the various commands and descriptions of the most commonly used options. For the full range of options provided by `spark`, including options provided for compatibility with previous `spark` releases, use the `spark help` command as described above. ### Publish The `publish` command is used to publish data to a topic on an AMPS server. #### Common Options - `spark publish` | Option | Definition | | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `server`

    (required)

    | AMPS server to connect to. | |

    `topic`

    (required)

    | Topic to publish to. | | `delimiter` | Decimal value of message separator character (default 10). | | `delta` | Use delta publish (sends a `delta_publish` command to AMPS). | | `file` |

    File to publish messages from, stdin when omitted. `spark` interprets each line in the input as a message.

    The file provided to this argument can be either uncompressed or compressed in ZIP format.

    If a ZIP file is provided, it must contain one message per file in the same structure produced by `amps_sow_dump -z`. The ZIP format requires an archive where the first message payload is stored in a file named `1`, the second in a file named `2`, and so on, with all message files located in the ZIP's top-level directory.

    | | `proto` |

    Protocol to use.

    In this release, `spark` supports `amps`, `fix`, `nvfix` and `xml`.

    Defaults to `amps`. `spark` also supports `json` as a synonym for `amps` in this release.

    | | `rate` |

    Messages to publish per second.

    This is a decimal value, so values less than 1 can be provided to create a delay of more than a second between messages. '.25' implies 1 message every 4 seconds. '1000' implies 1000 messages per second.

    | |

    `secure`

    (applies to SSL connections)

    |

    Specifies whether to use an SSL-secured connection (`tcps` URI scheme).

    Values: `false`/`0`/`no` or `true`/`1`/`yes`

    Default: `false`

    | | `type` | For protocols and transports that accept multiple message types on a given transport, specifies the message type to use. | | `uriopts` |

    Custom connection URI parameters to be passed in the URI query string.

    For example:

    `tcp_nodelay=true&tcp_sndbuf=8192`

    | |

    `urischeme`

    (applies to SSL connections)

    |

    Allows a custom URI scheme to be specified.

    When both `secure` and `urischeme` are specified, `urischeme` takes precedence.

    | #### Examples The examples shown below will demonstrate how to publish records to AMPS using the `spark` client in one of the three following ways: a single record, a python script or by file. **Publish a Single Message** ```bash %> echo '{ "id" : 1, "data": "hello, world!" }' | \ ./spark publish -server localhost:9007 -type json -topic order total messages published: 1 (50.00/s) ``` In the example above, a single record is published to AMPS using the `echo` command. If you are comfortable with creating records by hand this is a simple and effective way to test publishing in AMPS. The JSON message is published to the topic _order_ on the AMPS instance. Assuming the _order_ topic is configured as a SOW topic, this publish can be followed with a `sow` command in `spark` to test if the record was indeed published to the _order_ topic. **Publish using Python** ```bash %> python -c "for n in range(100): print('{\"id\":%d}' % n)" | \ ./spark publish -topic disorder -type json -rate 50 \ -server localhost:9007 total messages published: 100 (50.00/s) ``` In the example above, the `-c` flag is used to pass in a simple loop and print command to the python interpreter and have it print the results to `stdout`. The python script generates 100 JSON messages of the form `{"id":0}`, `{"id":1}` ... `{"id":99}`. The output of this command is then _piped_ to spark using the `|` character, which will publish the messages to the _disorder_ topic inside the AMPS instance. **Publish from a File** ```bash %> ./spark publish -server localhost:9007 -type json -topic chaos \ -file data.json total messages published: 50 (12000.00/s) ``` Generating a file of test data is a common way to test AMPS functionality. The example above demonstrates how to publish a file of data to the topic _chaos_ in an AMPS server. As previously mentioned, `spark` interprets each line of a text file as a distinct message. ### SOW The `sow` command allows a `spark` client to query the latest messages which have been persisted to a topic. The SOW in AMPS acts as a database last update cache, and the `sow` command in `spark` is one of the ways to query the database. This `sow` command supports regular expression topic matching and content filtering, which allow a query to be very specific when looking for data. For the `sow` command to succeed, the topic queried must provide a SOW. This includes SOW topics and views, queues, and conflated topics. These features of AMPS are discussed in more detail in this guide. #### Common Options - `spark sow` | Option | Definition | | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `server`

    (required)

    | AMPS server to connect to. | |

    `topic`

    (required)

    | Topic to query. | | `batchsize` |

    Batch Size to use during query.

    A batch size > 1 can help improve performance, as described in the [Querying the State of the World](../sow-queries) chapter of this guide.

    | | `copy` | Publishes records to the secondary server specified. | | `filter` | The content filter to use. | | `format` |

    Optional format used for displaying messages. May contain literal separator characters mixed with format tags.

    Allowed tags are:

    `{bookmark}`, `{command}`, `{correlation_id}`, `{data}`, `{expiration}`, `{lease_period}`, `{length}`, `{sowkey}`, `{user_id}`, `{timestamp}`, `{topic}`

    Notice that not all headers may be available on every request, depending on the options provided to the request. See the [AMPS Command Reference](../../amps-command-reference/) for details.

    Example: `-format "{command}:{data}"`

    | | `orderby` | An expression that AMPS will use to order the results. | | `proto` |

    Protocol to use.

    In this release, `spark` supports `amps`, `fix`, `nvfix` and `xml`.

    Defaults to `amps`. `spark` also supports `json` as a synonym for `amps` in this release.

    | |

    `secure`

    (applies to SSL connections)

    |

    Specifies whether to use an SSL-secured connection (`tcps` URI scheme).

    Values: `false`/`0`/`no` or `true`/`1`/`yes`

    Default: `false`

    | | `topn` | Request AMPS to limit the query response to the first N records returned. | | `type` | For protocols and transports that accept multiple message types on a given transport, specifies the message type to use. | | `uriopts` |

    Custom connection URI parameters to be passed in the URI query string.

    For example:

    `tcp_nodelay=true&tcp_sndbuf=8192`

    | |

    `urischeme`

    (applies to SSL connections)

    |

    Allows a custom URI scheme to be specified.

    When both `secure` and `urischeme` are specified, `urischeme` takes precedence.

    | #### Examples ```bash %> ./spark sow -server localhost:9007 -type json -topic order \ -filter "/id = '1'" { "id" : 1, "data" : "hello, world" } Total messages received: 1 (Infinity/s) ``` This `sow` command will query the _order_ topic and filter results which match the xpath expression `/id = '1'`. This query will return the results in the topic, for example, the record published in the previous publish command. If the topic does not provide a SOW, the command returns an error indicating that the command is not valid for that topic. ### Subscribe The `subscribe` command allows a spark client to register an interest in incoming messages to a topic, so that they will be delivered in real time. Similar to the `sow` command, the `subscribe` command supports regular expression topic matching and content filtering, which allow a subscription to be very specific when looking for data as it is published to AMPS. Unlike the `sow` command, a subscription can be placed on a topic which does not have a persistent SOW cache configured. This allows a `subscribe` command to be very flexible in the messages it can be configured to receive. #### Common Options - `spark subscribe` | Option | Definition | | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `server`

    (required)

    | AMPS server to connect to. | |

    `topic`

    (required)

    | Topic to subscribe to. | | `ack` |

    Enable acknowledgments when receiving from a queue.

    Notice that, when this option is provided, `spark` acknowledges messages from the queue, signaling to AMPS that the message has been fully processed. (See the chapter on [Message Queues](/docs/intro-guide/queues) in this guide for more information.)

    | | `backlog` | Request a `max_backlog` of greater than 1 when receiving from a queue. (See the chapter on [Message Queues](/docs/intro-guide/queues.md) in this guide for more information.) | | `copy` | Publishes records to the secondary server specified. | | `delta` | Use delta subscription (sends a `delta_subscribe` command to AMPS). | | `filter` | Content filter to use. | | `format` |

    Optional format used for displaying messages. May contain literal separator characters mixed with format tags.

    Allowed tags are:

    `{bookmark}`, `{command}`, `{correlation_id}`, `{data}`, `{expiration}`, `{lease_period}`, `{length}`, `{sowkey}`, `{user_id}`, `{timestamp}`, `{topic}`

    Notice that not all headers may be available on every request, depending on the options provided to the request. See the [AMPS Command Reference](../../amps-command-reference/) for details.

    Example: `-format "{command}:{data}"`

    | | `proto` |

    Protocol to use.

    In this release, `spark` supports `amps`, `fix`, `nvfix` and `xml`.

    Defaults to `amps`. `spark` also supports `json` as a synonym for `amps` in this release.

    | |

    `secure`

    (applies to SSL connections)

    |

    Specifies whether to use an SSL-secured connection (`tcps` URI scheme).

    Values: `false`/`0`/`no` or `true`/`1`/`yes`

    Default: `false`

    | | `type` | For protocols and transports that accept multiple message types on a given transport, specifies the message type to use. | | `uriopts` |

    Custom connection URI parameters to be passed in the URI query string.

    For example:

    `tcp_nodelay=true&tcp_sndbuf=8192`

    | |

    `urischeme`

    (applies to SSL connections)

    |

    Allows a custom URI scheme to be specified.

    When both `secure` and `urischeme` are specified, `urischeme` takes precedence.

    | #### Examples ```bash %> ./spark subscribe -server localhost:9007 -topic chaos \ -type json -filter "/name = 'cup'" { "name" : "cup", "place" : "cupboard" } ``` The example above places a subscription on the _chaos_ topic with a filter that will only return results for messages where `/name = 'cup'`. If we place this subscription before a matching message is published, then we will get results similar to above. ### SOW and Subscribe The `sow_and_subscribe` command is a combination of the `sow` command and the `subscribe` command. When a `sow_and_subscribe` is requested, AMPS will first return all messages which match the query and are stored in the SOW. Once this has completed, all messages which match the subscription will then be sent to the client as they update the topic. The `sow_and_subscribe` is a powerful tool to use when it is necessary to examine both the contents of the SOW, and the live subscription stream. #### Common Options - `spark sow_and_subscribe` | Option | Definition | | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `server`

    (required)

    | AMPS server to connect to. | |

    `topic`

    (required)

    | Topic to query and subscribe to. | | `batchsize` | Batch size to use during query. | | `copy` | Publishes records to the secondary server specified. | | `delta` | Request delta for subscriptions (sends a `sow_and_delta_subscribe` command to AMPS). | | `filter` | Content filter to use. | | `format` |

    Optional format used for displaying messages. May contain literal separator characters mixed with format tags.

    Allowed tags are:

    `{bookmark}`, `{command}`, `{correlation_id}`, `{data}`, `{expiration}`, `{lease_period}`, `{length}`, `{sowkey}`, `{user_id}`, `{timestamp}`, `{topic}`

    Notice that not all headers may be available on every request, depending on the options provided to the request. See the [AMPS Command Reference](../../amps-command-reference/) for details.

    Example: `-format "{command}:{data}"`

    | | `orderby` | An expression that AMPS will use to order the SOW query results. | | `proto` |

    Protocol to use.

    In this release, `spark` supports `amps`, `fix`, `nvfix` and `xml`.

    Defaults to `amps`. `spark` also supports `json` as a synonym for `amps` in this release.

    | |

    `secure`

    (applies to SSL connections)

    |

    Specifies whether to use an SSL-secured connection (`tcps` URI scheme).

    Values: `false`/`0`/`no` or `true`/`1`/`yes`

    Default: `false`

    | | `topn` | Request AMPS to limit the SOW query results to the first N records returned. | | `type` | For protocols and transports that accept multiple message types on a given transport, specifies the message type to use. | | `uriopts` |

    Custom connection URI parameters to be passed in the URI query string.

    For example:

    `tcp_nodelay=true&tcp_sndbuf=8192`

    | |

    `urischeme`

    (applies to SSL connections)

    |

    Allows a custom URI scheme to be specified.

    When both `secure` and `urischeme` are specified, `urischeme` takes precedence.

    | #### Examples ```bash %> ./spark sow_and_subscribe -server localhost:9007 -type json \ -topic chaos -filter "/name = 'cup'" { "name" : "cup", "place" : "cupboard" } ``` In the previous example, the same topic and filter are being used as in the `sow_and_subscribe` example above. The results of this query initially are similar, since only the messages which are stored in the SOW are returned. If a publisher were started that published data to the topic that matched the content filter, those messages would then be printed out to the screen in the same manner as a subscription. ### SOW Delete The `sow_delete` command is used to remove records from the SOW topic in AMPS. If a filter is specified, only messages which match the filter will be removed. If a file is provided, the command reads messages from the file and sends those messages to AMPS. AMPS will delete the matching messages from the SOW. If no filter or file is specified, the command reads messages from standard input (one per line) and sends those messages to AMPS for deletion. It can be useful to test a filter by first using the desired filter in a `sow` command and making sure the records returned match what is expected. If that is successful, then it is safe to use the filter for a `sow_delete`. Once records are deleted from the SOW, they are not recoverable. #### Common Options - `spark sow_delete` | Option | Definition | | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `server`

    (required)

    | AMPS server to connect to. | |

    `topic`

    (required)

    | Topic to delete records from. | | `file` | File from which to read messages to be deleted. | | `filter` |

    Content filter to use.

    Notice that a filter of `1=1` is true for every message and will delete the entire set of records in the SOW.

    | | `proto` |

    Protocol to use.

    In this release, `spark` supports `amps`, `fix`, `nvfix` and `xml`.

    Defaults to `amps`. `spark` also supports `json` as a synonym for `amps` in this release.

    | |

    `secure`

    (applies to SSL connections)

    |

    Specifies whether to use an SSL-secured connection (`tcps` URI scheme).

    Values: `false`/`0`/`no` or `true`/`1`/`yes`

    Default: `false`

    | | `type` | For protocols and transports that accept multiple message types on a given transport, specifies the message type to use. | | `uriopts` |

    Custom connection URI parameters to be passed in the URI query string.

    For example:

    `tcp_nodelay=true&tcp_sndbuf=8192`

    | |

    `urischeme`

    (applies to SSL connections)

    |

    Allows a custom URI scheme to be specified.

    When both `secure` and `urischeme` are specified, `urischeme` takes precedence.

    | #### Examples ```bash %> ./spark sow_delete -server localhost:9007 \ -topic chaos -type json -filter "/name = 'cup'" Deleted 1 records in 10ms. ``` With the `sow_delete` command above, we are asking for AMPS to delete records in the topic _chaos_ which match the filter `/name = 'cup'`. In this example, we delete the record we queried previously in the `sow_and_subscribe` example. `spark` reports that one matching message was removed from the SOW topic. ### Ping The spark `ping` command is used to connect to the amps instance and attempt to logon. This tool is useful to determine if an AMPS instance is running and responsive. #### Common Options - `spark ping` | Option | Definition | | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

    `server`

    (required)

    | AMPS server to connect to. | | `proto` |

    Protocol to use.

    In this release, `spark` supports `amps`, `fix`, `nvfix` and `xml`.

    Defaults to `amps`. `spark` also supports `json` as a synonym for `amps` in this release.

    | |

    `secure`

    (applies to SSL connections)

    |

    Specifies whether to use an SSL-secured connection (`tcps` URI scheme).

    Values: `false`/`0`/`no` or `true`/`1`/`yes`

    Default: `false`

    | | `uriopts` |

    Custom connection URI parameters to be passed in the URI query string.

    For example:

    `tcp_nodelay=true&tcp_sndbuf=8192`

    | |

    `urischeme`

    (applies to SSL connections)

    |

    Allows a custom URI scheme to be specified.

    When both `secure` and `urischeme` are specified, `urischeme` takes precedence.

    | #### Examples ```bash %> ./spark ping -server localhost:9007 -type json Successfully connected to tcp://user@localhost:9007/amps/json ``` In the example above, `spark` was able to successfully log onto the AMPS instance that was located on port `9007`. ```bash %> ./spark ping -server localhost:9119 Unable to connect to AMPS (com.crankuptheamps.client.exception.ConnectionRefusedException: Unable to connect to AMPS at localhost:9119). ``` In the example above, `spark` was not able to successfully log onto the AMPS instance that was located on port `9119`. The error shows the exception thrown by `spark`, which in this case was a `ConnectionRefusedException` from Java. ## Spark Authentication `spark` includes a way to provide credentials to AMPS for use with instances that are configured to require authentication. For example, to use a specific user ID and password to authenticate to AMPS, simply provide them in the URI in the format `user:password@host:port`. The command below shows how to use spark to subscribe to a server, providing the specified username and password to AMPS. ```bash $AMPS_HOME/bin/spark subscribe -type json \ -server username:password@localhost:9007 ``` AMPS also provides the ability to implement custom authentication, and many production deployments use customized authentication methods. To support this, the `spark` authentication scheme is customizable. By default, the authentication scheme used by `spark` simply provides the username and password from the `-server` parameter, as described above. Authentication schemes for `spark` are implemented in Java, as classes that implement `Authenticator` -- the same method used by the AMPS Java client. To use a different authentication scheme with `spark`, you implement the `AuthenticatorFactory` interface in `spark` to return your custom authenticator, adjust the CLASSPATH to include the `.jar` file that contains the authenticator, and then provide the name of your `AuthenticatorFactory` on the command line. See the _AMPS Java Client_ API documentation for details on implementing a custom `Authenticator`. The command below explicitly loads the default factory, found in the `spark` package, without adjusting the CLASSPATH. ```bash $AMPS_HOME/bin/spark subscribe –server username:password@localhost:9007 \ -type json -topic foo \ -authenticator com.crankuptheamps.spark.DefaultAuthenticatorFactory ``` ## Spark Support for TCPS `spark` supports secure connections over SSL for AMPS TCPS transports. Details about the available `spark` command options are provided in the tables above. `spark` recognizes the `AMPS_SPARK_OPTS` environment variable for passing Java properties to the underlying JVM. This is needed for SSL properties such as `-Djavax.net.ssl.trustStore` and `-Djavax.net.ssl.trustStorePassword`. Typically, it is sufficient to set the trust store JVM properties in the `AMPS_SPARK_OPTS` environment variable and specify the `secure` option with a valid value. If `secure` is set to `true`, `yes`, or `1`, the connection will use the `tcps` URI scheme; otherwise, it will use `tcp`. ```bash %> export AMPS_SPARK_OPTS="-Djavax.net.ssl.trustStore=./cacerts -Djavax.net.ssl.trustStorePassword=changeit" %> ./spark subscribe -secure 1 -server localhost:10110 -type json -topic Orders ``` The example above places a subscription on the Orders topic over a `tcps` connection. If additional customization of the AMPS connection URI is required, the `urischeme` and `uriopts` options can be specified. For example, a custom AMPS Client Transport can be created and associated with a specific URI scheme by defining a custom Transport class and mapping it to the chosen URI scheme name. In the example below, a custom AMPS Client Transport is associated with the URI scheme `protected`: ```bash %> export AMPS_SPARK_OPTS="-Djavax.net.ssl.trustStore=./cacerts -Djavax.net.ssl.trustStorePassword=changeit -cp secure.jar:spark.jar" %> ./spark subscribe -urischeme protected -server localhost:10110 -type json -topic Orders ``` This will generate the following URI: ```bash protected://localhost:10110/amps/json ``` In this case you would specify the option `urischeme`, which refers to a custom AMPS Client Transport. The custom class would need to be added to the Java classpath using `AMPS_SPARK_OPTS`. If both `urischeme` and `secure` were provided in this example, `urischeme` would override `secure`. For more information on properties to define to customize SSL connections, see the SSL section in the [Advanced Topics](/clients/amps-client-java/advanced-topics#providing-ssl-certificates-to-the-amps-java-client) chapter of the _AMPS Java Developer Guide_. For information on URI parameters that can be used to customize `spark` connections, using the `uriopts` option, see the [Connection Parameters for AMPS](/clients/amps-client-java/connection-parameters) chapter of the _AMPS Java Developer Guide_. --- # Utilities AMPS provides several utilities that are not essential to message processing, but can be helpful in troubleshooting or tuning an AMPS instance. ## File Inspection and Search Utilities The following table lists utilities for inspecting and searching the files created by an AMPS instance. | Utility | Description | | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | [`amps_file`](utilities/amps\_file) | Can be used to report the type of a file, if the file is in a format known to AMPS. | | [`amps_grep`](utilities/amps\_grep) | Can be used to search and extract information from the AMPS error and event log or AMPS journal files. | | [`amps_journal_dump`](utilities/amps\_journal\_dump) | Can be used to examine the contents of an AMPS journal file during troubleshooting, debugging and program tuning. | | [`amps_journal_search`](utilities/amps\_journal\_search) | Can be used to find the transaction log record for a specific bookmark or transaction ID. | | [`amps_sow_dump`](utilities/amps\_sow\_dump) | Can be used to inspect the contents of a SOW topic store. | | [`amps_clients_ack_dump`](utilities/amps\_clients\_ack\_dump) | Can be used to inspect the contents of an AMPS clients acknowledgment file. | | [`amps_queues_ack_dump`](utilities/amps\_queues\_ack\_dump) | Can be used to inspect the contents of an AMPS queues acknowledgment file. | | [`amps_tx_topic_index_dump`](utilities/amps\_tx\_topic\_index\_dump) | Can be used to inspect the contents of an AMPS journal topic index. | ## Submitting Minidump AMPS will create a minidump file to capture the code execution point of all threads when it determines that a problem may be occurring and this information could be useful for troubleshooting. The AMPS distribution includes a utility for submitting a minidump to 60East. | Utility | Description | | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | [`amps_report_minidump`](utilities/amps\_report\_minidump) | Utility for submitting minidump files to 60East at the [crash@crankuptheamps.com](mailto:crash@crankupthe) email address. | ## Working with Statistics The AMPS distribution includes utilities for working with the statistics database. | Utility | Description | | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`amps-sqlite3-report`](utilities/amps\_sqlite3\_report) | Utility for extracting a subset of information from an AMPS sqlite3 database. | | [`amps-sqlite3`](utilities/amps\_sqlite3) |

    Utility for easily querying the AMPS statistics database.

    Provides functions for easily working with AMPS statistics timestamps, and automatically handles joins between the STATIC and DYNAMIC tables in the AMPS statistics schema.

    | ## Planning and Informational Utilities The following table lists utilities for capacity planning and for getting more information about AMPS errors and events. | Utility | Description | | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`ampserr`](utilities/ampserr) |

    Can be used to expand and examine error messages that may be observed in the logs.

    This utility allows a user to input a specific error code, or a class of error codes, examine the error message in more detail, and where applicable, view common recommendations.

    | | [`amps_bio_perf_test`](utilities/amps\_bio\_perf) | Used to measure sequential write performance on a drive (as a way of measuring maximum throughput for writes to the AMPS transaction log). | ## Minimal Command-Line Client The AMPS distribution also includes a minimal client for AMPS that can be invoked on the command line. This minimal client does not offer the full range of functionality available through programmatic access (which includes access through python and javascript). However, this client can be useful for basic diagnostics or simple scripting tasks. | Utility | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`spark`](utilities/spark) |

    Provided as a minimal command-line AMPS client. Some functions of the programmatic AMPS clients can also be accomplished with the `spark` utility.

    This is most useful for adhoc troubleshooting (for example, publishing a diagnostic message to a topic).

    | ## Obsolete Utilities The AMPS distribution also includes the obsolete `amps_upgrade` script, provided for backward compatibility purposes for distribution scripts. It is no longer necessary to run this script for upgrades from AMPS 5.0.0 and later. For upgrades from instances earlier than 5.0.0 to current versions, please contact 60East support for assistance. | Utility | Description | | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [`amps_upgrade`](utilities/amps\_upgrade) |

    Provided for backward compatibility.

    This utility is no longer necessary and should not be used for upgrades from 5.0.0 and later versions of AMPS.

    | --- # Aggregated Subscriptions In addition to precomputed views and aggregates, AMPS provides the ability for the server to compute an aggregation for an individual subscription. When an application requests an _aggregated subscription_, rather than providing messages for the subscription verbatim, the AMPS server will calculate the requested aggregates and produce a message that contains the aggregated data. Most of the time, AMPS applications use views to provide aggregation, as described in the section on [Understanding Views](understanding-views). AMPS views are shared across subscriptions, and are calculated once, when a message updates a view, regardless of the number of subscribers that subscribe to the view. AMPS provides aggregated subscriptions as a way to do _ad hoc_ aggregation in cases where a specific aggregate is only needed for a short period time, will only be used by a single subscriber, or must be provided before the server can be restarted with a defined view. If the aggregation is frequently used, or if multiple subscribers will use the aggregation, consider using a view rather than an aggregated subscription. To request an aggregated subscription, the subscriber provides a definition of the fields to project and the grouping to apply with each subscription. AMPS performs the aggregation and constructs the specified message before delivering the message. For example, imagine a topic in the SOW that uses the `/id` field to create the SOW key. The topic contains the following messages: ```javascript { "id":1, "tickerId" : "IBM", "price" : 150.34 } { "id":2, "tickerId" : "IBM", "price" : 149.76 } { "id":3, "tickerId" : "IBM", "price" : 149.32 } { "id":4, "tickerId" : "IBM", "price" : 151.10 } ``` A subscriber enters a SOW query with the following options: ``` projection=[MAX(/price) AS /max,/tickerId as /ticker],grouping=[/tickerId] ``` AMPS aggregates the messages in the SOW and delivers the following projected record: ```javascript { "ticker" : "IBM", "max" : 151.10 } ``` Aggregated subscriptions are supported for commands that use the SOW: `sow`, `sow_and_subscribe`, and `sow_and_delta_subscribe`. However, there are limitations on some variants of the commands, as described in the following sections. The memory consumed to maintain an aggregated subscription is counted as part of the total memory for the client that submitted the subscription when considering the `MessageMemoryLimit` for that client. ## When to Use Aggregated Subscriptions Aggregated subscriptions require AMPS to compute the aggregate for each subscription individually, at the time that messages are processed for the subscription. In addition, for aggregated subscriptions, the current state of the aggregation is retained for each subscription. In cases where more than one subscriber is using the same aggregation, a `View` is more efficient: each record in the view is only computed once, saving CPU cycles, and ongoing updates for the record are only stored once, requiring less memory. Likewise, if the aggregation uses more than one topic or aggregates messages of a different type than the final result, you must configure a `View` on the server. An aggregated subscription could be more appropriate than a persistent view if one or more of the following is true: * A subscription has **unique and unpredictable aggregation needs**. For example, if no other subscription is computing a given aggregation, and it is not possible to predict in advance the aggregates to compute, then per-subscription aggregation is a good solution. * The application is **under development and iterating quickly**. It can be convenient to use aggregated subscriptions while developing aggregate definitions that will be eventually provided as view topics. * The persistent view is **expensive and seldom needed**. For example, if an aggregation is memory-intensive and only needed once a week at a time when the instance is otherwise lightly-used, the overall memory usage of the AMPS instance may be reduced during the rest of the week by using an aggregated subscription. The considerations above are general guidance to help you consider options between per-subscription aggregation and a persistent view. In general, if it is possible to use an AMPS view for a given aggregation task and that view will be frequently used, a view is often the best option. If a view cannot be used (because the aggregation is not known in advance) or the view would seldom be used, an aggregated subscription may be a better option. ## Requesting an Aggregated Subscription To request an aggregated subscription, set the following options on the subscription: | Option | Description | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `projection=[field specifications]` |

    Specifies a comma-delimited set of fields to project, within brackets. Each entry has the format described in [Constructing View Fields](../builtin_functions/constructing-fields).

    This option must contain an entry for every field in the aggregated message. If there is no entry for a field in this option, that field will not appear in the aggregated message, even if the field is in the underlying message.

    For example, to project the total value of orders for a specific item, you might take the sum of the `/price` multiplied by the `/quantity` for each item, along with the original `/description`, as follows:

    ` projection=[SUM(/price * /quantity) AS /total, /description]`

    When a field appears in the projection option, but is not part of a grouping clause or used in an aggregation function, the message will have the value of that field in the last message processed by AMPS.

    There is no default for this option. When this option is provided, a `grouping` must also be provided.

    | | `grouping=[keys]` |

    For an aggregated subscription, the format of this option is a comma-delimited list of XPath identifiers within brackets.

    For example, to aggregate entries based on their `/description` (producing one record in the aggregation for each distinct value in `/description`), you would use the following option:

    ` grouping=[/description]`

    There is no default for this option. When this option is provided, a `projection` must also be provided.

    | For example, to request a count, by customer, of the order records stored in a topic in the SOW, you could use the following options: ```sql projection=[COUNT(/orderId)AS /orderCount, /customer AS /customer],grouping=[/customer] ``` ## Considerations for Aggregated Subscriptions When planning to use an aggregated subscription, the following considerations apply: * The source of an aggregated subscription is a single topic, of the same message type as will be produced by the aggregation. The topic for the subscription must not be a regular expression. * The topic for the subscription must be a topic in the SOW. This includes views and the SOW view of a queue. * When subscribing to a queue, an aggregated subscription does _not_ remove messages from the queue. Like a view definition with the queue as an underlying topic, an aggregated subscription browses the queue without taking messages from the queue. * Filters for the subscription apply to the original messages, _not_ the results of the projection. A filter for an aggregated subscription is equivalent to the `Filter` element in a `View` definition rather than a filter for a subscription that uses the view. * A subscription that uses per-subscription aggregation does not support the `replace` option **except** for changing pagination options. * An aggregated subscription _cannot_ be a bookmark subscription. That is, replay from the transaction log does not support aggregated subscriptions. * Select lists cannot be used with aggregated subscriptions. --- # Configuring Views in a SOW This section lists the parameters for defining a `View` within the SOW section of an AMPS configuration file. Expand each item for more details. For backward compatibility, AMPS accepts `ViewDefinition` as a synonym for `View`.
    `Name` (required) Defines the topic name for this view. If no `Name` is provided, AMPS accepts `Topic` as a synonym for `Name` to provide compatibility with versions of AMPS previous to 5.0.
    `MessageType` (required) The message type of the view. This does not need to be the same type as any of the topics in the aggregation, but does need to be a message type that supports views. The message type must be one of the message types configured for the instance. AMPS includes `fix`, `xml`, `nvfix`, `json`, `bson` and `bflat` message types with full support for views. You can also use any custom message type defined for the configuration file, provided that the message type supports views. Notice that the `binary` message type does not specify a fixed format for the message contents, so that message type cannot be used in a view. Other message types provided with AMPS may have limitations in their support for views. See the [Message Types](/docs/amps-user-guide/message-types) section for a discussion of the limitations.
    `UnderlyingTopic` (required) Defines the SOW topic or topics on which this view is based. This element can contain a single topic name, or any number of `Join` elements.
    `Projection/Field` (required) Defines what the view will contain. This element can be specified multiple times to compose a complex view. Complex expressions that use aggregation functions and conditional branching can also be used.
    `Grouping/Field` (required) Defines how the records in the underlying topic will be grouped. This is analogous to a SQL `GROUP BY` clause.
    `KeyDomain` The seed value for `SowKeys` used within this topic. The default is the topic name, but it can be changed to a string value to unify `SowKey` values between different topics.
    `Join` Within an `UnderlyingTopic`, each `Join` specifies two topics to join together to create the view, as well as the relationship between those topics. An `UnderlyingTopic` can have any number of `Join` specifications. For more information on `Join` specifications, see the [Multiple Topic Aggregation](/docs/amps-user-guide/views/defining-views#multiple-topic-aggregation-join) section.
    `Conflation` Defines whether AMPS will attempt to conflate updates to the view. This item accepts two values:
    • `none` - No conflation. AMPS fully processes every update to the view, in publication order.
    • `inline` - Conflation is active. AMPS conflates multiple updates to the same underlying record where possible. See the [Inline Update Conflation](/docs/amps-user-guide/views/defining-views#inline-update-conflation) section for details.
    Default: `none`
    `Filter` Defines a filter for the view. When provided, only messages from the `UnderlyingTopic` that match this filter will be included in the view. This option is only valid when the view uses a single `UnderlyingTopic`. When the view contains a `Join` specification, this option may not be used. Default: No filter, which includes all messages from the `UnderlyingTopic`.
    `HashIndex` AMPS provides the ability to do fast query lookup for records in a view based on specific fields. When one or more `HashIndex` elements are provided, AMPS creates a hash index for the fields specified in the element. These indexes are created on startup, and are kept up to date as records are added, removed, and updated. The `HashIndex` element contains a `Key` element for each field in the hash index. AMPS uses a hash index when a query uses an exact string match for all of the fields in the index. AMPS does not use hash indexes for range queries or regular expressions, or for numeric comparisons. AMPS automatically creates a hash index for the fields specified in the `Grouping` for the view.
    `JoinNullEquivalency` Specifies whether the `Join` expressions in this view should consider `NULL` values to be equivalent. When this is `enabled`, missing values, empty strings, and `NULL` values will be considered to match in a `Join` expression. When this is `disabled`, these values will not match. Default: `disabled`
    `FileName` File location to store view data. Unused in this version of AMPS.
    Below is an example of a `SOW` configuration, that shows different approaches to defining a `View`. ```xml showLineNumbers /ett/order fix /orderId nvfix TOTAL_VALUE /ett/order /109 SUM(/14 * /6) AS /71406 /109 orders json /orderId ./sow/%n.sow CompleteByRegion orders json COUNT(/orderId) AS /completedOrders /region AS /region /region /status = 'complete' source-for-hash-sample json /id ./sow/%n.sow hash-sample-view json source-for-hash-sample SUM(/qty) as /quantity /customerName /orderType /customerName /orderType /name /orderType example json /id ./sow/%n.sow example nvfix [json].[example] [json].[example]./id AS /id [json].[example]./id ORDERS nvfix /OrderID ./sow/%n.sow COMPANIES nvfix /CompanyId ./sow/%n.sow TOTAL_COMPANY_VOLUME [ORDERS]./Tick = [COMPANIES]./Tick nvfix [COMPANIES]./CompanyId [COMPANIES]./Tick [COMPANIES]./Name SUM([ORDERS]./Shares) AS /TotalVolume [ORDERS]./Tick ``` --- # Constructing Field Contents The AMPS expression language is used to construct fields in aggregates. The expression language is described in [AMPS Expressions](../amps-expressions), and the available functions that you can use for constructing fields in a view are described in [AMPS Functions](../amps-functions). Using an expression to construct a field for a view is described in [Constructing View Fields](../builtin\_functions/constructing-fields.md#constructing-view-fields). --- # Defining Views and Aggregations Multiple topic aggregation creates a view using more than one topic as a data source. This allows you to enrich messages as they are processed by AMPS, enabling aggregate calculations using information published to more than one topic. You can combine messages from multiple topics and use filtered subscriptions to determine which messages are of interest. For example, you can set up a topic that contains orders from high-priority customers. You can join topics of different message types, and you can project messages of a different type than the underlying topic. To create an aggregate using multiple topics, each topic needs to maintain a SOW. Since views maintain an underlying SOW, you can create views from views. To define an aggregate, you decide: * The topic, or topics, that contain the source for the aggregation * If the aggregation uses more than one topic, how those topics relate to each other * What messages to publish, or _project_, from the aggregation * How to group messages for aggregation * The message type of the aggregation Message types provided with AMPS fully support views, with the following exceptions: * `binary` message types cannot be the underlying topic for a view or the type of a view. * `protobuf` message types can be the underlying topic for a view, but cannot be the type of a view. * `composite-global` message types can be the underlying topic for a view, but cannot be the type of a view. * `struct` message types can be the underlying topic for a view, but cannot be the type of a view. If you are using a custom message type, check with the message type developer as to whether that message type supports aggregation. ## Single Topic Aggregation: UnderlyingTopic For aggregations based on a single topic, use the `UnderlyingTopic` element to tell AMPS which topic to use. All messages from the `UnderlyingTopic` will appear in the aggregation. ```xml MyOriginalTopic ``` ## Multiple Topic Aggregation: Join `Join` definitions tell AMPS how to relate underlying topics to each other. You use a separate `Join` element for each relationship in the view. Most often, the `Join` definition describes a relationship between topics: ``` [topic].[field]=[topic].[field] ``` The topics specified must be previously defined in the AMPS configuration file. The square brackets `[]` are optional. If they are omitted, AMPS uses the first `/` in the expression as the start of the field definition. You can use any number of `Join` expressions to define a multiple topic aggregation. A `Join` definition is an equality comparison between the values of two fields. The `Join` definition is not evaluated as an AMPS expression, so functions, operators (other than `=`) and so forth are not evaluated in these definitions. Within a `Join` definition, values are always compared as strings. This means that values such as `12345`, `12345.00`, and `1.2345E+04` can be considered to be different values by the `Join` expression since these are different strings, even though these strings contain the same numeric value. If your aggregation will join messages of different types, or produce messages of a different type than the underlying topics, you add message type specifiers to the `Join` definition: ``` [messagetype].[topic].[field]=[messagetype].[topic].[field] ``` In this case, the square brackets `[]` around the _messagetype_ are mandatory. AMPS creates a projection in the aggregation that combines the messages from each topic where the expression is true. In other words, for the expression: ```xml showLineNumbers [Orders].[/CustomerID]=[Addresses].[/CustomerID] ``` AMPS projects every message where the same `CustomerID` appears in both the `Addresses` topic and the `Orders` topic. If a `CustomerID` value appears in only the `Addresses` topic, AMPS does not create a projection for the message. If a `CustomerID` value appears in only the `Orders` topic, AMPS projects the message with `NULL` values for any projected values for the `Addresses` topic. In database terms, this is equivalent to a `LEFT OUTER JOIN`. :::info In a `Join` expression, AMPS does not consider `NULL` values, including empty strings, to be equivalent. This behavior matches ANSI SQL behavior. You can disable this behavior and cause `NULL` values to match by including the `JoinNullEquivalency` option in the `View` definition and setting that option to `enabled`. 60East does not recommend setting this option unless it is necessary to preserve backward compatibility with previous versions of AMPS. ::: You can use any number of `Join` definitions in an underlying topic: ```xml [nvfix].[Orders].[/CustomerID]=[json].[Addresses].[/CustomerID] [nvfix].[Orders].[/ItemID]=[nvfix].[Catalog].[/ItemID] ``` In this case, AMPS creates a projection that combines messages from the `Orders`, `Addresses`, and `Catalog` topics for any published message where matching messages are present in all three topics. Where there are no matching messages in the `Catalog` and `Addresses` topics, AMPS projects values used from those topics as `NULL`. To join topics on multiple fields, each expression should be a `Join` clause in the `UnderlyingTopic`. These conditions are considered as a logical `AND` for the records. For example: ```xml showLineNumbers [Orders].[/OrderId]=[OrderExtraInfo].[/OrderId] [Orders].[/OrderType]=[OrderExtraInfo].[/OrderType] ``` This `UnderlyingTopic` specifies that a message in the `OrderExtraInfo` topic must match both the `/OrderId` and `/OrderType` of a message in the `Orders` topic to be joined. ## Setting the Message Type The `MessageType` element of the definition sets the type of the outgoing messages. The message type of the aggregation does not need to be the same as the message type of the topics used to create the aggregation. However, if the `MessageType` differs from the type of the topics used to produce the aggregation, you must explicitly specify the message type of the underlying topics. For example, to produce JSON messages regardless of the types of the topics in the aggregation, you would use the following element: ```xml json ``` ## Defining Projections AMPS makes available all fields from matching messages in the join specification. You specify the fields that you want AMPS to project and how to project them. To tell AMPS how to project a message, you specify each field to include in the projection. The specification provides a name for the projected field and one or more source fields to use for the projected field. The data can be projected as-is, or aggregated using one of the AMPS aggregation functions, as described in the section on [Aggregate Functions](../builtin\_functions/aggregate\_functions) in the [Constructing Fields](../builtin\_functions/constructing-fields) topic. You refer to source fields using the XPath-like expression for the field. You name projected fields by creating an XPath-like expression for the new field. AMPS uses this expression to name the new field. ```xml showLineNumbers [Orders].[/CustomerID] [Addresses].[/ShippingAddress] AS /DestinationAddress SUM([Orders].[/TotalPrice]) AS /AccountTotal ``` :::tip The `Projection` needs to specify a `Field` definition for every field to appear in the `View`. AMPS does not implicitly add fields to a `View`. ::: The sample above uses the `CustomerID` from the orders topic and the shipping address for that customer from the `Addresses` topic. The sample calculates the sum of all of the orders for that customer as the `AccountTotal`. The sample also renames the `ShippingAddress` field as `DestinationAddress` in the projected message. For more information on constructing fields in a view, see the [Constructing Fields](../builtin\_functions/constructing-fields.md#constructing-view-fields) topic. ## Data Types and Projections When projecting views, AMPS converts the original values into the AMPS internal type system and serializes those values into a new message. This approach allows AMPS to efficiently aggregate messages of different types and produce predictable results. The data type of the serialization is determined by the message type of the projected message: the message types provided by 60East in this release project the AMPS internal type. This means that, for message types that rely on type markers to identify the type (such as `bson`), the type of the field in the projected message may reflect the AMPS internal type rather than the original type. This conversion is typically a widening conversion for numeric types (for example, input typed as a 32-bit integer will typically be widened to a 64-bit integer). For other types, the most common conversion is from a specific data type (such as regular expression) to a string type. A projection is evaluated as projecting a single value in the AMPS type system. This means that complex or nested data types are typically projected as the string equivalent. For example, a nested set of XML elements could be projected as an empty string (the text value of the containing element), or an array could be projected as the first value in the array. If necessary, and if the destination data type supports nested data structures, you can project the individual fields of a complex type. For example, given a set of messages like the following (in a Topic with keys of `/orderId` and `/line`): ```js {"orderId":42, "line":1, "detail":{"product":"AAPL", "qty":40}} {"orderId":42, "line":2, "detail":{"product":"AAPL", "qty":60}} ``` You could produce summaries for each detailed product by using a projection like: ```xml showLineNumbers /orderId /detail/product SUM(/detail/qty) as /detail/qty /orderId /detail/product ``` For the messages above, this would produce the following summary record: ```js {"detail":{"product":"AAPL","qty":100.0},"orderId":42} ``` For details on the AMPS data types, see the section that describes the [AMPS Data Types](../amps-expressions/amps-data-types). ## Grouping Use `Grouping` statements to tell AMPS how to aggregate data across messages and generate projected messages. For example, an `Orders` topic that contains messages for incoming orders could be used to calculate aggregates for each customer, or aggregates for each symbol ordered. The `Grouping` statement tells AMPS which way to group messages for aggregation. ```xml [Orders].[/CustomerID] ``` The sample above groups and aggregates the projected messages by `CustomerId`. Since this statement tells AMPS to group by `CustomerId`, AMPS projects a message for each distinct `CustomerId` value. A message to the `Orders` topic will create an outgoing message with data aggregated over the `CustomerId`. :::info Fields used in the `Grouping` element must be fields in the underlying topics. ::: Each field in the projection should either be an aggregate or be specified in the `Grouping` element. Otherwise, AMPS returns the last processed value for the field. :::info Unlike ANSI SQL, AMPS allows you to include fields in the projection that are not included in the `Grouping` or used within the aggregate functions. In this case, AMPS uses the value present in the last message inserted or updated within the grouping as the value for these fields. The value of the field in this case depends entirely on the order in which this instance of AMPS processes inserts and updates to the underlying topic (or topics). Deleting the last message processed does not update this value. Unlike an aggregation function (which processes deletes), a non-aggregated field is not changed by a delete. Upon recovery, AMPS enforces a consistent order of updates when rebuilding the view from the SOW topic to ensure that the value of the field is consistent across recovery and restart. ::: ## Inline Update Conflation AMPS has the ability to _conflate_ updates to a view. Conflation is particularly useful when a view receives a high velocity of updates and subscribers to the view have no need to track every update, but instead want to see the current state of the view as quickly as possible. For applications that have a high update rate and relatively complicated view processing, inline conflation can significantly reduce the total number of updates processed for the view and increase overall throughput. Inline conflation changes how AMPS manages pending updates for a view. Without inline conflation enabled for a view, AMPS processes all messages for a view strictly in the order in which those messages were published. Even if there are multiple pending updates to the same record, AMPS processes each of those messages in turn and updates the view for each message. When inline conflation is enabled and a message arrives with the same `Grouping` value as a message waiting to be processed, AMPS replaces the pending message with the new message, and only processes the new message. Inline conflation _does not_ cause AMPS to slow down the rate at which AMPS processes updates for a view. AMPS continues to process updates for the view as fast as possible, and makes no guarantees as to the number of updates to a view produced by a given set of updates to an underlying topic. The diagram below shows a simplified representation of inline conflation for a view where the underlying SOW uses the `id` field of the message as the `Key`. With conflation set to `none` (the default for a view), each message is added to the end of the messages waiting to be processed, whether or not an update for that group is already waiting. Both updates are processed. By contrast, when conflation is set to `inline`, if there is an existing update waiting, the new update replaces the existing update, and only the new update is processed. Given that inline conflation replaces messages while processing is pending, the following considerations apply to views that enable inline conflation: * Not every update to the underlying topic will produce an individual update to the view: when multiple updates occur to the same record in a short period of time, AMPS may only process the last update. * Updates to the view may be produced in an order different than the order in which the messages were published to the underlying topic, since AMPS replaces messages waiting to be processed. * The final state of the view will be exactly as if each update were processed, since it will be based on the latest values in the underlying topic (or topics). To enable inline conflation, add the `Conflation` element to the configuration for the `View`, as shown below: ```xml showLineNumbers ... inline ... ``` ## Filtering Single Topic Aggregations When a view aggregates a single topic, you can use a `Filter` element in the view definition to limit the messages included in the view to only those messages that match the filter. For example, to aggregate only messages from an underlying topic where the `/status` is complete, you could define your view as follows: ```xml showLineNumbers ... orders json /orderId ./sow/%n.sow CompleteByRegion orders json COUNT(/orderId) AS /completedOrders /region AS /region /region /status = 'complete' ... ``` The `Filter` element is not supported for multiple topic aggregation. --- # Understanding Views Views allow you to aggregate messages from one or more SOW topics in AMPS and present the aggregation as a new SOW topic. AMPS stores the contents of the view as serialized messages in memory, similar to a materialized view in RDBMS software. As the contents of the underlying SOW topic (or topics) change, AMPS updates the view to reflect the current contents of the underlying topic or topics. Views are often used to simplify subscriber implementation and can reduce the network traffic to subscribers. For example, if some clients will only process orders where the total cost of the order exceeds a certain value, you can both simplify subscriber code and reduce network traffic by creating a view that contains a calculated field for the total cost. Rather than receiving all messages and calculating the cost, subscribers can filter on the calculated field. You can also combine information from multiple topics. For example, you could create a view that contains orders from high-priority customers that exceed a certain dollar amount. AMPS sends messages to view topics the same way that AMPS sends messages to SOW topics: when a publish arrives for a message that is used to calculate the view, AMPS recalculates the values in the view as necessary and sends a message on the view topic. Likewise, you can query a view the same way that you query a SOW topic. Defining a view is straightforward. You set the name of the view, the SOW topic or topics from which messages originate and describe how you want to aggregate, or project, the messages. AMPS creates a topic and projects the messages as requested. A view requires one or more underlying topics. Any topic, view, or queue defined in the `SOW` section can be the underlying topic for a view. However, the underlying topic for the view must be defined in the AMPS configuration file before the view itself is defined. All message types that you specify in a view must support view creation. Since AMPS uses the SOW topics of the underlying messages to determine when to update the view, the underlying topics used in a view must be in the SOW. Any topic, view, conflated topic, or queue defined in the `SOW` section can be the underlying topic for a view. However, the underlying topic or topics must be defined in the AMPS configuration file before the view is defined. AMPS updates each view after a publish or delta publish to a message in an underlying topic. Updates are processed for each view in the order in which AMPS processed the updates to the underlying topic. AMPS processes these updates asynchronously, after each SOW update is persisted. For additional performance, AMPS provides the ability to conflate updates to views that process high velocity updates, as described in [Inline Update Conflation](defining-views.md#inline-update-conflation). As with a SOW topic, an incoming publish that does not change the value of the underlying message or the calculated value in the view is considered to be an update to the view topic. If an application needs to see only changed fields, that application should use a delta subscription (with the `no_empties` option). :::tip When the underlying topic for a view is a [queue](../queues), the view will show only the messages in the queue that are not currently leased to subscribers. ::: --- # Best Practices for Views When creating a view, consider the following best practices: * AMPS must compute and serialize each field in the view. Smaller numbers of views, and less expensive calculations, may provide better performance. * AMPS must determine the update to the view for each change to an underlying topic. * For views that join multiple topics, consider the amount of work produced by an update to each topic in the join. You can estimate this by paying attention to the number of matching messages on each side of the join. Consider a view that joins a large `orders` topic to an `order_type` topic with a much smaller number of messages. If, for each `order_type`, there are 10000 matching messages in the `orders` topic, then a publish that updates a message in the `order_type` topic would produce 10000 updates to a view joining these two topics together. In cases like this, avoid making unnecessary updates to an underlying topic with messages that match a large number of messages on the other side of the join. (For example, a change to update a value in `order_type` would be a necessary change. Simply republishing the same messages to `order_type` on a periodic basis would produce a large number of updates to the view without changing the results, and is more likely to be unnecessary work.) * If an underlying message can have frequent updates and subscribers only need to receive the final state of the message in the view, consider using the `inline` element to allow the view to avoid processing intermediate changes where possible. * For topics that are the underlying topics of views, avoid publishing updates that do not change the values of the fields used in the view. Each update to an underlying topic causes an update to the view. In particular, an approach such as republishing a set of lookup values every few minutes will produce a large amount of work (while AMPS fully recalculates the view) without a change in the results. In general, unless the set of values that was republished is significant, avoid republishing values to a topic underlying a view. --- # View Examples The following sections provide examples of practical scenarios that demonstrate AMPS views and how they can be used to aggregate and analyze data. ## Simple Aggregate View Example For a potential usage scenario, imagine the topic `ORDERS` which includes the following NVFIX message schema: | NVFIX Tag | Description | | ------------- | ------------------------------------------------- | | OrderID | Unique order identifier | | Tick | Symbol | | ClientId | Unique client identifier | | Shares | Currently executed shares for the chain of orders | | Price | Average price for the chain of orders | This topic includes information on the current state of executed orders, but may not include all the information we want updated in real-time. For example, we may want to monitor the total value of all orders executed by a client at any moment. If `ORDERS` was a SQL Table within an RDBMS, the “view” we would want to create would be similar to: ```sql showLineNumbers CREATE VIEW TOTAL_VALUE AS SELECT ClientId, SUM(Shares * Price) AS TotalCost, SUM(Shares * Price)/SUM(Shares) AS WeightedAveragePrice FROM ORDERS GROUP BY ClientId ``` As defined above, the `TOTAL_VALUE` view would only have two fields: 1. ClientId: the client identifier 2. TotalCost: the summation of current order values by client Views in AMPS are specified in the AMPS configuration file in `View` sections, which are defined in the `SOW` section. The example above would be defined as: ```xml showLineNumbers ORDERS nvfix /OrderID ./sow/%n.sow TOTAL_VALUE ORDERS nvfix /ClientId SUM(/Shares * /Price) AS /TotalCost SUM(/Shares * /Price) / SUM(/Shares) AS /WeightedAveragePrice /ClientId ``` :::tip Views require an underlying topic in the SOW (which includes queues, conflated topics, or other views). ::: The `Topic` element is the name of the new topic that is being defined. This `Topic` value will be the topic that can be used by clients to subscribe for future updates or perform SOW queries against. The `UnderlyingTopic` is the SOW topic or topics that the view operates on. That is, the `UnderlyingTopic` is where the view gets its data from. All XPath references within the `Projection` fields are references to values within this underlying SOW topic (unless they appear on the right-hand side of the `AS` keyword.) The `Projection` section is a list of 1 or more `Field` tags that define what the view will contain. The field specifications can contain either a raw XPath value, as in `/ClientId` above, which is a straight copy of the value found in the underlying topic into the view topic using the same target XPath or an expression as described in the section on [Constructing View Fields](../builtin\_functions/constructing-fields.md#constructing-view-fields). In the case of `ClientId`, if we had wanted to translate the tag into a different tag, such as `CID`, then we could have used the `AS` keyword to do the translation as in `/ClientId AS /CID`. :::info Unlike ANSI SQL, AMPS allows you to include fields in the `Projection` that are not included in the `Grouping` or used within the aggregate functions. In this case, AMPS uses the last value processed for the value of these fields. AMPS enforces a consistent order of updates to ensure that the value of the field is consistent across recovery and restart. ::: :::warning An unexpected 0 (zero) or null value in an aggregate field within a view usually means that the value is either zero or `NaN`. Most AMPS message types default to using 0 instead of `NaN`. However, any numeric aggregate function will result in a `NaN` if the aggregation includes a field that is not a number. ::: Finally, the `Grouping` section is a list of one or more `Field` tags that define how the records in the `UnderlyingTopic` will be grouped to form the records in the view. In this example, we grouped by the tag holding the client identifier. However, we could have easily made this the “Symbol” tag `/Tick`. In the below example, we group by the `/ClientId` because we want to count the number of orders _for each client_ that have a value greater than 1,000,000: ```xml showLineNumbers ... NUMBER_OF_ORDERS_OVER_ONEMILL ORDERS /ClientId 1000000, /shares * /price, NULL)) AS /AggregateValue]]> SUM(IF(/Shares * /Price > 1000000, /Shares * /Price, NULL)) AS /AggregateValue2 /ClientId nvfix ... ``` Notice that the `/AggregateValue` and `/AggregateValue2` will contain the same value; however `/AggregateValue` was defined using an XML `CDATA` block and `/AggregateValue2` was defined using the XML `>` entity reference. :::tip Since the AMPS configuration is XML, special characters in projection expressions must either be escaped with XML entity references or wrapped in a CDATA section. ::: Updates to underlying topics can potentially cause many more updates to downstream views, which can create stress on downstream clients subscribed to the view. If any underlying topic has frequent updates to the same records and/or a real-time view is not required, as in a GUI, then a replica of the topic may be a good solution to reduce the frequency of the updates and conserve bandwidth. For more on conflated topics, please see [Conflated Topics](../conflated-topics). ## Multiple Topic Aggregate Example This example demonstrates how to create an aggregate view that uses more than one topic as a data source. For a potential usage scenario, imagine that another publisher provides a `COMPANIES` topic which includes the following NVFIX message schema: | NVFIX Tag | Description | | ------------- | --------------------------------- | | CompanyId | Unique identifier for the company | | Tick | Symbol | | Name | Company name | This topic includes the name of the company, and an identifier used for internal record keeping in the trading system. Using this information, we want to provide a running total of orders for that company, including the company name. If `ORDERS` and `COMPANIES` were a SQL Table within an RDBMS, the “view” we would want to create would be similar to: ```sql showLineNumbers CREATE VIEW TOTAL_COMPANY_VOLUME AS SELECT COMPANIES.CompanyId, COMPANIES.Tick, COMPANIES.Name, SUM(ORDERS.Shares) AS TotalVolume FROM COMPANIES LEFT OUTER JOIN ORDERS ON COMPANIES.Tick = ORDERS.Tick GROUP BY ORDERS.Tick ``` As defined above, the `TOTAL_COMPANY_VOLUME` table would have four columns: 1. CompanyId: the identifier for the company 2. Tick: the ticker symbol for the company 3. Name: the name of the company 4. TotalVolume: the total number of shares involved in orders To create this view, use the following definition in the AMPS configuration file: ```xml showLineNumbers ORDERS nvfix /OrderID ./sow/%n.sow COMPANIES nvfix /CompanyId ./sow/%n.sow TOTAL_COMPANY_VOLUME [ORDERS]./Tick = [COMPANIES]./Tick nvfix [COMPANIES]./CompanyId [COMPANIES]./Tick [COMPANIES]./Name SUM([ORDERS]./Shares) AS /TotalVolume [ORDERS]./Tick ``` As with the single topic example, first define the underlying topics as SOW topics. Next, the view defines the underlying topic that is the source of the data. In this case, the underlying topic is a join between two topics in the instance. The definition next declares the message type of the projected messages. The message types that you join can be different types, and the projected messages can be a different type than the underlying message types. The projection uses three fields from the `COMPANIES` topic and one field that is aggregated from messages in the `ORDERS` topic. The projection groups results by the `Tick` symbols that appear in messages in the `ORDERS` topic. ## View Projected Into Different Message Type This example shows how to project an underlying topic of one message type into a topic of a different message type. There is very little difference between this example and the single topic view example above. The main difference is that, because the destination view has a different message type than the underlying topic, every reference to a field from the underlying topic must be fully-qualified with the message type. As before, imagine the topic `ORDERS` which includes the following NVFIX message schema: | NVFIX Tag | Description | | ------------- | ------------------------------------------------- | | OrderID | Unique order identifier | | Tick | Symbol | | ClientId | Unique client identifier | | Shares | Currently executed shares for the chain of orders | | Price | Average price for the chain of orders | As before, we want to project the summation of current order values by client. The `TOTAL_VALUE` view will have two fields: 1. ClientId: the client identifier 2. TotalCost: the summation of current order values by client However, in this case, we want to project the summary into a JSON document. To do this we simply specify that the final view will be in JSON format, and fully qualify all references to the underlying topic in the view definition. The example above would be defined as: ```xml showLineNumbers ORDERS nvfix /OrderID ./sow/%n.sow TOTAL_VALUE [nvfix].[ORDERS] json [nvfix].[ORDERS]./ClientId AS /ClientId SUM([nvfix].[ORDERS]./Shares * [nvfix].[ORDERS]./Price) AS /TotalCost [nvfix].[ORDERS]./ClientId ``` This example uses an underlying topic in NVFIX format, computes an aggregation by `ClientId`, and then produces output in JSON format. --- # Aggregation and Analytics AMPS contains a high-performance aggregation engine, which can be used to project one SOW topic onto another, similar to the `CREATE VIEW` functionality found in most RDBMS software. The aggregation engine can join input from multiple topics, of the same or different message types, and can produce output in different message types. View topics are part of the AMPS SOW, which means that views support delta subscriptions and out of focus (OOF) tracking. A view can also be used as the underlying topic for another view. In addition, for the limited cases where a view is not practical, AMPS allows an individual subscription to request aggregation and projection of a single SOW topic. Notice that the features described in this chapter are designed for cases where an application needs to aggregate data across messages or to perform a calculation on an individual message that should not be preserved as a part of that message. To modify a message as it is published to AMPS, use *preprocessing or enrichment*. To simply retrieve a subset of the fields in a message, use *select lists*. --- # Deployment Checklist ## Introduction and Overview This document presents a basic checklist for successfully deploying AMPS. The checklist is meant to describe the general processes and considerations for an AMPS instance. Depending on the specific needs of your installation, there may be additional factors to consider. The [Operations Best Practices](../amps-user-guide/operation/operations-best-practices) and Deployment Guidance sections of the documentation includes more details on operation and deployment. Every deployment of AMPS should consider the factors in the checklist below. With this list, you can check off each step as it is completed:
    Completed?Task
    [Ensure Sufficient Capacity](#ensure-sufficient-capacity) for the system, including capacity testing as necessary.
    Review [Host Guidance](../amps-user-guide/operation/host-guidance), verify the recommended settings for the host system, [Apply System Configuration](#system-configuration), and add [AMPS NUMA Configuration](#amps-configuration) if necessary for the deployment.
    [Create a Maintenance Plan](#create-maintenance-plan).
    Create and implement a [Monitoring Strategy](#create-monitoring-strategy).
    [Create a Patch and Upgrade Plan](#create-patch-and-upgrade-plan).
    [Create and test the support plan](#create-support-plan-verify-process) to define and verify the process for providing artifacts to 60East, responding to issues, and so on.
    ## Ensure Sufficient Capacity Successful deployment of any system includes making sure that the server capacity and network capacity is sufficient to support the application. 60East recommends using the capacity planning metrics in the latest version of the _AMPS User Guide_ to ensure sufficient memory, disk, and network capacity. Notice that the advice in the [AMPS User Guide](../amps-user-guide/operation/capacity-planning) also recommends allowing extra capacity to ensure that the system has "headroom" in the case of unexpected increases in traffic. ### Capacity Testing 60East recommends testing capacity estimates by running the system in a test environment with hardware and networking as similar to the production environment as possible. In these tests, it is important to simulate peak production load (often 150-200% or more of the historical peak of the system) to ensure that the system can handle load with the expected SLA. Work with the consumers of the application to ensure that the application is not doing unnecessary work (for example, publishing messages that no subscriber is expected to consume, or publishing fields that are not used by subscribers). ### Capacity for Virtual Machines and Containers There is no difference in the capacity planning guidance for AMPS instances hosted on a virtual machine (VM) or in a container as compared with AMPS instances hosted on physical hardware. When the AMPS instance will be deployed on a VM or container, consider both the capacity needs of the host system and the capacity needs of _all_ of the virtual machines and containers hosted on the physical hardware. For details, see [Host Guidance](../amps-user-guide/operation/host-guidance). ## Apply System and AMPS Configuration ### System Configuration Once you have determined the required capacity for the system and the hardware is available, ensure that the system is configured to support AMPS. 60East recommends the following general settings: * NUMA enabled in the OS * Hyperthreading enabled In addition, 60East recommends disabling any settings in the BIOS designed to save power, as these settings limit performance. While each BIOS differs how to enable the changes, 60East recommends: * Disable C-states (set C state to mode 0, active mode) * Disable P-states (set P state to mode 0, highest frequency) The [Linux OS Settings](../amps-user-guide/operation/linux-configuration) section of the documentation contains detailed settings for tuning the Linux operating system for AMPS. Ensure that you have applied the recommended settings to your operating system installation. ### AMPS Configuration If the system will host multiple instances of AMPS, any other process that is pinned to a specific core, or any other process expected to have high CPU usage, disable the NUMA tuning in the AMPS engine to allow the operating system NUMA tuning to distribute AMPS instances across processors. When AMPS runs in a container on a NUMA host, follow the guidance in [Host Guidance](../amps-user-guide/operation/host-guidance#containers). AMPS-level NUMA tuning should remain enabled in a container only when every condition in that section is true. To disable AMPS NUMA tuning, include the following element in the AMPS configuration file: ```xml disabled ``` :::info Set NUMA to `disabled` for an AMPS instance in cases where a given system will host multiple instances of AMPS, where AMPS (or another process) will be pinned to specific cores, where another process will use a significant amount of CPU, where AMPS will be running in a virtual machine, or where a container deployment does not meet every condition in the [Host Guidance](../amps-user-guide/operation/host-guidance#containers) section. ::: Notice that this advice applies only to the NUMA tuning done by AMPS for the AMPS instance itself. The operating system-level NUMA tuning should always remain enabled. ## Create Maintenance Plan To keep AMPS running smoothly, 60East recommends developing a maintenance plan, and creating an action configuration to implement the maintenance plan. AMPS actions provide a way for the server to perform specific tasks in response to specific events. The most common use for actions is to create a maintenance schedule, as described below. Details and configuration for AMPS actions are provided in the [Configuring AMPS for Automation with Actions](/docs/amps-user-guide/actions) section. The following table describes maintenance for various features of AMPS. If your installation of AMPS doesn't use a given feature, there is no need to have a maintenance plan for that feature. | Feature | Maintenance Required | Actions | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | | Statistics Database |

    Truncate unneeded statistics.

    60East recommends retaining statistics for the period of time that you will need to have available for troubleshooting problems.

    For example, if your monitoring policies evaluate throughput twice a week, truncating statistics the day after each evaluation may be reasonable.

    | `amps-action-do-truncate-statistics` | | Transaction Log |

    Remove unneeded journal files.

    60East recommends removing journal files when the messages contained within the file are no longer needed.

    | `amps-action-do-remove-journal` | | Event and Error Logs |

    Remove unneeded log files.

    60East recommends retaining log files for the period of time that you will need to have available for troubleshooting problems.

    | `amps-action-do-remove-files` | 60East recommends that you consider the following maintenance actions, and whether they are appropriate for your installation: | Feature | Optional Maintenance | Actions | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------- | | Transaction Log |

    Archiving journals to higher-capacity storage.

    Journal files moved to the archive directory are still part of the transaction log and are available for replay, replication, queue message delivery, and so on.

    60East recommends moving journals to higher capacity storage in the event that high-speed storage does not have enough space to hold the journals required to meet the needs of the application for access to older messages.

    | `amps-action-do-archive-journal` | | Transaction Log |

    Compressing journals to save space.

    Compressed journal files are still part of the transaction log and are available for replay, replication, queue message delivery, and so on.

    60East recommends compressing journals in cases where storage space is restricted, or where storage performance for older journals is low, such that spending the extra CPU time to uncompress a journal will provide higher performance than retrieving an uncompressed journal.

    The amount of space saved by compression depends on the data contained in each journal file.

    | `amps-action-do-compress-journal` | | State of the World (SOW) |

    Removing unnecessary records.

    60East recommends deleting records from the SOW when they are no longer needed. Many times, this happens as a part of the workflow for the application. In cases where the workflow does not inherently remove unneeded data, consider using an action to maintain the topic.

    | `amps-action-do-delete-sow` | ### Example Maintenance Plan For example, the following maintenance plan runs each day, at 23:00 (11 PM local time). The plan performs the following maintenance: * Archives journal files older than three days * Removes journal files older than 14 days * Truncates statistics to keep the last 7 days of statistics * Removes log files in the `$AMPS_ADMIN/logs` directory that are older than 7 days, and * Removes any records in the `Orders` topic in the SOW that have a status of `canceled` that have not been updated in the last two days. There are a few important points to notice in the filter used to remove records from the `Orders` topic: * Since the timestamps have a unit of seconds, the `Filter` describes 48 hours as 172800 seconds (60 seconds in a minute \* 60 minutes in an hour \* 48 hours). * Since the configuration file is in XML format, the `Filter` uses the XML escape for the `<` symbol (`<`). This is translated into a `<` symbol when the configuration file is parsed. * Within an action, AMPS replaces the token `{{AMPS_UNIX_TIMESTAMP}}` with the point in time that the action begins running. We can use this to create a `Filter` that uses the correct value no matter when the filter is run. ```xml amps-action-on-schedule Nightly maintenance 23:00 amps-action-do-archive-journal 3d amps-action-do-remove-journal 14d amps-action-do-truncate-statistics 7d amps-action-do-delete-files ${AMPS_ADMIN}/logs/*.log 7d amps-action-do-delete-sow Orders json /status = 'canceled' AND LAST_UPDATED() < ({{AMPS_UNIX_TIMESTAMP}} - 172800) ``` ## Create Monitoring Strategy To detect any problems that arise in AMPS or in the underlying hardware, it's important to develop and implement a monitoring strategy. AMPS is designed to be able to work with your existing monitoring infrastructure, including systems such as ITRS Geneos, Grafana, DataDog, and so on. The [Admin and Statistics Reference](/docs/amps-monitoring-guide) describes the metrics available through the Administrative Interface. A complete monitoring strategy would include metrics tailored to the use case, and would use alerting thresholds based on the environment and the guarantees provided by the application. This chapter includes a suggested _minimal_ set of metrics to be tracked in a monitoring system. A full monitoring strategy would likely include additional metrics that are relevant to the specific environment and AMPS features used for the application. :::info The metrics offered here are one suggested baseline set of metrics. Not every metric applies to every installation. For any given installation of AMPS, other metrics are likely also important. See the [Admin and Statistics Reference](/docs/amps-monitoring-guide) for details on the metrics available, and create a monitoring strategy that reflects your environment and how your application uses AMPS. ::: ### Event Logging The AMPS error and event log contains an ordered log of significant events in AMPS. The detail provided depends on the verbosity at which the logging is configured. For a production instance of AMPS, a logging level of `info` (or more verbose) is recommended. Any event recorded with a severity level of `error`, `critical`, or `emergency` indicates that an operation has failed in way that an application may have received partial or incorrect data and should be investigated. Events at an `error` or `critical` level do not necessarily mean that AMPS is not functioning as expected but should still be investigated. For example, if an application submits a command that AMPS doesn't recognize, that would be logged as an `error` level event since that application will not get the expected data, even though AMPS is correctly rejecting an unknown command. However, this event indicates that an application submitted an incorrect operation and is likely not functioning as expected, even though there is no issue in the AMPS server itself. A robust monitoring strategy will monitor the event logs for events of `error` level and above so that those events can be investigated and corrected. ### Baseline Host Metrics Typically, a monitoring system will capture, at a minimum, the following metrics about host-level performance. Since these related to the underlying system rather than AMPS itself, many sites already collect the equivalent of these statistics by default. This is not a complete list of statistics available for the host, but provides a starting point for developing your monitoring plan. **Base Metrics Path**: `/amps/host` | Metric | Short Description | | -------------------------------------------- | ------------------------------------- | | `/memory/free` | Amount of memory currently free. | | `/memory/in_use` | Amount of memory in use. | | `/memory/swap_free` | Amount of swap currently free. | | `/memory/swap_total` | Total amount of swap. | | `/network//bytes_in` | Total bytes in (by interface name). | | `/network//bytes_out` | Total bytes out (by interface name). | | `/disks//file_system_free_percent` | Free space (by device). | | `/cpus/all/iowait_percent` | Amount of CPU time waiting on I/O. | | `/cpus/all/idle_percent` | Amount of CPU time idle. | ### Baseline Message Flow Metrics The following metrics monitor overall message flow to the instance. This is not a complete list of statistics available for message flow, but provides a starting point for developing your monitoring plan. **Base Metrics Path**: `/amps/instance/processors/all` | Metric | Short Description | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `/messages_received_per_sec` | Incoming messages per second from all sources. | | `/denied_reads` | Outgoing messages denied due to entitlement (on valid subscription). | | `/denied_writes` | Incoming messages denied due to entitlement (publishes). | | `/throttle_count` |

    Number of times that the processor had to wait to add a message to the processing pipeline due to the instance reaching capacity limits on the number of in-progress messages.

    This metric can indicate resource constraints on AMPS.

    | | `/last_active` |

    The last active time for a message processor.

    If this time grows, either there is no traffic to the instance, or there is a delay in processing.

    | ### SOW Topic Traffic Metrics The following metrics monitor message flow for specific topics in the SOW (including Topics, Views, ConflatedTopics, Unions, and all replication models for Queues). Depending on your application, of course, a given metric may not be relevant. (For example, if an application only uses queues, then the "update" metrics would not be relevant, since a message can be added to the queue or removed from the queue, but cannot be modified while in the queue.) **Base Metrics Path**: `/amps/instance/sow/!` | Metric | Short Description | | ------------------ | ----------------------------------------------- | | `/inserts_per_sec` | Number of new records added per second. | | `/updates_per_sec` | Number of records updated per second. | | `/deletes_per_sec` | Number of records deleted per second. | | `/queries_per_sec` | Number of queries of the topic per second. | | `/insert_count` | Total count of new records added to the topic. | | `/delete_count` | Total count of records removed from the topic. | | `/update_count` | Total count of updates to records in the topic. | #### View-Specific Metrics If your application uses views, the following metric, when combined with the general topic metrics above, can give you insight into queue activity and the processing load for the queue. **Base Metrics Path**: `/amps/instance/views/!` | Metric | Short Description | | -------------- | --------------------------------------------------- | | `/queue_depth` | The current number of pending updates for the view. | Comparing the queue depth in a statistics snapshot with maximum recorded values for update, insert, and delete for the previous intervals can provide a rough approximation of the current latency for this view. For example, if the maximum number of total updates per second, as calculated by the sum of `updates_per_sec` `inserts_per_sec` and `deletes_per_sec` for the topic is 15,000, a current queue depth of 1500 would be expected to be processed in 100ms or less. #### Queue-Specific Metrics If your application uses queues, the following _minimal_ metrics monitor traffic for a queue. These should be monitored _in addition to_ the general topic metrics above. **Base Metrics Path**: `/amps/instance/queues/!` | Metric | Short Description | | ----------------- | -------------------------------------------------- | | `/seconds_behind` | Age of oldest unacknowledged message in the queue. | | `/queue_depth` | Number of messages in the queue. | ### Replication Destination Metrics The following metrics monitor traffic to an outgoing replication destination. **Base Metrics Path**: `/amps/instance/replication/` | Metric | Short Description | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `/is_connected` | Whether or not this destination is currently connected. | | `/seconds_behind` |

    The current point in the transaction log that has been acknowledged by this destination.

    This is calculated as the difference in seconds between the time that the last message acknowledged by the destination was written to the transaction log and the time that the most recent message was written to the transaction log.

    That is, if the last message that the destination has acknowledged was written to the local transaction log at `12:00:01.100` (one second and 100 ms after 12:00) and the last message in the transaction log was written at `12:00:03.212`, the seconds behind shown in the current statistics would be approximately `2.112`. Acknowledgments are transmitted at a specific interval (1s by default) from the destination instance to the source instance.

    AMPS rounds any value below `1` to `0`.

    Note: This is not an estimate of the time required to synchronize the downstream instance.

    | | `/messages_out_per_sec` | Number of messages sent to the destination (per second). | ### Replicated Queue Metrics The following metrics monitor traffic for a replicated queue. These should be monitored _in addition to_ the general replication metrics above and the instance-specific metrics for the queue. **Base Metrics Path**: `/amps/instance/queues/!` | Metric | Short Description | | ------------------ | ------------------------------------------------------------------------------------------------------ | | `/transferred_in` | Number of messages that have had ownership transferred to this instance. | | `/transferred_out` | Number of messages that this instance has previously owned, but granted ownership to another instance. | | `/owned` | Number of messages currently owned. | ### Application Connection Metrics The following metrics monitor network activity for a client connection. **Base Metrics Path**: `/amps/instance/clients/` | Metric | Short Description | | --------------------- | ------------------------------------------------------------------- | | `/transport_rx_queue` | Number of bytes in the transport receive queue for this connection. | | `/transport_tx_queue` | Number of bytes in the transport send queue for this connection. | | `/bytes_out_per_sec` | Number of bytes per second sent to the client. | | `/bytes_in_per_sec` | Number of bytes per second received from the client. | | `/queue_depth_out` | Number of messages buffered in AMPS for the client. | | `/queue_max_latency` | Oldest message buffered in AMPS for the client. | ## Create Patch and Upgrade Plan 60East releases ongoing hotfixes for supported versions of AMPS. 60East recommends that every AMPS deployment create a regular schedule for updating the version of AMPS in development, test, and production environments. Hotfix releases are designed to address a single issue (although, of course, a single issue may present multiple different symptoms). A hotfix release typically involves simply deploying the updated AMPS distribution and restarting the server. Every hotfix is a cumulative rollup. All hotfixes pass the release certification process, and all hotfixes produced are released as soon as they pass certification. This means that an installation of AMPS can treat _any_ hotfix release as a cumulative upgrade, and that any hotfix release will also contain every previously-released fix. An installation can get the latest fixes at any point in time, without needing to raise a support case. 60East recommends that production deployments develop two processes for qualifying and deploying upgrades: * _Planned patch/upgrade -_ This process is followed for periodic updates and evaluates new releases of AMPS on an ongoing basis, with the intent of having qualified releases roll into production every few weeks or months. Most often, organizations will have patch or upgrade releases continually under evaluation in their development and test environments for regular deployment to production. * _Accelerated patch/upgrade -_ This process streamlines updates in the event that an issue affects a production deployment of AMPS. This is often an accelerated set of tests that provides a high level of confidence for a given fix in a shorter time than a full qualification cycle. Regardless of the process you settle on, 60East recommends that you evaluate and deploy patched versions on a regular basis. Since every patch released by 60East is cumulative for a given version, there is no need to wait for "rollup" or "service pack" releases, which means that an installation can upgrade at any time. 60East typically recommends that critical applications upgrade on a monthly or quarterly cadence for planned upgrade. Less critical applications may be patched less frequently, but should still be updated **at least** every six months. Once you set a cadence for planned upgrade, scheduling testing and upgrade at that interval. Since any hotfix release can be considered a "quarterly rollup" or "cumulative update", when the planned patch window arrives, pick the most current hotfix to test, certify, and deploy. # Create and Test Support Process A robust deployment and operations plan includes being able to easily get support if and when it becomes necessary. 60East recommends creating a support plan and testing the plan, including the process for contacting 60East, running diagnostic tools, and providing diagnostic artifacts such as statistics and event logs. ## Create Support Plan, Verify Process Develop a plan and procedure for contacting support in the event that assistance is needed for a production application. For each instance of AMPS, be sure that the team contacting support can provide the following information as necessary: * Version of AMPS * AMPS configuration files * Event/error logging from AMPS * Statistics database from AMPS It may be helpful to develop a template that operations teams can use when working with 60East to ensure that the relevant information is captured quickly, without requiring additional questions from 60East support. For example, you might use a template like: ``` Hi 60East: We are running AMPS Version: The impact of this issue is: We are seeing the following behavior: What we would expect to see is: ``` This gives 60East support enough background to immediately identify the best available engineer to work on the issue. More information (such as server logs and statistics) may still be needed, but this will help the assigned engineer understand whether those artifacts would be helpful. ### Verify Access to Diagnostic Tools The AMPS distribution includes diagnostic tools for inspecting AMPS artifacts (for example, `amps-grep`, `amps-sqlite3`, `amps_journal_dump`, and so on). An operations team responsible for running AMPS will typically need access to these tools to be able to troubleshoot issues. These tools make it easier to work with artifacts that are stored in text format (such as the error and event logs), and make it possible to inspect artifacts that are stored in binary format (such as journal files and SOW files). It may not be necessary for these tools to be available on production servers. Often, a good strategy is to be able to run the tools on non-production servers or a sandbox set aside for investigation and to be able to move artifacts to the sandbox or non-production server for analysis. ### Test Support Channels Before Deployment Before going into production, it is helpful to test that any team that will be responsible for troubleshooting AMPS operations has the ability to retrieve logs and statistics for the AMPS servers and has a process in place for providing those artifacts to 60East. If necessary, the 60East support team can assist with providing a "process test" support case for an end-to-end test of the process. It is particularly important to verify that artifacts can be retrieved from the actual production servers that AMPS is deployed on _before_ an issue emerges. At many sites, production servers have more restrictions than development or test servers. A process designed and verified in a test environment may need to be modified for production. Once a process for retrieving artifacts from production is developed, document the process so that information can be efficiently transferred if an issue arises. ### Verify Support Coverage / Plan 60East support responds as soon as possible to issues that emerge. During hours when coverage is provided (as specified in your license agreement), 60East provides a guaranteed SLA to respond to operational issues and work with you toward a resolution. Outside of those hours, 60East still attempts to provide support for critical issues -- even if those issues arise outside of the covered support times that an installation has chosen. However, support outside of contracted times is provided on an "as available" basis, with no guaranteed response time or level of engineer availability. Although 60East support typically responds to issues rapidly, teams should have a plan in place for a worst case scenario where 60East support is unable to respond until support coverage begins if an issue arises outside of the agreed upon support coverage hours. A team should have a strategy for managing issues that emerge until 60East can respond, or until coverage hours begin. For example, if an application is intended to be available 7 days a week, but the team has chosen a support plan that provides support during weekday business hours, the operations team for the application should have a plan in place for managing issues that fall outside of coverage hours, since responses outside of contracted support hours may be slower than the typical response times. ## Conclusion This document lists the minimum set of considerations for deploying a production instance of AMPS. By necessity, the checklist presents general guidance and is meant to be an outline to help with your planning and rollout process rather than attempting to cover all of the factors that might be involved in a particular deployment. For assistance and evaluation of your individual deployment plan, contact 60East at [https://crankuptheamps.com/support](/support). --- # Glossary | Term | Definition | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Acknowledgment | A networking technique in which the receiver of a message is responsible for informing the sender that the message was received.

    In AMPS:

    • Commands to the AMPS server from an application are asynchronous: AMPS responds with acknowledgment messages to indicate the results of the command.
    • An application acknowledges messages from an AMPS queue to indicate that the message has been processed and the application can accept more work.

    | | Authentication | The process of establishing a proven identity for a connection to AMPS. | | Bookmark | Unique identifier for a message, formed from a combination of a number derived from the client name (which is used as the publisher session ID) and a sequence number managed by the client. Unlike a *SOW Key* (defined later), this identifier is unique to each individual message. That is, two updates to the same record have different bookmarks, even though they update the same key. | | Conflated Topic | A copy of a SOW topic that conflates updates on a specified interval. This helps to conserve bandwidth and processing resources for subscribers to the conflated topic. | | Conflation | The process of merging a group of messages into a single message. For example, when a particular record in the SOW is updated hundreds or thousands of times a second, conflation can enable an application to receive the most recent update every 300ms, reducing the network traffic to the application while still guaranteeing that the application has recent data. | | Delta | A message that contains only the differences between the previous state of a stored message and the new state of the stored message. AMPS supports delta messaging for both publish (changing a subset of fields in a message) and subscribe (receiving only the fields of a message that have changed). | | Entitlement | The process of assigning permissions to a connection based on the identity established for that connection. | | Expression | A text string that produces a specific value. AMPS uses expressions in filters and when constructing fields for enrichment or projecting views. | | Filter | A text string that is used to match a subset of messages from a larger set of messages. In AMPS, every filter is an AMPS expression that returns TRUE or FALSE. | | Message Expiration | The process where the life span of records stored in a State-of-the-World topic or queue are limited. | | Message Type | The data format used to encapsulate messages. Each message within AMPS has a single, defined message type. Each connection to AMPS uses a single, defined message type. | | Module | A shared object that provides functionality to the AMPS server. Modules are used to extend AMPS functionality: capabilities such as authentication, entitlement, the ability to parse and serialize messages of a given format (message type), and functions in the AMPS expression language are implemented as modules. AMPS loads a default set of modules automatically. Other modules may be loaded using directives in the AMPS configuration file. | | oof (out of focus) | Notification to a subscriber that a message which was previously a result of a SOW or a SOW subscribe filter result has either expired, been deleted from the SOW or has been updated such that it no longer matches the filter criteria. | | Queue | A topic that provides competitive consumption with the goal of allowing an application to process a message once, regardless of the number of consumers. Notice that a queue is not necessary to provide reliable replay or consistent ordering -- the transaction log provides those properties. | | Replication | The process of duplicating the messages stored into an AMPS instance to one or more additional AMPS instances. The instance that contains messages (and related commands, for example, deletions from a SOW topic) pushes those messages to the instance that receives the commands over a transport designed for that purpose. AMPS replication occurs on a command-by-command basis for low latency. | | Replication Destination | An instance of AMPS that is receiving messages directly from another AMPS instance (the *replication source*). | | Replication Source | An instance of AMPS which receives a message and then sends the message directly to one or more other AMPS instances (the *replication destinations*). The source is responsible for ensuring that the destination or destinations have received the messages. | | Replication Transport | A transport used only for replication (`amps-replication` transport), that allows the flow of incoming replication messages from another AMPS instance (the *replication source*). | | Slow Client | A client that is being sent messages at a rate which is faster than it can consume, to the point where AMPS detects that the network buffer to the client has filled. This term is used any time that the outgoing network buffer to a client is filled, regardless of whether this is due to processing speed on the client, a network slowdown, or AMPS simply producing more results than the network can immediately transmit. | | SOW (State of the World) Topic | A last value cache used to store the current state of messages belonging to a topic. These topics can also be treated as a message database with the key fields determining message uniqueness. | | SOW Key | A value used to identify a unique record in an AMPS SOW topic. For a given topic, you can configure AMPS to generate the SOW key based on content in the message, provide the SOW key on each message published, or use a SOW key generator module to programmatically create the SOW key. Within a SOW topic, publishes to the same SOW key value are interpreted updates to the same record. | | Topic | A label, which is affixed to every message by a publisher, which is used to group messages for routing and delivery. Messages within a topic all have the same message type, persistence, and delivery paradigm. Topics are a broad partition of messages within AMPS. AMPS also allows subscribers to further refine subscriptions based on the content of the message. | | Transaction Log | A history of all messages published for a configurable set of topics which can be used to recreate an up to date state of all messages processed. Applications can query and replay messages from the transaction log. The transaction log preserves the order in which messages are processed by an instance of AMPS both within a topic and across topics. | | Transport | The network protocol used to to transfer messages between AMPS subscribers, publishers and other instances of AMPS (via replication). | | View | An in-memory topic constructed by AMPS from the contents of one or more SOW topics. A view can aggregate or transform the underlying topics, and can be of a different message format than the underlying topics. As the contents of the SOW topic change, the view is updated to reflect the current contents of the underlying topic or topics. | --- # Introduction to AMPS The guide focuses on a broad overview of the most-commonly used features in AMPS. For detailed information, see the _AMPS User Guide_. --- # Galvanometer and RESTful Statistics When the `Admin` interface is configured (as it is in the sample configuration), you can get information about the state of the AMPS instance using either the Galvanometer monitoring tool or the RESTful interface to the AMPS statistics. The Galvanometer is a Javascript-based application that runs in your browser and provides a visualization of the data provided by the RESTful interface. The Galvanometer also includes a lightweight read-only AMPS client application, based on the Javascript client library. The RESTFul admin interface is a lightweight view of the statistics database that AMPS maintains. These two interfaces are available at the following URIs: | Interface | URI | | ------------------ | --------------------------- | | Galvanometer | `http://:/` | | RESTful Statistics | `http://:/amps` | In the URIs above, `` is the host the AMPS instance is running on and `` is the administration port configured in the configuration file (this is `8085` in the sample configuration). Monitoring applications typically collect information from the RESTful statistics interface. Interactive or ad hoc monitoring can use either the Galvanometer, or the interface offered by the monitoring application in use locally once the statistics are collected. For more information on the monitoring capabilities available in AMPS, see the chapter on [Monitoring AMPS](../amps-user-guide/monitoring) in the _AMPS User Guide_. For detailed information on the statistics available, see the [AMPS Monitoring Guide](../amps-monitoring-guide/). --- # Advanced Topics ## Further Reading While there is much more content beyond the scope of this document, here are some of the topics to learn about after reading this guide. ### Event Logging AMPS provides a rich logging framework that supports logging to many different targets including the console, syslog, and files. Every error and event message within AMPS is uniquely identified and can be filtered out or explicitly included in the logger output. The [Logging](../amps-user-guide/logging) section in the [AMPS User Guide](../amps-user-guide/) describes the AMPS logger configuration and the unique settings for each logging target. ### Conflation for Topics and Subscriptions Another challenge that faces developers working with high-volume data flows is the fact that not every consumer can keep up with the rate at which data arrives. For example, an application may display a view of data that is updated hundreds or thousands of times a second. The update rate for some data can be faster than the UI framework can redraw the grid that holds the data. Without a strategy for managing these updates, the application can be unresponsive, show outdated data, consume a large amount of memory -- or have all of those problems at the same time. To help in this situation, AMPS provides built-in support for limiting the volume of updates. This feature is called _conflation_. With AMPS conflation, an application receives updates for a particular message at most once within a specified interval. When an update for a record is sent to the application, the update contains the most current state of the record at that time. The application always receives the most current data. However, no matter how many times the record is updated during the interval, the application only receives the most current update at the end of the interval. AMPS provides two forms of conflation: * _Conflated topics_ are declared in the server configuration. The AMPS server keeps only a single copy of the current message state for all subscribers. * _Conflated subscriptions_ are requested by an individual subscriber. AMPS keeps a copy of the current message state for each subscription, and that copy of the message state is not shared between subscriptions. As an example of the value of conflation, imagine a SOW topic called `PRICING` that contains the current price for a set of instruments, and imagine that updates to the pricing are being published to the topic in real time. Several applications subscribe to this topic to display the latest prices for a subset of the instruments in a GUI front-end. If this GUI front-end only needs updates in two second intervals from the `PRICING` topic, then more frequent updates would be wasteful of network and client-side processing resources. Likewise, if the GUI front end attempted to process and display every update to the prices, the incoming volume of updates might well outpace the ability of the grid to update. Using conflation in this case can both reduce network traffic and ease the load on the application. In this case, every instance of the application is likely to have the same performance characteristics and benefit from the same interval for conflation. Therefore, configuring a conflated topic for the server would be a good approach. If there were only a single instance of this application or the application ran intermittently (for example, a monitoring or diagnostic tool), using a conflated subscription might be more appropriate. The _User Guide_ provides more info on conflation, conflated topics, and conflated subscriptions. ### View Topics and Aggregation AMPS contains a high-performance aggregation engine, which can be used to project one topic onto another, similar to the `CREATE VIEW` functionality found in most RDBMS software. Views can JOIN multiple topics together, including topics with different message types. In addition to views configured by an administrator, individual subscriptions can create _ad hoc_ aggregates and views on demand. ### Paginated Subscriptions For some use cases, in particular interactive applications that display large sets of records, it's useful to be able to display a subset of all of the records of interest. This saves network bandwidth by only delivering records that the application intends to display to a user, and saves CPU time in the application by removing the requirement for the application to process and discard records that aren't in the current result set. For example, a web application may potentially show thousands of orders, but may only need to render a page of 20 records at any given time. With a paginated subscription, the application can request exactly the records it needs to render, and can be notified when those records change, are deleted, or if another record is inserted within the page. ### Historical SOW Query AMPS allows you to configure a SOW topic to retain the historical state of the SOW, on a configurable granularity. You can then query for the state of the SOW at a point in time, and retrieve results from the saved state. ### Utilities AMPS provides several utilities that are not essential to message processing, but can be helpful in troubleshooting or tuning an AMPS instance. The _User Guide_ and _Utility Reference_ describe these utilities in detail. The utilities include: * `spark` - a command-line client, which is a useful tool for diagnostics, such as checking the contents of a SOW topic. The `spark` client can also be used for simple scripting to run queries, place subscriptions and publish data. * `ampserr` - used to expand and examine error messages that may be observed in the logs. This utility allows a user to input a specific error code, or a class of error codes, examine the error message in more detail, and where applicable, view known solutions to similar issues. * `amps-grep` - used to search the AMPS errors and events log _or_ AMPS journal files to quickly locate items of interest. The [AMPS User Guide](../amps-user-guide/) includes information on the utility, including command-line templates for common searches in the [Find Information in Error Log or Transaction Log ](../amps-user-guide/utilities/amps\_grep)topic. * `amps_sow_dump` - used to inspect the contents of a SOW topic store. * `amps_journal_dump` - used to examine the contents of an AMPS journal file during debugging and program tuning. More information about each of these utilities, including usage and examples, can be found in the [Utilities](../amps-user-guide/utilities) chapter of the [AMPS User Guide](../amps-user-guide/). ### Monitoring Interface AMPS provides a monitoring interface which contains information about the state of the host system (CPU, memory, disk and network) as well as statistics about the state of the AMPS instance it is monitoring (clients, SOW state, Journal state and more). AMPS provides this information through a RESTful interface for ease of integration into existing enterprise monitoring systems. AMPS can also record statistics in a persistent SQLite database, which can be queried using the standard SQLite toolset. More information about the monitoring system provided in AMPS can be found in the [Monitoring AMPS](../amps-user-guide/monitoring) chapter of the [AMPS User Guide](../amps-user-guide/). The [AMPS Monitoring Guide](../amps-monitoring-guide/) contains information about the statistics available and how the monitoring statistics are recorded in the statistics database. ### High Availability The [Replicating Messages Between Instances](../amps-user-guide/replication) chapter and the [Highly Available AMPS Installations](../amps-user-guide/ha) chapter in the [AMPS User Guide](../amps-user-guide/) explains the powerful high availability features that AMPS provides. This chapter describes how to use the AMPS transaction log and AMPS replication to provide failover strategies and high availability guarantees. To provide high availability and failover, AMPS provides **replication** of topics between instances. A set of features in the AMPS clients work with the AMPS server to provide reliable publishing and resumable subscriptions. The **transaction log**, described earlier, is the foundation of AMPS replication. AMPS replication is designed to ensure that the messages in the transaction log of one AMPS instance are also in the transaction log of another AMPS instance. The AMPS client libraries provide optional reliable publication functionality, using a local store to retain messages, until the AMPS server notifies the publisher that the message has met the persistence guarantees that the server is configured for. Typically, the persistence guarantee is configured to be the point at which the message has been confirmed to have been written to both instances in a high availability pair, but stronger guarantees (such as also having been written to an offsite disaster recovery instance, or having been written to an instance in another region) can also be configured. The AMPS approach to high availability is based around the principle that each topic is a stream of messages. The basic concepts behind AMPS replication include: * AMPS replication is always treated as a message stream from a source instance, which pushes messages, to a destination instance, which receives messages. In many cases, an application will use two-way replication so that instances contain the same messages. For two-way replication, replication is configured in both directions. * The intent of AMPS replication is to ensure that every replicated message that the source instance is responsible for replicating has been recorded in the transaction log of the destination instance. * AMPS replicates sequences of commands (that is, each individual publish or delete) rather than the cumulative state of a set of publishes. * For a given data source (that is, an individual publisher), AMPS guarantees that it will preserve the order in which that data source provided messages. The order must be consistent both within an instance and between instances. For full details on AMPS replication, including recommendations and best practice advice, see the [Replicating Messages Between Instances](../amps-user-guide/replication) chapter and the [Highly Available AMPS Installations](../amps-user-guide/ha) chapter in the [AMPS User Guide](../amps-user-guide/). --- # Scenario and Feature Reference AMPS offers a wide array of messaging features to solve a variety of messaging scenarios. This section presents some basic mappings between common messaging scenarios and the AMPS features that support those scenarios. Of course, this list is just a sampling of the types of applications that use AMPS. | Scenario | AMPS Feature(s) | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Simple, low-latency publish and subscribe (many to many messaging) with no need to persist messages. | Ad hoc [Publish and Subscribe](/docs/amps-user-guide/pub-sub) | | Publish and subscribe with a replayable audit trail. | [Transaction Log and Bookmark Subscription](/docs/amps-user-guide/txlog/transaction-log-basics) | | Snapshot of the current state of a set of messages (for example, graphing the elapsed time for all pending orders). | [State of the World (SOW)](/docs/amps-user-guide/sow) | | Creating a view server that aggregates information about a high-velocity data feed for reporting. |

    [State of the World (SOW)](/docs/amps-user-guide/sow)

    [Views and Aggregation](/docs/amps-user-guide/views)

    [Transaction Log](/docs/amps-user-guide/txlog) and [Replication](/docs/amps-user-guide/replication)

    | | Snapshot of the current state of a set of messages followed by updates to those messages (for example, showing the current status of a set of orders when a UI starts and then showing real-time updates to those messages). |

    [State of the World (SOW)](/docs/amps-user-guide/sow)

    [SOW and Subscribe](/docs/amps-user-guide/sow-queries/query-and-subscribe) from client application

    | | Ensuring that a given message is processed once, by a single subscriber (for example, a workload distribution system). |

    [Message Queues](/docs/amps-user-guide/queues) and [Transaction Log](/docs/amps-user-guide/txlog/configuring-a-transaction-log)

    (Queues use the Transaction Log)

    | | Replaying messages from a point in time. | [Transaction Log and Bookmark Subscription](/docs/amps-user-guide/txlog/transaction-log-basics) | | Transforming messages as they are published to AMPS. | [State of the World (SOW)](/docs/amps-user-guide/sow) and [Enrichment](/docs/amps-user-guide/enrichment) | | Producing aggregate data for a stream of messages. | [State of the World (SOW)](/docs/amps-user-guide/sow) and [Views](/docs/amps-user-guide/views) or [Aggregated Subscriptions](/docs/amps-user-guide/views/aggregated-subscriptions) | | Coordinating work across a set of independent workers who are each assigned discrete tasks. | [Message Queues](/docs/amps-user-guide/queues) and [Transaction Log](/docs/amps-user-guide/txlog/configuring-a-transaction-log) | | Dividing work among a set of workers who each update a portion of a record. | [State of the World (SOW)](/docs/amps-user-guide/sow) and [Delta Publish](/docs/amps-user-guide/delta-publish) | | Providing highly available messaging with multiple servers providing failover. | [Transaction Log](/docs/amps-user-guide/txlog) and [Replication](/docs/amps-user-guide/replication) | The scenarios above describe just a few of the more common scenarios in which AMPS is used. For messaging scenarios that aren't described above, contact 60East at [http://crankuptheamps.com/support](/support) for advice and guidance. --- # Recovery Strategies The AMPS server and the AMPS client libraries provide various options for recovering and resuming subscriptions. Use this cross-reference to choose the recovery strategy that best matches the needs of your application. | Scenario | AMPS Feature(s) | | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Automatically recover subscription without replaying missed messages. | [HAClient](/docs/amps-user-guide/ha) / [Subscribe](/docs/amps-user-guide/pub-sub) | | Recover subscription and replay any messages missed while application is offline. | [HAClient](/docs/amps-user-guide/ha) / [Transaction Log](/docs/amps-user-guide/txlog) / Bookmark Subscription (refer to the relevant Client Guide) / Bookmark Store (refer to the relevant Client Guide) | | Recover subscription, get current state of a set of messages upon recovery and receive updates to that state. | [HAClient](/docs/amps-user-guide/ha) / [State of the World (SOW)](/docs/amps-user-guide/sow) / [SOW and Subscribe](/docs/amps-user-guide/sow-queries/query-and-subscribe) command | The scenarios above describe just a few of the most common recovery scenarios for a subscription. For recovery scenarios that aren't described above, contact 60East at [http://crankuptheamps.com/support](/support) for advice and guidance. --- # Getting Started with AMPS ## Crank Up the AMPS This chapter is for users who are new to AMPS and want to quickly get a simple instance of AMPS running. This chapter will describe how to install AMPS on a Linux system, describe the layout of the AMPS distribution, and use the included `spark` command line AMPS client to send and receive a simple message. If you are on a Windows system without easy access to a Linux installation, a section at the end of the chapter includes information on configuring a Linux virtual machine. This section covers the following topics: | Topic | Description | | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | [Installing AMPS](/docs/intro-guide/getting_started/installation) | Basic information on installing the AMPS server. | | [Starting AMPS](/docs/intro-guide/starting) | Information on running the server, including a description of the command line options for the server. | | [JSON Messages - A Quick Primer](/docs/intro-guide/getting_started/json_primer) | A brief description of the JSON format, which is used for the examples in this introduction. | | [spark: the AMPS command-line client](/docs/intro-guide/getting_started/spark) | An introduction to the basic command-line client included in the AMPS distribution. | | [Evaluating AMPS on Windows or MacOS](/docs/intro-guide/getting_started/windows_installation) | AMPS runs on x64 Linux. This section describes how to install a development or evaluation system on Windows or MacOS | | [Galvanometer and RESTful Statistics](/docs/intro-guide/admin_view) | This section describes the monitoring interfaces provided for the AMPS server. | --- # Installing AMPS On the 60East website at [http://www.crankuptheamps.com/evaluate](http://www.crankuptheamps.com/evaluate) the current release of AMPS is available for evaluation download. To get started, download the Linux installation to a directory on your Linux system. Installing AMPS is simply a matter of unpacking the distribution. The distribution contains the complete set of libraries and dependencies needed to run the AMPS server on a typical Linux server distribution. No additional software or packages are necessary for the server itself. To install AMPS, unpack the distribution in the directory where you want the binaries and libraries to be stored. For the remainder of this guide, the installation directory will be referred to as `$AMPSDIR` as if an environment variable with that name was set to the correct path. Within `$AMPSDIR` are the following sub-directories: | Directory | Description | | ------------- | ---------------------------------------- | | bin | AMPS engine binaries and utilities | | docs | Documentation | | lib | Library dependencies | | sdk | Include files for the AMPS extension API | :::info AMPS client libraries are available as a separate download from the AMPS web site. See the AMPS developer page at [http://www.crankuptheamps.com/develop](http://www.crankuptheamps.com/develop) to download the latest libraries. ::: --- # JSON Messages - A Quick Primer AMPS includes support for a wide variety of message types, as well as the ability to develop custom message types and to send binary payloads. This section focuses on JSON as the main message type used for samples in this guide. We use JSON for the guide because the format is simple, easily readable, and already in use in many environments. JSON format is a simple, standardized message format. JSON has two basic constructs: * Objects that consist of key / value pairs * Arrays of values JSON supports hierarchical construction: the value for a key can be a single value, an array of values, or another set of key/value pairs. For example, the following JSON message includes two nested sets of key value pairs. Notice that a key only needs to be unique within each set of values -- the `name` value for the ship does not conflict with the `name` value for the character. ```javascript showLineNumbers { "id" : 73, "character" : { "name" : "Han Solo", "occupation" : "smuggler", "ship" : { "name" : "Millennium Falcon", "speed" : ".5 past light speed", "cargo" : [ "widgets", "baskets", "spice"] } } } ``` Many AMPS applications use JSON as the payload. In addition, the `amps` protocol used to send commands to AMPS represents commands in a simplified subset of JSON. For example, a publish command might look like: ```javascript {"c":"publish","t":"test-topic"}{ "id" : 1, "message" : "Hello, World!" } ``` The command to AMPS, using the `amps` protocol, can be treated as a JSON document which contains the header information for AMPS -- in this case, a `publish` to the topic `test-topic`. The header is followed by the message body, the payload of the command. While the `amps` protocol is implemented as a subset of JSON, you can use any message type with the `amps` protocol. The header for the command will still be JSON, while the body can be in the message type of your choice, as in the sample below, which publishes to an XML topic: ```json { "c":"publish","t":"xml-topic"}1Hello, world! ``` The AMPS client libraries create and parse AMPS headers. For example, the `publish` method in the AMPS client libraries creates the appropriate header for a publish command based on the provided parameters. Your applications use the `Message` and `Command` interfaces of the AMPS client libraries to work with the AMPS headers. There is no need for your application to parse or serialize the AMPS headers directly. :::info The AMPS client libraries handle creating and parsing AMPS headers. They do not parse or interpret the payload data on a received `Message`, instead the payload is returned as a sequence of bytes (or as a string). There's one exception to this: the JavaScript client can optionally deserialize JSON messages into objects. ::: --- # spark: the AMPS command-line client ## Interacting with AMPS Using Spark AMPS provides the [`spark`](../../amps-user-guide/utilities/spark) utility as a command line interface to interacting with an AMPS server. `spark` provides many of the capabilities of the AMPS client libraries through this interface. The utility lets you execute AMPS commands from the command line. `spark` is a Java application, and requires Java runtime environment version 1.7 or later on the system. `spark` is most commonly used for ad hoc testing or simple maintenance tasks. For more complicated tasks or more sophisticated maintenance, 60East recommends using one of the client libraries (such as the AMPS Python Client). To test `spark` with the sample configuration, run the following command: ``` $ $AMPSDIR/bin/spark ping -server localhost:9007 -type json ``` This command tests connectivity to the AMPS server running at port `9007` on the local system. It confirms that the server is listening on that port using the default protocol for AMPS and accepts JSON messages on that port. The command should produce output like the following: ``` Successfully connected to tcp://username@localhost:9007/amps/json ``` You can read more about [`spark`](../../amps-user-guide/utilities/spark) and other useful tools for troubleshooting AMPS in the [Utilities](../../amps-user-guide/utilities) chapter of the[ AMPS User Guide](../../amps-user-guide/). :::info It's important to keep in mind that `spark` only provides basic functionality -- that is, operations that don't require any particular application logic or special handling. This guide uses `spark` for examples where possible, but some features of AMPS (for example, setting certain headers on messages published to AMPS) are only available through the AMPS client libraries. ::: --- # Evaluating AMPS on Windows or MacOS The AMPS server runs on 64-bit Linux operating systems. If you do not have access to a Linux system or a recent version of Windows, 60East recommends creating a Linux virtual machine to host the instance of AMPS. This is a convenient option for development systems and allows you to easily experiment with different AMPS configurations on a dedicated system. This section provides general information for creating a virtual machine image for use as a local development or evaluation environment. This section assumes that you are familiar with Linux and the virtualization program you will be working with. It focuses on information specific to AMPS. ### Using Windows Subsystem for Linux 2 If your development system is running a recent version of Windows, then Windows Subsystem for Linux 2 is a good option for developing with AMPS. Getting AMPS running is simply a matter of starting a Linux shell, downloading AMPS, and following the directions for Linux. Notice that Windows Subsystem for Linux 2 does not provide access to some of the functionality that the AMPS server expects: in particular, the AMPS NUMA subsystem may not be able to determine the physical processor layout and may report warnings on startup. Nevertheless, this can be a very good option for doing AMPS evaluation and development on a Windows system. ### Creating a Virtual Machine Image When creating the virtual machine image, 60East recommends the following parameters: * x64 processor * At least 4GB of memory allocated to the virtual machine * Minimum of 120GB drive space (most will be consumed by the operating system image) * At least 2 virtual processors AMPS itself can run with less memory, processor, and disk capacity than recommended here. However, these settings will typically provide reasonable performance and enough capacity to do basic development work. #### Virtual Box Settings When installing AMPS on Virtual Box, 60East strongly recommends setting the network hardware emulation to use the **Paravirtualized network adapter (virtio-net)**. For recent versions of Linux, performance is dramatically improved (even over the loopback interface) when using this setting. ### Choosing a Linux Distribution AMPS runs well on any Linux distribution that meets the basic requirements. The Ubuntu Linux distribution is a good choice, and is frequently used by both customers and the 60East developers as a development workstation environment. Visit [https://www.ubuntu.com/download](https://www.ubuntu.com/download) to download the latest released version of Ubuntu. Whichever distribution you choose, 60East recommends that you download the .iso file and use that file to install the operating system. ### Installing the Linux Distribution AMPS itself doesn't require anything beyond a basic operating system distribution. For the best experience while you are evaluating and getting to know AMPS, 60East recommends that you choose a profile optimized for software development or desktop use. Select the following additional packages if your distribution does not already install them: * Python 2.6/2.7 or 3. The utility scripts in the AMPS distribution require Python. * Java runtime environment (1.7 or more recent). The `spark` command line AMPS client is written in Java, and requires a JRE. This guide assumes that you have a JRE available, and presents examples using `spark`. * g++, gdb, and your IDE of choice if you will be developing C++ applications with AMPS. * A web browser such as Firefox or Google Chrome ### Filesystem Considerations AMPS is designed and tested to use a Linux-based filesystem such as `ext4` or filesystems that provide full native Linux filesystem semantics (for example, using `nfs` mounted filesystems for testing or archival purposes). Mounting another type of filesystem (for example, an `NTFS` volume) in a VM, container, or WSL 2 may cause failures or unexpected results, since that approach may not provide all of the filesystem operations that AMPS uses. When using a container, VM, or WSL2, make sure that AMPS and the files that AMPS creates are hosted on filesystems that support Linux file operations, in particular, that a process running under the Linux environment can memory map files hosted on that filesystem. ### Next Steps Once you have created the virtual machine image and installed your Linux distribution of choice, you can install and start AMPS as described in [Installing AMPS](installation). --- # Introduction to AMPS Welcome to the Advanced Message Processing System (AMPS) from 60East Technologies! AMPS is designed to help you quickly and easily develop and deploy data-intensive applications, with demanding requirements, for low latency and high performance. AMPS takes a nontraditional approach to messaging, storage, and analytics that is designed from the ground up for streaming data and highly-parallelized multicore systems. AMPS isn't a traditional database or messaging product. This guide presents a brief introduction to help you understand the capabilities of AMPS and how AMPS operates. AMPS is widely used for applications such as: * Trade plant operations (including backtesting and historical analysis) * Risk calculations * Elastic worker farms * View servers * Message flow integration and "shock absorbers" AMPS combines a set of capabilities that cut across traditional boundaries between applications that work with data. AMPS is built around a fast messaging engine that supports both publish and subscribe (fan-out) and queued (competitive consumption) message delivery with full content filtering. AMPS also provides an integrated database that applications can use as a current value cache, key/value document store, and fully queryable database -- or all of these at once. With this database, AMPS includes a built-in aggregation and analytics engine for near-real time analysis of streaming data, including aggregation across multiple topics or message formats. Integrated message logging provides the ability to record and replay streams of messages with full fidelity. AMPS is designed from the ground up for enterprise deployment at scale. AMPS provides an extensive set of high-availability features, including integrated replication and automatic failover and recovery for applications. Detailed monitoring and statistics are included from a RESTful interface for ease of data collection and integration with enterprise monitoring and management systems. Authentication and entitlement capability applies to every operation in AMPS, for fine-grained control over permissions to meet enterprise policy and regulatory requirements. Access to data can be controlled at a topic level, at a message level (content-based security), or at the level of individual fields within a message (limiting the fields a given user has access to view). 60East developed AMPS to serve the needs of some of the most demanding data-intensive applications on the planet. The feature set and capabilities have been engineered for the highest levels of performance, designed for ease of use, and proven in production applications worldwide. ### Getting to Know AMPS AMPS is designed to be a developer-friendly product. 60East recommends reading about AMPS with a running instance of AMPS and your development environment of choice available. Although 60East makes every effort to clearly describe how AMPS works, there is no substitute for seeing exactly how a running instance behaves (not to mention the advantages of being able to try out ideas or do quick prototyping while you read). The table below lists the main parts of the AMPS documentation: | **Title** | **Description** | | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Introduction to AMPS](intro) |

    Overview of AMPS functionality.

    This is a good place to start if you are new to AMPS or if you are familiar with older versions of AMPS.

    | | [AMPS User Guide](../amps-user-guide/) |

    Guide to AMPS functionality and configuration.

    This guide provides detailed descriptions of AMPS features, best practices, and in-depth explanations of how AMPS works. It also includes a complete reference to the configuration file syntax, accepted values, and examples for configuring commonly-used options.

    | | [AMPS Monitoring Guide](../amps-monitoring-guide/) |

    Guide to the RESTful monitoring interface and the AMPS statistics database.

    Use this guide when creating a monitoring strategy or when collecting statistics about an instance.

    | | [AMPS Command Reference](../amps-command-reference/) | Description of the commands sent from an AMPS client to the AMPS server and responses from the server. | | [Client Language Developer Guides](https://www.crankuptheamps.com/develop/) | Guide to using a client library to work with AMPS. | This guide uses the `spark` command line utility for basic examples for simplicity, although a production installation would use an application to perform these functions. --- # Next Steps ## Learning More Now that you understand the basics of how AMPS works, you have two potential paths forward in your usage of the product: * On one path, you may want to learn how to configure, deploy, and administer your own instance of AMPS. For this path, see the [AMPS User Guide](../amps-user-guide/), which provides complete information for system administrators who are responsible for the deployment, availability and management of data to other users. * Alternatively, you may need to develop an application to work with AMPS, using one of the _Developer Guides_ for Java, Python, C++, or C#. For this path, visit the developer page at [https://www.crankuptheamps.com/develop/](https://www.crankuptheamps.com/develop/) to download one of the evaluation kits. The following sections provide more information about each of these paths and also briefly describe some use cases for AMPS. #### Operation and Deployment In preparing to deploy your instance of AMPS, you must size your host environment according to multiple dimensions: memory, storage, CPU, and network. The [Operation and Deployment](../amps-user-guide/operation) chapter in the [AMPS User Guide](../amps-user-guide/) provides guidelines and best practices for configuring the host environment. The chapter also specifies recommended settings for running AMPS on a Linux operating system. When preparing to deploy AMPS, consult the _Deployment Checklist_ whitepaper, available on the 60East website. Advice on preparing to deploy AMPS in production is available under your support agreement from [http://crankuptheamps.com/support](/support). 60East provides review of configuration and application architecture on demand, and new deployments are especially encouraged to take advantage of this review. #### Application Development Each language-specific _Development Guide_ explains how to install, configure, and develop applications that use AMPS. In order to develop applications using an AMPS client, you must understand the basic concepts of AMPS, such as _topics_, _subscriptions_, _messages_ and _SOW_. You will also need an installed and running AMPS server to use the product. Although you can type and compile programs that use AMPS without a running server, you will get the most benefit by running the programs against a working server. Visit the 60East website at [http://www.crankuptheamps.com/evaluate](http://www.crankuptheamps.com/evaluate) for an evaluation version of AMPS. --- # Overview of AMPS ## AMPS Concepts This section describes the overall AMPS approach and the features AMPS provides. The AMPS messaging system is designed around a few simple principles: * **Parallelize work** and minimize waits and blocking to take full advantage of modern multisocket, multicore systems. * **Eliminate redundant or unused work** by only performing tasks that are necessary to provide the functionality requested by a given operation. * **Reduce or eliminate cross-system coordination** by solving the full range of data delivery and storage problems commonly faced by data-intensive applications. * **Provide a concise and flexible command set** for intuitive use. * **Provide multiple delivery paradigms** supporting both publish-subscribe delivery (many to many) and message queues (single consumption of a message) as well as the ability to query the state of a topic at a point in time. * **Stay application-focused** to provide exactly the capabilities that are heavily used in demanding high-performance applications. * **Stay hardware aware and build for the future** by engineering for next-generation commodity hardware and designing AMPS to fully exploit non-uniform memory access (NUMA), flash-based storage, and high-bandwidth networking. These concepts are the foundation of how AMPS works and are helpful for understanding how to best use AMPS. To best take advantage of AMPS, applications typically use the built-in features of AMPS rather than their traditional equivalents. For example, rather than keeping a separate, independent record of each message published to AMPS for audit purposes, applications most often use one of the data persistence features in AMPS. This speeds development and simplifies deployment by eliminating integration effort, and also solves potential correctness issues which could be caused if messages in persistent storage become inconsistent with the messages provided through the messaging system. With AMPS, the messaging system itself can contain a fully-queryable and replayable record of the system. As another example, AMPS provides integrated replication rather than relying on an external process. AMPS replication is aware of the format and semantics of the transaction log, the configuration of the instance, and the commands sent by publishers. This integration allows AMPS to very efficiently provide a full-fidelity message stream and to provide "self-healing" for an instance to catch up when it has been offline. Further, the message store used for replication (the AMPS transaction log) is also used for durable subscriptions and message replay. Designing and implementing these features together reduces complexity, storage requirements, and overhead to enable both capabilities. Within the AMPS server, the implementation uses a sophisticated parallelized algorithm for storage and replay that reduces overall latency and prevents slow consumers or replication destinations from affecting faster consumers. The overall result is to simplify configuration and application development, provide strong consistency and reliability guarantees, and provide the highest possible level of performance. As a final example, rather than requiring a complex topic structure, requiring applications to oversubscribe and discard messages, AMPS provides both topic filtering and content filtering. AMPS includes an expressive filter grammar to provide precise selection of messages of interest to an individual subscriber. AMPS provides this capability to fully decouple publishers and subscribers. With AMPS, there's no need to maintain and administer a granular topic structure. Precise filtering and routing improves both network and processor utilization by providing only actionable messages to a subscriber. Likewise, for many applications, there is no need for a publisher to be aware of the processing performed by the subscriber or by AMPS itself. The examples above highlight just a few of the capabilities AMPS provides and how the AMPS approach simplifies development, administration, and operations while providing reliability and performance benefits over conventional systems. ## Feature Highlights Some of the highlights of AMPS features include: * Topic based publish and subscribe, including full support for regular expressions to specify topic names. * Content filtering based on XPath identifiers (to specify the fields of a message) and SQL-92 (to form a predicate), with added support for Perl-Compatible Regular Expressions (PCRE2). * Message queues including content filtering for both publishers and subscribers, configurable strategies for delivery fairness, and truly distributed queues that can efficiently enforce queue semantics and delivery guarantees across a replicated network of AMPS instances. * Content-aware messaging support for a wide range of message types, including standard formats such as JSON, FIX, MessagePack, XML, Google Protocol Buffers, and BSON. AMPS also supports simple key/value pairs in FIX format (called NVFIX to emphasize that the format uses name/value pairs rather than FIX tags), and a high-performance binary protocol called BFlat. AMPS also supports uninterpreted binary messages, and allows you to create composite message types from existing types to easily combine messages of different types in a single payload. * An integrated database and record-aware current value storage (called State of the World, or SOW), with optional historical query capability. * Historical replay of message streams, including the ability to preserve the total message ordering across independent topics. * Integrated replication and high availability, including automatic resynchronization for instances that fail over. * Aggregation and Complex Event Processing (CEP), including the ability to aggregate information across different message streams and message streams of different formats. * Advanced messaging capabilities such as atomic query-and-subscribe, incremental (delta) updates, and out-of-focus notifications that tell a subscription when a record no longer matches. * Built in statistics and monitoring, with data provided via a standard RESTful interface. * Integrated authentication and entitlement across all AMPS features. * Actions for automating AMPS functionality, including both routine maintenance tasks and dataflow-aware processing (such as alerting in response to slowdowns or invalid data). * Client development kits for popular programming languages such as Java, C#/.NET, C++, Python, JavaScript, and Go. * Extensibility API in the AMPS server for adding message types, extending the functions available to the AMPS query language, adding new actions, integrating with enterprise authentication and entitlement systems, and more. This guide provides an overview of the most commonly used functionality of AMPS, but it is not intended to cover all of the features of AMPS or provide an exhaustive discussion of any individual feature. As mentioned earlier, the [AMPS User Guide](../amps-user-guide/) provides full details about AMPS features. --- # AMPS Basics: Subscribe and Publish to Topics ## Topics, Publish and Subscribe The simplest way to use AMPS is as a low-latency publish and subscribe messaging system. Publish and subscribe messaging is at the heart of AMPS, and all of the other features of AMPS build on this foundation. In a publish and subscribe messaging system, publishers send messages without necessarily knowing which subscribers will receive the message. This decouples publishers from subscribers for maximum flexibility. While publishers do not need specific knowledge about the subscribers, publishers are responsible for adding information to the message so that subscribers know which messages are of interest. In publish and subscribe messaging systems, including AMPS, publishers send messages to a specific topic. The topic most often indicates the type of message, and is a way for the subscriber to locate the messages of interest. For example, in an order processing system, a publisher might publish messages to an `ORDERS` topic. Subscribers that need to receive orders then subscribe to the `ORDERS` topic, and receive messages that are sent to that topic. Each message in AMPS is published to a specific topic. The publisher chooses the topic when the message is published, and subscribers can receive messages from that topic. Unlike many messaging systems, AMPS provides an additional layer of selectivity for subscribers. Rather than receiving every message from a given topic, an AMPS subscriber can use content filtering to receive only the messages that the subscriber needs to process. Content filtering provides several advantages. First, being more selective about the messages delivered to the subscriber makes better use of bandwidth between AMPS and the subscriber, since the subscriber only receives messages that are of interest. Subscriber code is easier to write and more efficient because the subscriber is guaranteed that the messages received have the values requested. Further, because the subscriber chooses which messages to receive, content filtering makes publishers and subscribers less tightly-coupled. A publisher does not need to know what fields are important to a subscriber, or whether a field that was previously unused is now important. The diagram below shows the basic concept of publish and subscribe messaging: In the diagram above, there is a Publisher sending AMPS a message to the `ORDERS` topic. The message being sent contains information on Ticker `IBM` with a Price of `125`. Both of these fields are contained within the message payload itself (i.e., the message content). AMPS routes the message to Subscriber 1 because it is subscribing to all messages on the `ORDERS` topic. Similarly, AMPS routes the message to Subscriber 2 because it is subscribed to any messages having the Ticker equal to `IBM`. Subscriber 3 is looking for a different Ticker value and is not sent the message. ### Topic Configuration for Basic Pub/Sub Unlike many messaging systems, AMPS does not require any configuration for simple publish and subscribe. For this basic functionality, there is no need to declare topics in advance. Since there is no need to declare these topics, topics that provide basic publish and subscribe are often referred to as _ad hoc_ topics in AMPS. It is valid for a publisher to publish to any topic, whether or not that topic has been previously configured. Likewise, it is valid for a subscriber to subscribe to any topic, whether or not a message has been previously published to that topic or whether the topic appears in a configuration file. Features that rely on persisted messages, however, are not available without configuration for a topic. Every topic in AMPS has a specific message type. Publishers and subscribers don't need to explicitly set the message type when publishing to or subscribing to a topic. Each connection to AMPS specifies the message type to be used for that connection -- either implicitly, by connecting to a port that provides a specific message type, or explicitly (when connecting to a port that can provide multiple message types). AMPS allows topics that use different message types to have the same name, but considers them to be different topics. Messages published to an XML topic named `quotes` will not be delivered to subscriptions to a JSON topic named `quotes`. ### Regular Expression Subscriptions AMPS allows subscribers to provide a regular expression that defines a set of topics rather than a literal topic name. This further decouples publishers and subscribers. It's important to remember that each message is published to a specific topic. Regular expressions are only applicable to subscriptions: a publisher should not use regular expressions in the topic when publishing messages. #### Spark: Basic Publish and Subscribe Example Here's how to use `spark` to send and receive a message from AMPS. The example assumes that you're using the sample configuration file produced by the AMPS server, and that you are running `spark` on the same system that AMPS is running on. First, start a subscriber: 1. Open a terminal in your Linux environment. 2. Use the following command (with `AMPS_DIR` set to the directory where you installed AMPS) to start a subscription: ```bash $ $AMPS_DIR/bin/spark subscribe -server localhost:9007 \ -type json -topic test ``` This command starts a subscription to the JSON topic `test`. 3. `spark` will connect to AMPS, logon using default credentials (the current username and an empty password) and enter the subscription. Unless there are errors, the command will produce no output until a message arrives. 4. Leave this terminal running. When you publish a message to the test topic, `spark` will print the message in this terminal. Next, publish a message to the same topic: 1. Open a new terminal in your Linux environment. 2. Use the following command (with `AMPS_DIR` set to the directory where you installed AMPS) to send a single message to AMPS: ```bash $ echo '{"note":"Crank it up!"}' | \ $AMPS_DIR/bin/spark publish -server localhost:9007 \ -type json -topic test ``` 3. As with the subscriber sample, `spark` automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). With the publish command, `spark` reads the message from standard input and publishes the message to the JSON topic `test`. The command produces output similar to the following line (the rate calculation will likely be different): ```bash total messages published: 1 (333.33/s) ``` 4. When the publisher sends the message, the _subscriber_ should receive the message and produce the following output: ```bash {"note":"Crank it up!"} ``` Congratulations! You've just sent your first message through AMPS. Although this example is simple and relies on default behavior, the sample demonstrates some important AMPS concepts: 1. As mentioned earlier, there is no need to preconfigure simple publish/subscribe topics. Since the default configuration file doesn't specify any settings for the JSON topic `test`, AMPS knows to treat the topic as a simple pub/sub topic. Also, since the `spark` commands specify the JSON message type when connecting to the server, the topic is a JSON topic. 2. For simple publish and subscribe topics, AMPS delivers the message verbatim to the subscriber. AMPS doesn't interpret or normalize the message. In fact, AMPS doesn't even parse the message unless there's a need to. With this configuration and this subscription, there's no need for AMPS to parse the message, so no parsing happens. 3. The `spark` program connects to AMPS and logs on to AMPS before sending any commands. All AMPS installations include authentication and entitlement. By default, AMPS loads an authentication and entitlement policy (implemented as an AMPS _module_) that requires a logon, but accepts any username and password as credentials. This policy is intended for evaluating, testing and development purposes. More information on securing an AMPS instance is available in the _User Guide_. For this example, the important point is to be aware that an AMPS instance always has a security policy and that policy was at work even in this simple example. The default behavior for `spark` works with the default policy for AMPS. ### Content Filters In the Basic Sub-Pub example, the subscriber requested all messages on the JSON `test` topic. AMPS includes expressive, flexible and extensible content filtering that allows subscribers to specify exactly the messages that they want to receive. Content filtering is one of the most useful features of AMPS. When subscribers use content filtering, publishers can be completely independent of subscribers. The publisher does not need to know which parts of a message are important to subscribers. Subscribers can precisely declare the content that they are interested in, so they only receive relevant messages. Publishers do not need to be updated when subscribers add additional criteria or when new subscribers come online. You can think of content filtering as adding a WHERE clause to the subscription. Like a WHERE clause, AMPS returns only matching messages. AMPS content filters use a combination of XPath identifiers to locate a value within a message and SQL-92 operators for comparing those values. For example, given a JSON message like: ```javascript {"note":"Hello, World!"} ``` The following content filters would match the message: ```bash /note = 'Hello, World!' ``` This filter uses the equality operator, `=`, to compare the `/note` field in the message with an exact match for the string. ```bash /note LIKE '(?i)world' ``` The `LIKE` operator uses Perl Compatible Regular Expressions to match a field. In this case, the regular expression matches any string that contains `world`, using case-insensitive matching. ```bash /note BEGINS WITH 'Hello' ``` The AMPS `BEGINS WITH` operator matches any string that begins with the exact sequence of characters provided. The [AMPS User Guide](../amps-user-guide/) sections on [AMPS Expressions](../amps-user-guide/amps-expressions) and [AMPS Functions](../amps-user-guide/amps-functions) have full details on the expression language used in AMPS. #### Spark: Subscription with Content Filter Here's how to use `spark` to subscribe using a content filter. The example assumes that you're using the sample configuration file produced by the AMPS server and that you are running `spark` on the same system that AMPS is running on. First, start a subscriber: 1. Open a terminal in your Linux environment. 2. Use the following command (with `AMPS_DIR` set to the directory where you installed AMPS) to start a subscription: ```bash $ $AMPS_DIR/bin/spark subscribe -server localhost:9007 \ -type json -topic test \ -filter "/note LIKE '(?i)sample'" ``` This command starts a subscription to the JSON topic `test`. This subscription will only return messages where the filter matches. 3. `spark` will connect to AMPS, logon using default credentials (the current username and an empty password) and enter the subscription. Unless there are errors, the command will produce no output until a message arrives. 4. Leave this terminal running. When you publish a message to the test topic that matches the filter, `spark` will print the message in this terminal. Next, publish messages to the subscriber: 1. Open a new terminal in your Linux environment. 2. Use the following command (with `AMPS_DIR` set to the directory where you installed AMPS) to publish a message to AMPS. This message matches the filter: ```bash $ echo '{"note":"Filter sample!"}' | \ $AMPS_DIR/bin/spark publish -server localhost:9007 \ -type json -topic test ``` 3. Use the following command (with `AMPS_DIR` set to the directory where you installed AMPS) to publish a message to AMPS. This message does not match the filter: ```bash $ echo '{"note":"Not a match. Sorry."}' | \ $AMPS_DIR/bin/spark publish -server localhost:9007 \ -type json -topic test ``` 4. Each time you run `spark`, it automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). With each publish command, `spark` reads the message from standard input and publishes the message to the JSON topic `test`. Each of the commands above produces output similar to the following line (the rate calculation will likely be different): ```bash total messages published: 1 (333.33/s) ``` 5. When the publisher sends a message that matches the filter, the _subscriber_ should receive the message and produce the following output: ```bash {"note":"Filter sample!"} ``` ### Further Reading AMPS provides high-performance publish and subscribe messaging that requires minimal configuration and provides high performance, flexible publishing and message routing. See [Subscribe and Publish](../amps-user-guide/pub-sub) in the AMPS User Guide for a more complete discussion of subscribe and publish. The AMPS client libraries include samples of basic publish and subscribe functionality. See the client library distribution for those samples. :::info Notice that some libraries are distributed as pre-built binaries through package management systems. 60East also offers full distributions including documentation and source from the 60East website. If you've installed a pre-built library using a package manager, visit the 60East website to download the full distribution that contains the samples. ::: --- # Message Queues AMPS includes high performance queuing built on the AMPS messaging engine and the transaction log. AMPS queues combine elements of classic message queuing with the advanced messaging features of AMPS, including content filtering, aggregation and projection, and so on. AMPS queues help you easily solve some common messaging problems: * Ensuring that a message is only processed once. * Distributing tasks across workers in a fair manner. * Ensuring that a message that has been delivered is processed. * Ensuring that when a worker fails to process a message, that message is re-delivered. These uses of messaging require different behavior than the scenarios discussed in the section on [Subscribing and Publishing to Topics](pub\_sub). For basic subscribe and publish, each message is delivered to any number of subscribers. With queues, each message is fully processed by only one subscriber. While it's possible to create applications with these properties by using the other features of AMPS, message queues provide these functions built into the AMPS server for additional performance, simple administration, and ease of development. AMPS queues also allow you to: * Replicate messages between AMPS instances while preserving delivery guarantees. * Create views and aggregates based on the current contents of a queue. * Filter messages with specific content into specific queues. * Provide a subscriber _only_ messages that contain specific content. * Provide a single published message to multiple queues. * Aggregate multiple topics into a single queue. * Provide _content aware_ entitlement for security. * Provide prioritization of messages within a queue, so higher-priority messages are processed first. * Provide a synchronization point that guarantees that all messages prior to that point have been processed before messages after that point are delivered. ### How Do Queues Work? When an application needs to receive messages, there is little difference between subscribing to a queue and subscribing to a sub/pub topic. Both delivery models use the `subscribe` command, and both delivery models can provide a filter to specify messages of interest. Both types of topics provide the same message objects in the AMPS Client interfaces. Once a message is received from a queue, however, the application must let AMPS know when the message is successfully processed. This _acknowledgment_ lets AMPS know that the application is finished with the message, and has capacity to receive another message. In addition, if the queue is configured to retry messages if an application fails to process the message (`at-least-once` delivery), acknowledging the message indicates to AMPS that the message has been processed successfully and can be removed from the queue. Within AMPS, the server maintains an in-memory list of all of the messages currently available for delivery in a given queue and a list of all of the messages awaiting acknowledgment from subscribers. The messages themselves are stored in the transaction log for the instance. When a message has been successfully processed, the acknowledgment for that message is also stored in the transaction log. There is no separate storage required for a queue, since messages are recorded in the transaction log. Likewise, even when a message is removed from the queue, AMPS maintains a persisted record of that message and the acknowledgment in the transaction log. Given that the transaction log contains a full record of messages and acknowledgments, AMPS queues are persistent across server restarts, and can be replicated to other instances. (For details on replicating queues, see the [AMPS User Guide](../amps-user-guide/).) Keeping the delivery state -- that is, the queue itself -- independent of the topic in the transaction log has several other advantages. Since the set of messages in the queue is maintained separately from the physical storage for those messages, a queue in AMPS can hold messages from any number of underlying topics. Content filtering can be applied to the queue to selectively add messages to the queue: in fact, the same topic can easily be split into independent queues using content filtering. Messages to a single topic can also be included in multiple, independent queues (for example, one queue for immediate processing, and another queue for end-of-day auditing and reconciliation). AMPS includes the ability for a given consumer to declare the capacity of that consumer, using the _max\_backlog_ option on a queue subscription. This option declares the number of messages that the consumer is willing to have delivered at a given time. Using this option can improve throughput, since AMPS can ensure that a consumer is never idle waiting for a new message. This also helps AMPS to balance message delivery across consumers in the most efficient manner, as measured by the current available capacity of each consumer. For example, a consumer on a small VM might take 200 milliseconds, on average, to process a message, and might declare a `max_backlog` of 2. A consumer running on a larger physical server, in contrast, might take 50 ms on average to process a message, and might therefore declare a `max_backlog` of 8 or more. The maximum allowed backlog for a subscriber is configured for each queue, so that queues that hold large units of work can set a smaller maximum value than queues that provide smaller tasks. ### When Should I Use Queues? Queues are intended to guarantee delivery of each message to a single consumer that processes the messages. Use queues when the problem you are solving requires a message to be processed once. When you need to distribute messages to a large number of consumers, use the AMPS pub/sub delivery model. For example, a queue is a natural fit for a system that allocates work, such as a system that runs software builds or that executes financial transactions. A system that provides notifications to a large number of systems (for example, a system that distributes bids to sellers or a system that communicates status to a user interface) is a more natural fit for the pub/sub delivery model. Queues are often used to solve problems like: * Guaranteeing that a given set of work is distributed fairly across a set of workers, while each unit of work is only performed once: _A system that performs CPU-intensive calculations needs to ensure that any time a request comes into the system, it is serviced by the next available worker._ _A distributed compute grid has workers that vary widely in capacity. Each worker declares its capacity to AMPS. Workers with more capacity free receive work before workers with less free capacity, improving overall throughput for the compute grid._ * Guaranteeing that a specific message is fully processed once, regardless of the number of subscribers: _A system that processes refunds enters the refund orders into a queue. Each message is delivered to one, and only one, worker. If the worker successfully processes the message, the worker removes the message from the queue. If the worker fails, AMPS automatically delivers the message to another worker, ensuring the message is processed._ The AMPS client libraries, starting in version 5.0, are queue-aware and contain features to make it easier to work with queues and create the application behavior that you need. See the _Developer Guide_ for the client library of your choice for details on how to use these features. For further details on message queues and how they function, the chapter on [Message Queues](../amps-user-guide/queues) in the [AMPS User Guide](../amps-user-guide/) presents a more complete discussion. ### Configuration As described above, both the topic that holds the messages for the queue and the queue topic itself must be recorded in the AMPS transaction log. In addition, the queue itself must be declared in the `SOW` element of the AMPS configuration. For example, the configuration below records the topics `Work` and `WorkToDo` in the AMPS transaction log: ```xml showLineNumbers /fast-storage/journals Work json WorkToDo json ``` With these topics added to the transaction log, we can configure a `WorkToDo` queue that provides queuing for the messages in the `Work` topic. ```xml showLineNumbers WorkToDo json at-least-once Work 10 12h 5 10m ``` This declares a queue named `WorkToDo` that provides queuing for the messages in the `Work` topic. By setting the semantics to `at-least-once`, the topic is configured to redeliver a queue message to another subscriber in the event that a subscriber fails to process that message successfully. This example also provides an example of recommended configuration options to help manage message lifetime in cases where a message cannot be processed successfully, or where no consumers are available to process the message. For this queue, we provide the following options: * _Expiration_ specifies that a message will be available for, at most, 12 hours from the time it enters the queue. * _MaxDeliveries_ specifies that a given message can be delivered from the queue at most 5 times: after that, the message will be considered to be unable to be processed and removed from the queue. * _LeasePeriod_ specifies that a consumer has 10 minutes from the time the message is sent to acknowledge the message or the message will be automatically returned to the queue. AMPS also provides a mechanism for publishing expired messages to a dead-letter queue, as well as a wide variety of options for controlling delivery. Full details on these options are available in the [Configuring Queues in a SOW](/docs/amps-user-guide/queues/configuring-queues-in-sow) section of the [AMPS User Guide](../amps-user-guide/). ### Further Reading See the [Message Queues](../amps-user-guide/queues) chapter in the [AMPS User Guide](../amps-user-guide/) for a more complete discussion of message queues, including discussions of advanced features, replicated queues, and so on. The AMPS client libraries include samples for working with message queues. See the client library distribution for those samples. :::info Notice that some libraries are distributed as pre-built binaries through package management systems. 60East also offers full distributions including documentation and source from the 60East website. If you've installed a pre-built library using a package manager, visit the 60East web site to download the full distribution that contains the samples. ::: --- # Advanced Messaging and the SOW A SOW topic is the basis for many of the advanced messaging features in AMPS. While not all of these features are discussed in detail in this introduction, many features of AMPS are made possible because AMPS can retain the current state of each unique message. The advanced messaging features that the SOW enables include: - Views and aggregations over topics (including joins between topics) - Publishing incremental updates to a message (called *delta publishing* in AMPS) - Receiving incremental updates to a message (called *delta subscription* in AMPS) - Determining when a message no longer matches a filter (called *out-of-focus* notification in AMPS) - Providing a snapshot of an update to a rapidly changing record at regular intervals, rather than providing every update (called *conflation* in AMPS) These features can greatly simplify the processing an application needs to perform, making it easier to develop applications and increasing application performance. However, for a messaging system to provide these features, whenever a message arrives, the messaging system must have access to both the current message and the previous, saved state of the message. SOW topics provide that access for AMPS, and enable the advanced messaging features. --- # Configuration To create a SOW topic, you configure the topic in the SOW section of the AMPS configuration file. At a minimum, SOW topics require a `Name`, and the `MessageType` of the messages to store in the SOW. If the SOW will be persistent, a `FileName` is required. Most often, SOW topics use AMPS to generate the SOW Key, and one or more `Key` definition elements are required to specify the fields that AMPS will use for the SOW Key. For example, the following configuration file fragment specifies a SOW topic named `test-sow`. The topic stores JSON-format messages, and uses the `/id` field of incoming messages to that topic to uniquely identify messages. Records in this topic will be both maintained in memory and persisted to a file in the `./sow/` directory, so the contents of the topic will be retained across restarts of the AMPS instance. Notice that the file name specification uses the special format character `%n` as a placeholder for the topic name and message type. ```xml showLineNumbers test-sow json ./sow/%n.sow /id ``` The [Configuring Topics in a SOW](/docs/amps-user-guide/sow/configuring-topics-in-sow) section of the _AMPS User Guide_ contains full details on configuring a SOW topic. The practical examples later in this section use the configuration above. --- # How Does the SOW Work? AMPS SOW topics persist the most recent update for each message, in the same way that a relational database stores the current state of each record. For performance, AMPS SOW topics store the full content of the message verbatim rather than storing a deserialized or "shredded" version of the message. Each distinct record in a SOW topic is identified by a _SOW key_. AMPS treats the _SOW key_ for a SOW topic the same way a relational database uses the primary key for a table: each distinct _SOW key_ value is a unique message. There are several ways to create a SOW key for a message. Each topic defines one of the following strategies: * Most applications specify that AMPS will calculate a SOW key based on the content of the message. The configuration of the topic specifies the field, or fields, to be used for the key. * A topic can also be configured to require that a publisher provide a SOW key for each message when publishing the message to AMPS. This is less commonly used than determining the key based on the message content, however, since this strategy does not require any explicit configuration, AMPS will default to this strategy for identifying messages if no other strategy is specified. * AMPS also supports the ability for custom SOW key generation logic to be defined in an AMPS module, which will be invoked to generate the SOW key for each message. Although the SOW key is derived from the content of the message in many cases, the SOW key itself is metadata, distinct from the content of the message. Each record in a SOW topic has a distinct SOW key, which is stored with the record. For example, the diagram below shows how AMPS computes the SOW key for a topic named ORDERS with a key definition of `/orderId`. For each publish to the topic, AMPS uses the value of the key fields (in this case, simply `/orderId`) to compute a `SowKey`, then uses that `SowKey` to insert or update the appropriate record. --- # Queries At any point in time, applications can issue SOW queries to retrieve all of the messages that match a given topic and content filter. When a query is executed, AMPS will test each message in the SOW against the content filter specified and all messages matching the filter will be returned to the client. The topic can be a literal topic name or a regular expression pattern. For more information on issuing queries, please see [Querying the State of the World (SOW)](../../amps-user-guide/sow-queries) in the [AMPS User Guide](../../amps-user-guide/). A SOW query is atomic. Updates that occur while the query is running, or while a client is receiving results, are not returned as part of the query. ## Spark: Basic SOW Query Example Here's how to use `spark` to query the current state of an AMPS SOW topic. This example assumes that: * You have configured a topic named `test-sow` in the AMPS server of message type JSON. * The `test-sow` topic uses the `/id` field of the message to calculate the key for the topic. To retrieve the current state of the topic, an application issues the `sow` command. Unlike a subscription, which stays active until it is explicitly stopped (or the application disconnects), the `sow` command provides results for a specific point in time. Once the results are returned, the command is over. First, publish a message or two to the `test-sow` topic: 1. Open a new terminal in your Linux environment. 2. Use the following command (with `AMPS_DIR` set to the directory where you installed AMPS) to send a single message to AMPS: ```bash $ echo '{"id":1,"note":"Crank it up with a SOW!"}' | \ $AMPS_DIR/bin/spark publish -server localhost:9007 \ -type json -topic test-sow ``` 3. `spark` automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). With the `publish` command, `spark` reads the message from the standard input and publishes the message to the JSON topic `test-sow`. The command produces output similar to the following line (the rate calculation will likely be different: ```bash total messages published: 1 (333.33/s) ``` 4. When the publisher sends the message, AMPS parses the message to determine the value of the Key fields in the message, and then either inserts the message for that key, or overwrites the existing message with that key. 5. You can publish any number of messages this way. Each distinct `id` value will create a distinct record in the topic. Next, retrieve the current contents of the topic: 1. Open a new terminal in your Linux environment. 2. Use the following command (with `AMPS_DIR` set to the directory where you installed AMPS) to retrieve the contents of the topic: ```bash $ $AMPS_DIR/bin/spark sow -server localhost:9007 \ -type json -topic test-sow ``` 3. `spark` automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). `spark` then sends the `sow` command to AMPS. This command requests the current contents of the `test-sow` topic. Since the command is finished once the query is complete, `spark` will exit when the query results are complete. 4. `spark` shows the current contents of the topic. Notice that the output is strictly the message data, separated by newline characters. `spark` does not show any of the metadata for a message. --- # State of the World (SOW): The Message Database One of the core features of AMPS is the ability to persist the most recent update for each distinct message published to a topic. To enable this for a topic, you add the topic to the SOW. You can think of the SOW as a database that maintains a specific set of topics, equivalent to tables. Each distinct message published to that topic is equivalent to updating a row in the table. AMPS allows applications to query the table for the current state of the topic. SOW topics also provide full support for pub/sub messaging. Applications can use a combination of queries and subscriptions as necessary. AMPS also includes a set of commands that perform an atomic query and subscribe, allowing an application to query a SOW topic and register for updates to the topic in a single operation, without risk of missing messages or receiving duplicates. The most common uses of SOW topics include: * **Quickly loading initial state for an application**. For example, an application that tracks open orders can quickly retrieve a snapshot of all of the orders that are currently open, without having to wait for updates to the orders to be published. * **Queryable snapshots of data flows**. For example, an application that monitors telemetry data may need to quickly determine if any telemetry source has not provided an update within a given period of time. With a SOW topic, the application can run a simple query over the current state of the topic. * **NoSQL document stores**. SOW topics are frequently used as high-performance key/value stores: an application can choose to explicitly provide a key and store a document in the SOW. Documents can be efficiently retrieved by key, queried over the full content of the document, or any combination. As mentioned above, a consumer can retrieve the document and be automatically notified when the content of the document changes. SOW topics are also the foundation of many of the more advanced capabilities of AMPS, including out-of-focus tracking, aggregation, and delta messaging. These are described later in this chapter. For applications that are transitioning from topic-based routing and that, therefore, need to maintain the last value per topic for a large number of topics (hundreds, thousands, or more), AMPS provides the ability to reduce the overhead in creating a large number of identical topics that contain a single message. More details on the [State of the World](../../amps-user-guide/sow) are available in the [AMPS User Guide](../../amps-user-guide/). --- # Atomic Query and Subscribe When a topic is recorded in the SOW, an application can request the current state of the topic and simultaneously subscribe to updates from the topic. In this case, AMPS first delivers all of the messages that match the query and then provides any update to a record that matches the query. AMPS guarantees that no updates are missed or duplicated between the query and the subscription. As with a simple query, AMPS will test each message currently in the SOW against the content filter specified and all messages matching the filter will be returned to the client. When the query begins, AMPS enters a subscription with the provided filter. After the query completes, AMPS delivers messages from the subscription. In the event that a record is updated _while_ the query is running, AMPS saves the update and delivers it immediately after the query completes. As with a simple SOW query, the topic can be a literal topic name or a regular expression pattern. For more information on issuing queries, please see [Querying the State of the World](../../amps-user-guide/sow-queries) in the [AMPS User Guide](../../amps-user-guide/). ## Spark: Basic SOW Query and Subscribe Example Here's how to use `spark` to query the current state of an AMPS SOW topic and subscribe to updates. This example assumes that: * You have configured a topic named `test-sow` in the AMPS server of message type JSON. * The `test-sow` topic uses the `/id` field of the message to calculate the key for the topic. To retrieve the current state of the topic and subscribe, an application issues the `sow_and_subscribe` command. Since the command includes a subscription, the command stays active until it is explicitly stopped (or the application disconnects). First, publish a message or two to the `test-sow` topic: 1. Open a new terminal in your Linux environment. 2. Use the following command (with `AMPS_DIR` set to the directory where you installed AMPS) to send a single message to AMPS: ```bash $ echo '{"id":1,"note":"Crank it up with a SOW!"}' | \ $AMPS_DIR/bin/spark publish -server localhost:9007 \ -type json -topic test-sow ``` 3. `spark` automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). With the `publish` command, `spark` reads the message from the standard input and publishes the message to the JSON topic `test-sow`. The command produces output similar to the following line (the rate calculation will likely be different: ```bash total messages published: 1 (333.33/s) ``` 4. When the publisher sends the message, AMPS parses the message to determine the value of the Key fields in the message, and then either inserts the message for that key, or overwrites the existing message with that key. 5. You can publish any number of messages this way. Each distinct `id` value will create a distinct record in the topic. Next, retrieve the current contents of the topic: 1. Open a new terminal in your Linux environment. 2. Use the following command (with `AMPS_DIR` set to the directory where you installed AMPS) to retrieve the contents of the topic: ```bash $ $AMPS_DIR/bin/spark sow_and_subscribe -server localhost:9007 \ -type json -topic test-sow ``` 3. `spark` automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). `spark` then sends the `sow_and_subscribe` command to AMPS. This command requests the current contents of the `test-sow` topic and creates a subscription to the topic. 4. `spark` shows the current contents of the topic. Notice that the output is strictly the message data, separated by newline characters. `spark` does not show any of the metadata for a message. 5. `spark` remains running after the query completes, waiting for new publishes to arrive. Publish more messages (or updates to the existing messages) to the topic. In the terminal you opened to publish the first messages: 1. Use the following command (with `AMPS_DIR` set to the directory where you installed AMPS) to send a message to AMPS: ```bash $ echo '{"id":1,"note":"Crank it up with a SOW!"}' | \ $AMPS_DIR/bin/spark publish -server localhost:9007 \ -type json -topic test-sow ``` 2. Notice that the subscription receives the message. If you close the subscriber and re-run it, you will see that the second time the subscriber runs, it receives the updated messages in the query and, again, waits for changes to arrive. --- # When Should I Store a Topic in the SOW? Storing a topic in the State of the World is most useful when your application needs to use the current state of the data being tracked. Storing a topic in the State of the World can be especially useful if your application would benefit from automatically receiving updates as soon as they are made (described in more detail in the [Atomic Query and Subscribe](subscriptions) topic). Below you will find common uses of a SOW topic, which include examples of practical use cases: * An application needs the current state of a record, but does not need to recreate the message flow that created that record: _An order fulfillment system presents a view of all currently pending orders when the application starts up._ * An application needs the current state of a record or set of records, even when the topic is high-volume or quickly changing: _A warehouse management application locates the current inventory level for a product._ _A taxi dispatch company locates taxis currently within 10 blocks of an event._ * An application wants to be able to publish incremental updates to a record: _A customer updates her shipping address. All pending orders for the customer are automatically updated without affecting any other information in the order, and processors working with the orders are notified of the change._ * An application wants to receive only the changed fields of a record: _A mobile application displays the status of an order as the order progresses through the stages of validation: the application receives only the identifier for the record and the changed fields._ * An application needs the AMPS server to calculate values based on the current values of a record or set of records: _A management console constantly calculates the real-time value of pending orders. The console uses a view, calculated based on data saved in a topic in the SOW._ * An application wants to store application state for quick retrieval: _An order processing system publishes statistics on each step of the process: a separate process monitors and aggregates those statistics. The SOW also maintains historical state for the topic so the monitor can easily recreate a snapshot of the state at a point in time and compare day over day status._ Of course, the examples above are just a small sample of the ways the AMPS SOW can be used. --- # Starting AMPS The AMPS engine binary is named `ampServer` and is found in `$AMPSDIR/bin`. Start the AMPS engine with a single command line argument that includes a valid path to an AMPS configuration file. You use the configuration file to enable and configure the AMPS features that your application will use. This guide discusses the most commonly used configuration options for each feature. The full set of options is described in the [AMPS User Guide](../amps-user-guide/). The AMPS server generates a minimal sample configuration file with the `--sample-config` option. You can save the sample configuration file to `$AMPSDIR/amps_config.xml` with the following command line: ```bash $AMPSDIR/bin/ampServer --sample-config > $AMPSDIR/amps_config.xml ``` :::info The sample configuration file generated by AMPS includes a very minimal configuration. The client language distributions include a sample configuration file that sets up AMPS to work with the samples provided with that client, and the [AMPS User Guide](../amps-user-guide) contains a full description of the configuration items with sample configuration snippets. ::: The server sample configuration only provides configuration for using AMPS to subscribe to and publish to ad hoc topics. The sample configuration file does not include any persistence for AMPS messages. The file enables the instance monitoring interface (the "Galvanometer"), including the ability to query and subscribe to topics using a websocket connection. A production configuration would likely provide persistent event and error logging to a file to allow an operations team to troubleshoot the instance and would typically persist monitoring statistics to a file. Such a configuration would likely enable additional message delivery features for certain topics and would also include configuration for high-availability and disaster recovery. The configuration would typically configure AMPS actions to perform routine maintenance. :::info AMPS uses the current working directory for storing files (logs and persistence) for any relative paths specified in the configuration. While this is important for real deployments, the sample configuration used in this chapter does not persist anything, so you can safely start AMPS from any working directory using this configuration. ::: On older processor architectures (and in some emulated environments) `ampServer` will start the `ampServer-compat` binary. The `ampServer-compat` binary avoids using hardware instructions that are not available on these systems. You can also set the `AMPS_PLATFORM_COMPAT` environment variable to force `ampServer` to start the `ampServer-compat` binary. 60East recommends using this option only on systems that do not support the hardware instructions used in the standard binary. The `ampServer-compat` binary will not perform as well as `ampServer`, since it uses fewer hardware optimizations. Once you have a configuration file saved to `$AMPSDIR/amps_config.xml` you can start AMPS with that file as follows: ```bash $AMPSDIR/bin/ampServer $AMPSDIR/amps_config.xml ``` If your first start-up is successful, you should see AMPS display a simple message similar to the following to let you know that your instance has started correctly. ```bash AMPS A.B.C.D.973814.e1a57f7 - Copyright (c) 2006-202X 60East Technologies Inc. (Built: XXXX-YY-ZZT00:26:45Z) ``` The version numbers and dates will be appropriate for the version that you've started. If you see this, congratulations! You have successfully cranked up the AMPS! ## Command Line Options The AMPS server binary supports the following command line options: | Option | Effect | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--verify-config` | Parse and verify the specified configuration file, then exit. | | `--sample-config` | Produce a minimal AMPS config.xml file to standard output, then exit. | | `--dump-config` | Process the specified configuration file, resolving any Include directives and expanding environment variables. Dump the resulting file to standard output. | | `--version` | Print the AMPS version string, then exit. | | `--help` | Print usage information for the command line options accepted by the ampServer program, then exit. | | `--daemon` | Run AMPS as a daemon process. | | `-D=` |

    Set the specified environment **variable** to the specified **value** when running the AMPS process. AMPS accepts any number of `-D` options.

    For example, to set the variable `AMPS_PATH` to `/mnt/fast/AMPS` use the command line option `-DAMPS_PATH=/mnt/fast/AMPS`

    | --- # Getting Support ## Technical Support and Assistance At 60East, the most important part of what we do is helping people deploy systems that utilize AMPS and supporting them in maintaining the ongoing and intended operation of those systems. Considering that AMPS is often used in essential systems that push the limits of hardware, network and storage capacity, we know that support is essential to help you build, deploy and maintain the kinds of cutting-edge applications that we built AMPS to handle. During the evaluation and development stages, we encourage you to share details with us about what you are building. This way, we can provide assistance with the design and architecture process. Once your application goes into production, use 60East support to help diagnose and correct issues that fall outside of the normal operation of the application. The level of support you have available is dependent on your support agreement. For an outline of your specific support policies, please see your 60East Technologies License Agreement. Support contracts can be purchased through your 60East Technologies account representative. ### Support Steps You can save time if you complete the following steps before you contact 60East Technologies Support: 1. _**Check the documentation**_ The problem may already be solved and documented in the _AMPS_ _User Guide_ or _Configuration Guide_ for the product. Check the support site at [http://crankuptheamps.com/support](/support) where 60East Technologies also provides answers to frequently asked support questions. 2. _**Isolate the problem**_ If you require Support Services, please isolate the problem to the smallest test case possible. Capture erroneous output into a text file along with the commands used to generate the errors. 3. _**Collect your information**_ * Your product version number. * Your operating system and its kernel version number. * The expected behavior, observed behavior and all input used to reproduce the problem. * Submit your request. * In your email to [crash@crankuptheamps.com](mailto:crash@crankuptheamps.com) include the minidump file if you have one. The AMPS version number used when reporting your product version number follows a format listed below. The version number is composed of the following: ``` MAJOR.MINOR.FEATURE.HOTFIX.TIMESTAMP.TAG ``` ### Contacting 60East Technologies Support Please contact 60East Technologies Support Services according to the terms of your 60East Technologies License Agreement. Visit the support site at [http://crankuptheamps.com/support](/support) for evaluations. Support is offered through the United States: | Channel | Contact | | ---------------------- | ----------------------------------------------------------------------- | | Web | [http://crankuptheamps.com/support](/support) | | E-Mail (non-technical) | [sales@crankuptheamps.com](mailto:sales@crankuptheamps.com) | Other support options (such as support via phone, dedicated engineers, and so on) may be available, depending on the terms of your support agreement. --- # Record and Replay Messages with the AMPS Transaction Log AMPS provides the ability to record topics and replay those topics at a later time. This capability is called the _transaction log_. The AMPS transaction log fully supports topic and content filtering. You configure the transaction log to keep a journal of incoming messages for one or more topics, and then you can replay those messages, in order, from any point in time. With the (optional) high-availability features in the AMPS client libraries, this also provides a way to ensure that in case of failure, an application can resume the subscription without missing messages or receiving duplicate messages. The AMPS transaction log is most often used for: * **Fully Resumable Subscriptions** - With the transaction log, you can ensure that an application receives all messages of interest, even in the event of a failure. * **Backtesting and Audit** - The transaction log allows you to replay the precise messages published, in order across all topics in the instance, at a configurable maximum rate. You can use this feature to easily audit the flow of messages, perform backtesting, or replay a sequence of events. * **Capacity Planning and Stress Testing** - Since the transaction log allows you to set the maximum replay rate to be a multiple of the original publish rate, you can use the transaction log to measure the load on a system at various rates, and measure the capacity of the system and the ability of your application to correctly handle increased volumes. The transaction log is also the source of messages for: * [Message Queues](queues) * [Replicating Messages Between Instances](../amps-user-guide/replication) The AMPS transaction log can typically record messages at the maximum throughput of the underlying storage device. 60East recommends storing the transaction log on a device that supports fast sequential writes, and ensuring that the device has the speed and capacity necessary to support the expected throughput. (The [Operation and Deployment](../amps-user-guide/operation) chapter of the [AMPS User Guide](../amps-user-guide/) includes guidance on capacity planning.) ### How Does the Transaction Log Work? The AMPS transaction log records messages that are published to the topics specified in the configuration file. Every publish is stored, in the order in which the AMPS instance processed the message. For ease of maintenance, the AMPS server writes multiple sequential files, called _journal files_, for the transaction log rather than writing a single large file. The journals contain the full content of each message, as well as information on the topic, the publisher, the time at which the message was processed, and so on. The AMPS configuration sets the maximum size of a journal file. When a file reaches that size, AMPS begins writing to the next file. When AMPS records a message into the transaction log, it assigns each message a _bookmark_. The _bookmark_ identifies a single message, that is, a specific point in the transaction log of the local instance. The _bookmark_ is derived from the publisher name and sequence number, and (when replication is configured) identifies the same message on any instance of AMPS that contains the message. The AMPS server does not modify the contents of journal files. Once a message is written to a journal file, it is part of the transaction log and is considered to be immutable. Since journal files form part of the persistent state of the server, those files should not be modified or removed while the AMPS process is running except by the AMPS process itself. The AMPS server provides a set of maintenance actions for managing journal files (see the [AMPS User Guide](../amps-user-guide/) for details). ### Configuration To create a Transaction Log, you add the `TransactionLog` configuration element to your AMPS configuration file. You then specify a location for AMPS to create journal files, and specify the topics that you want recorded in the file. The following configuration is the minimum configuration to create a transaction log and record a single topic: ```xml showLineNumbers ./journals some-topic json ``` The configuration above writes journal files to the `journals` directory underneath the AMPS server's current working directory. The configuration records a single topic, `some-topic`, of message type `json` to the transaction log. The `Name` option of the `Topic` configuration element can be either a literal topic name, or a regular expression that matches the names of a set of topics to be recorded. Although this configuration works perfectly well, AMPS provides a number of additional options that are useful for managing transaction logs in production. AMPS also provides a set of administrative actions for setting the archival and retention policy for journal files. A more complete configuration might include options along the following lines: ```xml showLineNumbers ./journals /mnt/high-capacity/journals 100MB ^/orders json ^/status/customer fix /audit/events binary amps-action-on-schedule 21:30 Daily Journal Maintenance Plan amps-action-do-archive-journals 3d amps-action-do-remove-journals 7d ``` In this configuration, journals are created in the `journals` directory underneath the AMPS server's current working directory, as before. This configuration records two _sets_ of topics and one individual topic. Taking these in the order in which they appear in the configuration file, this instance of AMPS will record: * Any topic that begins with `/orders` and is of message type JSON. * Any topic that begins with `/status/customer` and is of message type FIX. * The topic `/audit/events` of message type binary. The sample above also includes a basic journal maintenance configuration. Configuring journal maintenance is strongly recommended for any instance of AMPS that will be running on a regular basis. For this AMPS installation, the size of the journal files has been reduced from the default `1GB` size to a `100MB` size. This typically indicates that the instance stores less than 1GB of messages during a day, so the default journal size would include multiple days worth of messages. This configuration specifies a two-step maintenance process: * After 3 days, journal files will be archived to the `/mnt/high-capacity/journals` directory -- the directory specified in the `JournalArchiveDirectory` parameter of the transaction log. These journal files remain a part of the transaction log but are moved to a different location (typically on a different device with higher capacity). * After 7 days, journal files will be deleted. AMPS will run this maintenance plan every day at 21:30 (9:30 PM) local time. When journal files are moved to the archive directory, they continue to be part of the transaction log, but they do not have to be on the same device as the `JournalDirectory`. Most often, a production installation of AMPS will keep journal files that are very active on fast storage and keep a longer period of history on storage that is higher capacity and lower cost. Since these devices typically also have lower throughput, these devices are best for files that must still be retained but are infrequently used. Full details on these options are available in the [Configuring a Transaction Log](/docs/amps-user-guide/txlog/configuring-a-transaction-log) section of the [AMPS User Guide](../amps-user-guide/). ### Further Reading See chapter on [Record and Replay Messages](../amps-user-guide/txlog) in the [AMPS User Guide](../amps-user-guide/) for a more complete discussion of the transaction log and message replay. The AMPS client libraries include samples for publishing messages and replaying messages from the transaction log. See the client library distribution for those samples. :::info Notice that some libraries are distributed as pre-built binaries through package management systems. 60East also offers full distributions including documentation and source from the 60East website. If you've installed a pre-built library using a package manager, visit the 60East web site to download the full distribution that contains the samples. ::: --- # Welcome to AMPS Welcome to the AMPS Documentation! This set of documentation contains detailed information on the AMPS server itself and [developer guides](/docs/amps-reference-to-clients) for the AMPS client libraries. :::info This documentation is also available in the PDF format: Download PDF ::: Here are some suggested starting points: | Scenario | Start With | | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | New to AMPS | [Introduction to AMPS](/docs/intro-guide/intro) | | Beginning an Evaluation of AMPS | [AMPS Evaluation Guide](/docs/amps-eval-guide/eval)
    [Introduction to AMPS](/docs/intro-guide/intro) | | Understanding AMPS Features and Configuration Options | [AMPS User Guide](/docs/amps-user-guide/)
    (see the chapter on the feature in question) | | Planning a Deployment of AMPS | [Introduction to AMPS](/docs/intro-guide/intro)
    [Operation and Deployment](/docs/amps-user-guide/operation)
    [Monitoring AMPS](/docs/amps-user-guide/monitoring)
    [Deployment Checklist](/docs/deployment-checklist/checklist) | | Developing Applications with AMPS |[Introduction to AMPS](/docs/intro-guide/intro)

    ([developer guide](/docs/amps-reference-to-clients) and API reference for your language of choice -- available here or from the AMPS [developer pages](/develop))

    (further reading in the [User Guide](/docs/amps-user-guide/)
    [Command Reference](docs/amps-command-reference/) for features you will use)| | Troubleshooting an Issue | [Troubleshooting AMPS](/docs/amps-user-guide/troubleshooting) | | Contacting 60East Support | [Support Site](/support) | You can also visit the [AMPS Server FAQ](/faq) site for frequently asked questions, and the AMPS [developer pages](https://crankuptheamps.com/develop) for resources on developing applications with AMPS. 60East strongly recommends setting up an AMPS environment for testing while you work with the documentation. Instructions for doing so are in the [Introduction to AMPS](/docs/intro-guide/intro). --- # Replication Validation Each `Topic` in a replication `Destination` can configure a unique set of validation checks. By default, all of these checks are applied. Described below are the available checks for replication validation. Expand each item for more details. Note: "This instance" refers to the instance sending messages via replication, and the "downstream instance" refers to the instance receiving messages via replication.
    `txlog` Validates that the topic is contained in the transaction log of the downstream instance. An error on this validation check indicates that this instance is replicating a topic that is not in the transaction log on the downstream instance. This means that the downstream instance is not persisting the messages in a way that can be used for replication, replay, or used as the basis for a queue.
    `replicate` Validates that the topic is replicated from the downstream instance back to this instance. An error on this validation check indicates that this instance is replicating a topic to the downstream instance that is not being replicated back to this instance. This means that any publishes or updates to the topic on the downstream instance are not replicated back to this instance.
    `sow` Validates that if the topic is a `SOW/Topic` in this instance, it must also be a `SOW/Topic` in the downstream instance. An error on this validation check indicates that this instance is replicating a topic to the downstream instance that is a `SOW/Topic` on this instance but is not a `SOW/Topic` on the downstream instance. This means that the topic has different behavior on the downstream instance, and does not maintain the current value of records in the topic in the SOW.
    `cascade` Validates that the downstream instance must enforce the same set of validation checks for this `Topic` as this instance does. When relaxing validation rules for a topic that the downstream instance itself replicates, adding an exclusion for `cascade` is often necessary as well. An error on this validation check indicates that this instance enforces a validation check for a topic that the downstream instance does not enforce when that instance replicates the topic. To understand the impact of this validation check, consider the validation checks that the downstream instance enforces. If the downstream instance enforces the appropriate validation checks, this instance can exclude the `cascade` check. It is sometimes necessary to exclude this check as part of a rolling upgrade, and then to leave this exclusion in place until all instances can be taken offline at the same time. If the `cascade` check is the only check being excluded on any instance, the topology can be considered to meet validation rules (and the `cascade` exclusion can be safely removed during a maintenance window when all of the instances can be updated simultaneously).
    `queue` Validates that if the topic is a queue in this instance, it must also be a queue in the downstream instance. A distributed queue will not function correctly if one of the instances it is replicated to does not define the topic as a queue. This is a mandatory validation check, and cannot be excluded.
    `keys` Validates that if the topic is a `SOW/Topic` in this instance, it must also be a `SOW/Topic` in the downstream instance and the SOW in the downstream instance must use the same `Key` definitions. An error on this validation check indicates that this instance is replicating a topic to the downstream instance that is a `SOW/Topic` on both instances, but that the definition of message identity (the `Key` configuration for the topic) does not match on the two instances. This means that the contents of this topic may be different on these two instances for the same set of messages published.
    `replicate_filter` Validates that if this topic uses a replication filter, the downstream instance must use the same replication filter for replication back to this instance. An error on this validation check indicates that this instance uses a replication filter for a topic that the downstream instance does not use when it replicates the topic. This means that, given the same set of publishes, the downstream instance may replicate a different set of messages than are replicated to that instance. This would produce inconsistent data across the set of replicated instances.
    `queue_passthrough` Validates that if the topic is a queue in this instance, the downstream instance must support passthrough from this group. An error on this validation check indicates that this instance does not pass through messages for one or more groups that the queue is replicated from. This could lead to a situation where a queue message is undeliverable if a network connection is unavailable or if additional instances are added to the set of instances that contain the queue.
    `queue_underlying` Validates that if the topic is a queue in this instance, it must use the same underlying topic definition and filters in the downstream instance. This is a mandatory validation check, and cannot be excluded.
    The sample below shows how to exclude validation checks for a replication destination. In this sample, the `Topic` does not require the downstream destination to replicate back to this instance, and does not require that the downstream destination enforce the same configuration checks for any downstream replication of this topic. ```xml ... json MyStuff-VIEW replicate,cascade ... ``` --- # Units, Intervals, and Environment Variables In AMPS there are a few special characters that you should be aware of when creating your configuration file. These characters can provide some handy shortcuts and make configuration creation easier, but you should also be aware of them so as not to introduce errors. ### State of the World File Name When specifying the file for a State of the World database, using the `%n` string in the file name specifies that the AMPS server will use the message type and topic name in that position to create a unique file name. The example below shows how to use this in the AMPS configuration file. ```xml showLineNumbers Customers ./sow/%n.sow json /customerId ``` ### Log Rotation Name When specifying an AMPS log file which has `RotationThreshold` specified, using the `%n` string in the log file name is a useful mechanism for ensuring the name of the log file is unique and sequential. The example below shows a file name token replacement in the AMPS configuration file. ```xml showLineNumbers file info log/log-%n.log 2G ``` In the above example, a log file will be created in the `AMPSDIR/log/` directory. The first time this file is created, it will be named `log-1.log`. Once the log file reaches the `RotationThreshold` limit of 2G, the previous log file will be saved, and the new log file name will be incremented by one. Thus, the next log file will be named `AMPSDIR/log/log-2.log`. ### Dates AMPS allows administrators to use date based file names when specifying the file name in the configuration, as demonstrated in the example below. ```xml showLineNumbers file info log/log-%Y-%m-%dT%H%M%S.log 2G ``` In the above example, a log file will be created in the `$AMPSDIR/log` named `2011-01-01-120000.log` if the log was created at noon on January 1, 2011. AMPS provides full support for the date tokens provided by the standard strftime function, with the exception of `%n`, as described above. The following table shows some of the most commonly used tokens: | Token | Description | Example | | ----- | ----------------------------------------------------- | ------------------------ | | %a | Short weekday name | Fri | | %A | Full weekday name | Friday | | %b | Short month name | Feb | | %B | Full month name | February | | %c | Simple date and time | Fri Feb 14 17:25:00 2014 | | %C | Century | 20 | | %d | Day of the month (leading zero if necessary) | 05 | | %D | Short date format (MM/DD/YY) | 02/20/14 | | %e | Day of the month (leading space if necessary) | 5 | | %F | Short date format (YYYY-MM-DD) | 2014-02-20 | | %H | Hour (00-23) | 17 | | %I | Hour (00-12) | 05 | | %j | Day of the year (001-366) | 051 | | %m | Month (01-12) | 02 | | %p | AM or PM | PM | | %r | Current time, 12 hour format | 05:25:00 pm | | %R | Current time, 24 hour format | 17:25 | | %T | ISO 8601 Time format | 17:25:00 | | %u | ISO 8601 day of the week (1-7, Monday = 1) | 5 | | %V | ISO 8601 week number (00-53) | 07 | | %y | Year, last two digits | 14 | | %Y | Year, four digits | 2014 | | %Z | Timezone name or abbreviation (blank if undetermined) | PST | ## Using Units in the Configuration To make configuration easy, AMPS permits the use of units to expand values. For example, if a time interval is measured in seconds, then the letter `s` can be appended to the value. For example, the following SOW topic definition uses the `Expiration` tag to set the record expiration to 86400 seconds (one day). ```xml showLineNumbers ... 86400s ... ``` An even easier way to specify an expiration of one day is to use the following `Expiration`: ```xml showLineNumbers ... 1d ... ``` The table below shows a listing of the time units AMPS supports in the configuration file. | Units | Description | | --------- | --------------- | | `ns` | nanoseconds | | `us` | microseconds | | `ms` | milliseconds | | `s` | seconds | | `m` | minutes | | `h` | hours | | `d` | days | | `w` | weeks | AMPS configuration supports a similar mechanism for byte-based units when specifying sizes in the configuration file. The table below shows a listing of the byte units AMPS supports in the configuration file. | Units | Description | | --------- | --------------- | | `kb` | kilobytes | | `mb` | megabytes | | `gb` | gigabytes | | `tb` | terabytes | Dealing with large numbers in AMPS configuration can also be simplified by using common exponent values to handle raw values. This means that instead of having to input `10000000` to represent ten million, a user can input `10M`. The table below contains a list of the exponents supported. | Units | Description | | --------- | --------------- | | k | 10^3 - thousand | | M | 10^6 - million | :::tip To make it easier to remember the units, AMPS interval and byte units are not case sensitive. ::: ## Environment Variables in AMPS Configuration AMPS configuration also allows for environment variables to be used as part of the data when specifying a configuration file. These variables can be set in the environment when AMPS starts or passed to AMPS using the `-D` option on the command line. If a global system variable is commonly used in an organization, then it may be useful to define this in one location and re-use it across multiple AMPS installations or applications. AMPS will replace any token wrapped in `${}` with the environment variable defined in the current user operating system environment. The example below demonstrates how the environment variable `ENV_LOG` is used to define an environment variable for the location of the host logging. ```xml showLineNumbers file ${ENV_LOG} info 2G ``` ## Internal Environment Variables In addition to supporting custom environment variables, AMPS includes a set of environment variables automatically populated by the server. These variables are listed below: | Variable Name | Contains | | ----------------------- | -------------------------------------------------------------------------------- | | `AMPS_CONFIG_DIRECTORY` | Directory in which the configuration file used to start AMPS is located. | | `AMPS_CONFIG_PATH` | Full path to the configuration file used to start AMPS, including the file name. | | `AMPS_VERSION` | Full version number of the AMPS server. | When AMPS processes the configuration file, AMPS expands these variables just as though they were set in the environment. For example, assume that AMPS was started with the following command at the command prompt: ```bash %>./ampServer ../amps/config/config.xml ``` Given this command, the log file configuration option shown in the example below can be used to instruct AMPS to create the log files in the same parent directory as the configuration file — in this case `../amps/config/logs/infoLog.log`. ```xml showLineNumbers file ${AMPS_CONFIG_DIRECTORY}/logs/infoLog.log info 2G ``` --- # AMPS Clients Welcome to developing applications with AMPS, the Advanced Message Processing System from 60East Technologies! These guides will help you learn how to develop applications using AMPS. ## Before You Start Before getting started with this guide, it is important to have a good understanding of the following topics: * *Developing Applications in your Language of Choice* - To be successful using this guide, and developing applications with AMPS, you will need to have a working knowledge of the language you are developing in. * *AMPS Concepts* - This guide focuses on using the AMPS client libraries and how those libraries work with the AMPS server. Before working through this guide, we recommend reading the [Introduction to AMPS](/docs/intro-guide/intro) guide. Detailed explanations of the AMPS server behavior are in the [AMPS Server Documentation](/docs). You will also need a system on which you can compile and run code, and a system where you can host the AMPS server. ## Setting Up a Development Instance You will need an installed and running AMPS server to use the product as well. You can write and compile programs that use AMPS without a running server, but you will get the most out of this guide by running the programs against a working server. Instructions for starting an instance of AMPS are available in the [Introduction to AMPS](/docs/intro-guide/intro) guide. :::tip The AMPS server runs on x64 Linux. The [Introduction to AMPS](/docs/intro-guide/intro) and [AMPS FAQ](/faq) contain information on how to run an AMPS server on a development system that does not run Linux. ::: --- # Welcome to the AMPS C/C++ Client This guide provides information you need to get started with the AMPS C/C++ client. It focuses specifically on the client and does not cover AMPS itself in detail. For an overview of AMPS and instructions on setting up your development environment, see the [Introduction to AMPS](/docs/intro-guide/intro) guide. :::tip This guide assumes that you have a development environment for C/C++ and access to an AMPS server using the configuration provided with the C/C++ samples (in the full source distribution of the client). ::: --- ## Understanding Message Objects So far, we have seen that subscribing to a topic involves working with objects of `AMPS::Message` type. A `Message` represents a single message to or from an AMPS server. Messages are received or sent for every client/server operation in AMPS. ## Header properties There are two parts of each message in AMPS: a set of headers that provide metadata for the message, and the data that the message contains. Every AMPS message has one or more header fields defined. The precise headers present depend on the type and context of the message. There are many possible fields in any given message, but only a few are used for any given message. For each header field, the `Message` class contains a distinct property that allows for retrieval and setting of that field. For example, the `Message.get_command_id()` function corresponds to the `commandId` header field, the `Message.get_batch_size()` function corresponds to the `BatchSize` header field, and so on. For more information on these header fields, consult the *AMPS User Guide* and *AMPS Command Reference*. To work with header fields, a `Message` contains `getXxx()`/`setXxx()` methods corresponding to the header fields. 60East does not recommend attempting to parse header fields from the raw data of the message. In AMPS, fields sometimes need to be set to a unique identifier value. For example, when creating a new subscription, or sending a manually constructed message, you’ll need to assign a new unique identifier to multiple fields such as `CommandId` and `SubscriptionId`. For this purpose, Message provides `newXxx()` methods for each field that generates a new unique identifier and sets the field to that new value. ### getData() method Access to the data section of a message is provided via the `getData()` method. The `data` contains the unparsed data in the message, returned as a series of bytes (a `string` or `const char *`). Your application code parses and works with the data. The AMPS C++ client contains a collection of helper classes for working with message types that are specific to AMPS (for example, FIX, NVFIX, and AMPS composite message types). For message types that are widely used, such as JSON or XML, you can use whichever library you typically use in your environment. ## Advanced Messaging Support The `client.subscribe()` function provides options for subscribing to topics even when you do not know their exact names, and for providing a filter that works on the server to limit the messages your application must process. ## Multiple Topics with One Subscription AMPS allows a regular expression to be supplied in the place of a topic name. When you supply a regular expression, AMPS treats this as a request for topic that matches your expression, including topics that do not yet exist at the time of creating the subscription. To use a regular expression, simply supply the regular expression in place of the topic name in the `subscribe()` call. For example: ```cpp for (auto message : client.subscribe("client.*")) { /* receive messages for any topic that begins with 'client' */ std::cout << "Received a message on topic '" << message.getTopic() << "' " << "with the data: " << message.getData() << std::endl; } ``` **Example (CHAPTER_NUMBER).(REGEX_TOPIC_SUBSCRIPTION):** *Regex topic subscription* In this example, messages on topics `client` and `client1` would match the regular expression, and those messages will be returned by the `MessageStream` . As in the example, you can use the `getTopic()` method to determine the actual topic of the message sent to the lambda function. ## Content filtering One of the most powerful features of AMPS is content filtering. With content filtering, filters based on message content are applied at the server, so that your application and the network are not utilized by messages that are uninteresting for your application. For example, if your application is only displaying messages from a particular user, you can send a content filter to the server so that only messages from that particular user are sent to the client. The *AMPS User Guide* provides full details on content filtering. To apply a content filter to a subscription, simply pass it into the `client.subscribe()` call: ```cpp for (auto message : ampsClient.subscribe("messages", 0, "/sender = 'mom'")) { // process messages from mom } ``` ## Next steps At this point, you are able to build AMPS programs in C/C++ that publish and subscribe to AMPS topics. For an AMPS application to be truly robust, it needs to be able to handle the errors and disconnections that occur in any distributed system. In the next chapter, we will take a closer look at error handling and recovery, and how you can use it to make your application ready for the real world. --- # Advanced Topics ## Transport Filtering The AMPS C/C++ client offers the ability to filter incoming and outgoing messages in the format they are sent and received on the network. This allows you to inspect or modify outgoing messages before they are sent to the network, and incoming messages as they arrive from the network. This can be especially useful when using SSL connections, since this gives you a way to monitor outgoing network traffic before it is encrypted, and incoming network traffic after it is decrypted. To create a transport filter, you create a function with the following signature: ```cpp void amps_tcp_filter_function(const unsigned char* data,size_t len,short direction, void* userdata); ``` You then register the filter by calling `setTransportFilterFunction` with a pointer to the function and a pointer to the data to be provided in the `userdata` parameter of the callback. For example, the following filter function simply prints the data provided to the standard output: ```cpp showLineNumbers void amps_tcp_trace_filter_function(const unsigned char* data, size_t len, short direction, void* userdata) { /* Output the direction marker */ if (direction == 0) { std::cout << "OUTGOING ---> "; } else { std::cout << "INCOMING ---> "; } /* Output the data */ std::cout << std::string(data, len) << std::endl; } ``` Registering the function is a matter of calling `setTransportFilterFunction` with this function and any callback data, as shown below: ```cpp showLineNumbers /* client is an existing Client object */ client.setTransportFilterFunction( &s_tcp_trace_filter_function, (void*)NULL); ``` The snippet above installs the filter function for the client. Notice that the transport filter function is called with the verbatim contents of data received from AMPS. This means that, for incoming data, the function may not be called precisely on message boundaries, and that the binary length encoding used by the client and server will be presented to the transport filter. ## Using SSL The AMPS C++ client includes support for Secure Sockets Layer (SSL) connections to AMPS. The client automatically attempts to make an SSL connection when the transport in the connection string is set to `tcps`, as described in the [Connection Strings for AMPS](./connection-strings.md) section of this guide. To use the `tcps` transport, your application must have an SSL library loaded before making the `tcps` connection. Notice that the AMPS C++ client uses the OpenSSL implementation that you provide. The AMPS client distribution doesn't include OpenSSL, and doesn't provide facilities for certificate generation, certificate signing, key management, and so forth. Those facilities are provided by the OpenSSL implementation you choose. ## Loading and Initializing the SSL Library To make an SSL connection, the AMPS client must have an SSL library loaded before making the SSL connection. There are two common ways to load the library: 1. At link time, specify the OpenSSL shared object file (Linux) or DLL (Windows) to the linker. With this approach, the operating system will load the SSL library for your application automatically when the application starts up. You then use call `amps_ssl_init` with `NULL` as the library name to initialize the library. 2. Use the `amps_ssl_init` function to load the SSL library. This function accepts either the library name, or a full path including the library name. When called with the library name, the AMPS C++ client will search appropriate system paths for shared libraries (for example, the `LD_LIBRARY_PATH` on Linux) and load the first object found that matches the provided name. When called with a full path, the AMPS C++ client will load exactly the object specified. The AMPS client will then initialize the library. Once the SSL library is loaded and initialized, you can connect using `tcps` as a transport type. The fact that the connection uses a secure socket is only important when making the connection, and does not affect the operation of the client once the connection has been made. --- # Asynchronous Message Processing The AMPS C++ client also supports an interface that allows you to process messages asynchronously. In this case, you add a message handler to the function call. The client returns the command ID of the subscribe command once the server has acknowledged that the command has been processed. As messages arrive, the client calls your message handler directly on the background thread. This can be an advantage for some applications. For example, if your application is highly multithreaded and copies message data to a work queue processed by multiple threads, there is usually a performance benefit to enqueuing work directly from the background thread. See [Understanding Threading](understanding-threading-section.md) for a discussion of threading considerations, including considerations for message handlers. The advantage of using asynchronous message processing is that it is extremely efficient -- your processing code runs directly on the thread that the AMPS client is using to read from the socket and has direct access to the buffer that the AMPS client uses. Further, when an `HAClient` is used (discussed later in this guide), the default disconnect handler for that client will automatically resume subscriptions that use asynchronous message processing. Last, but not least, since message processing runs directly on the receive thread, using asynchronous message processing will provide pushback on the socket in the event that messages are arriving faster than the application can process them (for example, in response to a `sow` query). In return for these advantages, your processing code must be careful not to block the processing thread for an excessive amount of time, and must make a deep copy of any data that will be used outside of the processing code. ### Asynchronous Processing Example Here is a short example (error handling and connection details are omitted for brevity): ```cpp showLineNumbers Client client(...); client.connect(...); /* Here we have created or received a Client that is properly connected to an AMPS server. */ client.logon(); /* Here we create a subscription with the following parameters: * command : This is the AMPS Command object that contains the subscribe command. * MessageHandler : This is an AMPS MessageHandler object that refers to our message * handling function myHandlerFunction. This function is * called on a background thread each time a message arrives. * The second parameter, NULL, is passed as-is from the * client.subscribe() call to the message handler with every message, * allowing you to pass context about the subscription through to the * message handler. * * We create a command object for the subscribe command, specifying the topic * messages. */ string subscriptionId = client.executeAsync(Command("subscribe").setTopic("messages"), MessageHandler(myHandlerFunction, NULL)); ... /* The myHandlerFunction is a global function that is invoked by AMPS whenever a * matching message is received. The first parameter, message, is a reference to an * AMPS Message object that contains the data and headers of the received message. * The second parameter, userData, is set to whatever value was provided in the * MessageHandler constructor -- NULL in this example. */ void myHandlerFunction(const Message& message, void* userData) { std::cout << message.getData() << std::endl; } ``` :::warning The AMPS client resets and reuses the message provided to this function between calls. This improves performance in the client, but means that if your handler function needs to preserve information contained within the message, you must copy the information rather than just saving the message object. Otherwise, the AMPS client cannot guarantee the state of the object or the contents of the object when your program goes to use it. ::: With newer compilers, you can use additional constructs to specify a callback function. Recent improvements in C++ have added lambda functions -- unnamed functions declared in-line that can refer to names in the lexical scope of their creator. If available on your system, both Standard C++ Library function objects and lambda functions may be used as callbacks. Check `functional.cpp` in the samples directory for numerous examples. ### Using an Instance Method as a Message Handler One of the more common ways of providing a message handler is as an instance method on an object that maintains message state. It's simple to provide a handler with this capability, as shown below. ```cpp showLineNumbers class StatefulHandler { private: std::string _handlerName; public: /* Construct the handler and save state. */ StatefulHandler(const std::string& name) : _handlerName(name) {} /* Message handler method. */ void operator()(const AMPS::Message & message) { std::cout << _handlerName << " got " << message.getData() << std::endl; } }; ``` You can then provide an instance of the handler directly wherever a message handler is required, as shown below: ```cpp client.subscribe(StatefulHandler("An instance"), "topic"); ``` --- # Backlog and Smart Pipelining AMPS queues are designed for high-volume applications that need minimal latency and overhead. One of the features that helps performance is the *subscription backlog* feature, which allows applications to receive multiple messages at a time. The subscription backlog sets the maximum number of unacknowledged messages that AMPS will provide to the subscription. When the subscription backlog is larger than `1`, AMPS delivers additional messages to a subscriber before the subscriber has acknowledged the first message received. This technique allows subscribers to process messages as fast as possible, without ever having to wait for messages to be delivered. The technique of providing a consistent flow of messages to the application is called *smart pipelining*. ## Subscription Backlog The AMPS server determines the backlog for each subscription. An application can set the maximum backlog that it is willing to accept with the `max_backlog` option. Depending on the configuration of the queue (or queues) specified in the subscription, AMPS may assign a smaller backlog to the subscription. If no `max_backlog` option is specified, AMPS uses a `max_backlog` of `1` for that subscription. In general, applications that have a constant flow of messages perform better with a `max_backlog` setting higher than `1`. The reason for this is that, with a backlog greater than `1`, the application can always have a message waiting when the previous message is processed. Setting the optimum `max_backlog` is a matter of understanding the messaging pattern of your application and how quickly your application can process messages. To request a `max_backlog` for a subscription, you explicitly set the option on the subscribe command, as shown below: ```cpp showLineNumbers Command cmd("subscribe"); cmd.setTopic("my_queue") .setOptions("max_backlog=10"); ``` ## Acknowledging Messages For each message delivered on a subscription, AMPS counts the message against the subscription backlog until the message is explicitly acknowledged. In addition, when a queue specifies `at-least-once` delivery, AMPS retains the message in the queue until the message expires or until the message has been explicitly acknowledged and removed from the queue. From the point of view of the AMPS server, this is implemented as a `sow_delete` from the queue with the bookmarks of the messages to remove. The AMPS C++ client provides several ways to make it easier for applications to create and send the appropriate `sow_delete`. ## Automatic Acknowledgment The AMPS client allows you to specify that messages should be automatically acknowledged. When this mode is on, AMPS acknowledges the message automatically in the following cases: - *Asynchronous message processing interface* - The message handler returns without throwing an exception. - *Synchronous message processing interface* - The application requests the next message from the `MessageStream`. AMPS batches acknowledgments created with this method, as described in the following section. To enable automatic acknowledgment, use the `setAutoAck()` method. ```cpp client.setAutoAck(true); // enable AutoAck ``` ## Message Convenience Method The AMPS C++ client provides a convenience method, `ack()`, on delivered messages. When the application is finished with the message, the application simply calls `ack()` on the message. (This, in turn, provides the topic and bookmark to the `ack()` function of the client that received the message.) For messages that originated from a queue with `at-least-once` semantics, this adds the bookmark from the message to the batch of messages to acknowledge. For other messages, this method has no effect. ```cpp message.ack(); // Add this message to the next // acknowledgment batch. ``` --- # Setting Batch Size The AMPS clients include a batch size parameter that specifies how many messages the AMPS server will return to the client in a single batch when returning the results of a SOW query. The 60East clients set a batch size of 10 by default. This batch size works well for common message sizes and network configurations. Adjusting the batch size may produce better network utilization and produce better performance overall for the application. The larger the batch size, the more messages AMPS will send to the network layer at a time. This can result in fewer packets being sent, and therefore less overhead in the network layer. The effect on performance is generally most noticeable for small messages, where setting a larger batch size will allow several messages to fit into a single packet. For larger messages, a batch size may still improve performance, but the improvement is less noticeable. In general, 60East recommends setting a batch size that is large enough to produce few partially-filled packets. Bear in mind that AMPS holds the messages in memory while batching them, and the client must also hold the messages in memory while receiving the messages. Using batch sizes that require large amounts of memory for these operations can reduce overall application performance, even if network utilization is good. For smaller message sizes, 60East recommends using the default batch size, and experimenting with tuning the batch size if performance improvements are necessary. For relatively large messages (especially messages with sizes over 1MB), 60East recommends explicitly setting a batch size of 1 as an initial value, and increasing the batch size only if performance testing with a larger batch size shows improved network utilization or faster overall performance. --- # Before You Start Welcome to developing applications with AMPS, the Advanced Message Processing System from 60East Technologies! These guides will help you learn how to develop applications using AMPS. Before getting started with this guide, it is important to have a good understanding of the following topics: * *Developing Applications in C or C++* To be successful using this guide, and developing applications with AMPS, you will need to have a working knowledge of the language you are developing in. * *AMPS Concepts* This guide focuses on using the AMPS client libraries and how those libraries work with the AMPS server. Before working through this guide, we recommend reading the [Introduction to AMPS](/docs/intro-guide/intro) guide. Detailed explanations of the AMPS server behavior are in the [AMPS Server Documentation](/). You will also need a system on which you can compile and run code, and a system where you can host the AMPS server. ## Setting Up a Development Instance You will need an installed and running AMPS server to use the product as well. You can write and compile programs that use AMPS without a running server, but you will get the most out of this guide by running the programs against a working server. Instructions for starting an instance of AMPS are available in the [Introduction to AMPS](/docs/intro-guide/intro) guide. :::tip The AMPS server runs on x64 Linux. The [Introduction to AMPS](/docs/intro-guide/intro) and [AMPS FAQ](/faq) contain information on how to run an AMPS server on a development system that does not run Linux. ::: --- # Client Identification AMPS uses the name of the client as a session identifier and as part of the identifier for messages originating from that client. For this reason, when a transaction log is enabled in the AMPS instance (that is, when the instance is recording a sequence of publishes and attempting to eliminate duplicate publishes), an AMPS instance will only allow one application with a given client name to connect to the instance. When a transaction log is present, AMPS **requires** the client name for a publisher to be: - Unique within a set of replicated AMPS instances - Consistent from invocation to invocation *if* the publisher will be publishing the same *logical* stream of messages If publishers do not meet this contract (for example, if the publisher changes its name and publishes the same messages, or if a different publisher uses the same session name), message loss or duplication can happen. 60East recommends always using consistent, unique client names. For example, the client name could be formed by combining the application name, an identifier for the host system, and the ID of the user running the application. A strategy like this provides a name that will be different for different users or on different systems, but consistent for instances of the application that should be treated as equivalent to the AMPS system. Likewise, if a publisher is sending a completely independent stream of messages (for example, a microservice that sends a different, unrelated sequence of messages each time it connects to AMPS), there is no need for a publisher to retain the same name each time it starts. However, if a publisher is resuming a stream of messages (as in the case when using a file-backed publish store), that publisher **must** use the same client name, since the publisher is resuming the session. --- # Client Side Conflation In many cases, applications that use SOW topics only need the current value of a message at the time the message is processed, rather than processing each change that lead to the current value. On the server side, AMPS provides *conflated topics* to meet this need. See [Conflated Topics]/docs/amps-user-guide/conflated-topics) under the AMPS *User Guide* for more detail. It require no special handling on the client side. In some cases, though, it's important to conflate messages on the client side. This can be particularly useful for applications that do expensive processing on each message, applications that are more efficient when processing batches of messages, or for situations where you cannot provide an appropriate conflation interval for the server to use. A `MessageStream` has the ability to conflate messages received for a subscription to a SOW topic, view, or conflated topic. When conflation is enabled, for each message received, the client checks to see whether it has already received an unprocessed message with the same `SowKey`. If so, the client replaces the unprocessed message with the new message. The application never receives the message that has been replaced. To enable client-side conflation, you call `conflate()` on the `MessageStream`, and then use the `MessageStream` as usual: ```cpp showLineNumbers /* Query and subscribe */ MessageStream results = ampsClient.sowAndSubscribe("orders", "/symbol == 'ROL'"); /* Turn on conflation */ results.conflate(); /* Process the results */ for (auto message : results) { // Process message here } ``` Notice that if the `MessageStream` is used for a subscription that does not include `SowKeys` (such as a subscription to a topic that does not have a SOW), no conflation will occur. When using client-side conflation with delta subscriptions, bear in mind that client-side conflation replaces the whole message, and does not attempt to merge deltas. This means that updates can be lost when messages are replaced. For some applications (for example, a ticker application that simply sends delta updates that replace the current price), this causes no problems. For other applications (for example, when several processors may be updating different fields of a message simultaneously), using conflation with deltas could result in lost data, and server-side conflation is a safer alternative. --- The named convenience methods and the `Command` class provide a `timeout` setting that specifies how long the command should wait to receive a `processed` acknowledgment from AMPS. This can be helpful in cases where it is important for the caller to limit the amount of time to block waiting for AMPS to acknowledge the command. If the AMPS client does not receive the processed acknowledgment within the specified time, the client sends an `unsubscribe` command to the server to cancel the command and throws an exception. Acknowledgments from AMPS are processed by the client receive thread on the same socket as data from AMPS. This means that any other data previously returned (such as the results of a large query) must be consumed before the acknowledgment can be processed. An application that submits a set of SOW queries in rapid succession should set a timeout that takes into account the amount of time required to process the results of the previous query. --- # Connection Parameters for AMPS When specifying a URI for connection to an AMPS server, you may specify a number of transport-specific options in the parameters section of the URI connection parameters. Here is an example: ```bash tcp://localhost:9007/amps/json?tcp_nodelay=true&tcp_sndbuf=100000 ``` In this example, we have specified the AMPS instance on `localhost`, port `9007`, connecting to a transport that uses the `amps` protocol and sending JSON messages. We have also set two parameters: `tcp_nodelay`, a Boolean (true/false) parameter, and `tcp_sndbuf`, an integer parameter. Multiple parameters may be combined to finely tune settings available on the transport. Normally, you'll want to stick with the defaults on your platform, but there may be some cases where experimentation and fine-tuning will yield higher or more efficient performance. The AMPS client supports the value of `tcp` in the *scheme* component connection string for TCP/IP connections, and the value of `tcps` as the scheme for SSL encrypted connections. For connections that use Unix domain sockets, the client supports the value of `unix` in the scheme, and requires an additional option, as described in the Unix Transports Parameters section below. ## IPv6 Connections Starting with version 5.3.3.0, the AMPS client supports creating connections over both IPv4 and IPv6 protocols if supported by the underlying Operating System. By default, the AMPS client will prefer to resolve host names to IPv4 addresses, but this behavior can be adjusted by supplying the `ip_protocol_prefer` transport option, described in the table below. ## TCP and SSL Transport Options The following transport options are available for TCP connections: |Option |Description | |-------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`bind` |

    (IP address) Sets the interface to bind the outgoing socket to.

    Starting with version 5.3.3.0, both IPv4 and IPv6 addresses are fully supported for use with this parameter.

    | |`tcp_connecttimeout` |(integer) Sets the connect timeout in milliseconds. This helps enable failover in cases where an attempt to connect to a server is unresponsive without returning a failure. | |`tcp_rcvbuf` |(integer) Sets the socket receive buffer size. This defaults to the system default size. (On Linux, you can find the system default size in `/proc/sys/net/core/rmem_default`.)| |`tcp_sndbuf` |(integer) Sets the socket send buffer size. This defaults to the system default size. (On Linux, you can find the system default size in `/proc/sys/net/core/wmem_default`.)| |`tcp_nodelay` |(boolean) Enables or disables the `TCP_NODELAY` setting on the socket. By default `TCP_NODELAY` is disabled.| |`tcp_linger` |(integer) Enables and sets the `SO_LINGER` value for the socket By default, `SO_LINGER` is enabled with a value of `10`, which specifies that the socket will linger for 10 seconds.| |`tcp_keepalive` |(boolean) Enables or disables the `SO_KEEPALIVE` value for the socket. The default value for this option is true.| |`ip_protocol_prefer` |

    (string) Influence the IP protocol to prefer during DNS resolution of the host. If a DNS entry of the preferred protocol can not be found, the other non-preferred protocol will then be tried.

    If this parameter is not set, the default will be to prefer IPv4.

    If an explicit IPv4 address or IPv6 IP address is provided as the host, the format of the IP address is used to determine the IP protocol used and this setting has no effect.

    Supported Values:

    `ipv4`: Prefer an IPv4 address when resolving the host

    `ipv6`: Prefer an IPv6 address when resolving the host

    This parameter is available starting with version 5.3.3.0.

    | ## Unix Transport Parameters The `unix` transport type communicates over Unix domain sockets. This transport **requires** the following additional option: |Option |Description | |-----------------------|-------------------------------------------------| |`path` |The path to the Unix domain socket to connect to.| Unix domain sockets always connect to the local system. When the scheme specified is `unix`, the host address is *ignored* in the connection string. For example, the connection string: ```bash unix://localhost:0/amps/json?path=/sockets/the-amps-socket ``` and the connection string: ```bash unix://unix:unix/amps/json?path=/sockets/the-amps-socket ``` are equivalent. The other components of the connection string, including the *protocol*, *message type*, *username*, and *authentication token* are processed just as they would be for TCP/IP sockets. ## AMPS Additional Logon Options The connection string can also be used to pass logon parameters to AMPS. AMPS supports the following additional logon option: |Option |Description | |-----------------------|-----------------------------------------------------------------------------------------------| |`pretty` |Provide formatted representations of binary messages rather than the original message contents.| --- # Monitoring Connection State The AMPS client interface provides the ability to set one or more connection state listeners. A connection state listener is a callback that is invoked when the AMPS client detects a change to the connection state. A connection state listener may be called from the client receive thread. An application should not submit commands to AMPS from a connection state listener, or the application risks creating a deadlock for commands that wait for acknowledgement from the server. The AMPS client provides the following state values for a connection state listener: |State |Indicates | |--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`Connected` |

    The client has established a connection to AMPS. If you are using a `Client`, this is delivered when `connect()` is successful.

    If you are using an `HAClient`, this state indicates that the `connect` part of the connect and logon process has completed. An `HAClient` using the default disconnect handler will attempt to log on immediately after delivering this state.

    Most applications that use `Client` will attempt to log on immediately after the call to `connect()` returns.

    An application should not submit commands to AMPS from the connection state listener while the client is in this state unless the application knows that the state has been delivered from a `Client` and that the `Client` does not call `logon()`.

    | |`LoggedOn` |

    The client has successfully logged on to AMPS. If you are using a `Client`, this is delivered when `logon()` is successful.

    If you are using an `HAClient`, this state indicates that the `logon` part of the connect and logon process has completed.

    This state is delivered after the client is logged on, but before recovery of client state is complete. Recovery will continue after delivering this state: the application should not submit commands to AMPS from the connection state listener while the client is in this state if further recovery will take place.

    | |`HeartbeatInitiated`|

    The client has successfully started heartbeat monitoring with AMPS. This state is delivered if the application has enabled heartbeating on the client.

    This state is delivered before recovery of the client state is complete. Recovery may continue after this state is delivered. The application should not submit commands to AMPS from the connection state listener until the client is completely recovered.

    | |`PublishReplayed` |

    Delivered when a client has completed replay of the publish store when recovering after connecting to AMPS.

    This state is delivered when the client has a `PublishStore` configured.

    If the client has a subscription manager set, (which is the default for an `HAClient`), the application should not submit commands from the connection state listener until the `Resubscribed` state is received.

    | |`Resubscribed` |

    Delivered when a client has re-entered subscriptions when recovering after connecting to AMPS.

    This state is delivered when the client has a subscription manager set (which is the default for an `HAClient`). This is the final recovery step. An application can submit commands to AMPS from the connection state listener after receiving this state.

    | |`Disconnected` |The client is not connected. For an `HAClient`, this means that the client will attempt to reconnect to AMPS. For a `Client`, this means that the client will invoke the disconnect handler, if one is specified.| |`Shutdown` |The client is shut down. For an `HAClient`, this means that the client will no longer attempt to reconnect to AMPS. This state is delivered when `close()` is called on the client or when a server chooser tells the `HAClient` to stop reconnecting to AMPS.| The enumeration provided for the connection state listener also includes a value of `UNKNOWN`, for use as a default or to represent additional states in a custom `Client` implementation. The 60East implementations of the client do not deliver this state. The following table shows examples of the set of states that will be delivered during connection, in order, depending on what features of the client are set. Notice that, for an instance of the `Client` class, this table assumes that the application calls both `connect()` and `logon()`. For an `HAClient`, this table assumes that the `HAClient` is using the default `DisconnectHandler` for the `HAClient`. |Configuration |States | |----------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| |

    subscription manager

    publish store

    |

    `Connected`

    `LoggedOn`

    `PublishReplayed`

    `Resubscribed`

    | |

    subscription manager

    publish store

    heartbeat set

    |

    `Connected`

    `LoggedOn`

    `HeartbeatInitiated`

    `PublishReplayed`

    `Resubscribed`

    | |subscription manager |

    `Connected`

    `LoggedOn`

    `Resubscribed`

    | |

    subscription manager

    heartbeat set

    |

    `Connected`

    `LoggedOn`

    `HeartbeatInitiated`

    `Resubscribed`

    | |(default `Client` configuration) |

    `Connected`

    `LoggedOn`

    | --- # Connection Strings for AMPS The AMPS clients use connection strings to determine the server, port, transport, and protocol to use to connect to AMPS. When the connection point in AMPS accepts multiple message types, the connection string also specifies the precise message type to use for this connection. Connection strings have a number of elements: ![Diagram showing the different elements of a connection string](./assets/client-connection-string-light.svg#gh-light-mode-only) ![Diagram showing the different elements of a connection string](./assets/client-connection-string-dark.svg#gh-dark-mode-only) As shown in the figure above, connection strings have the following elements: * _Transport_ - Defines the network used to send and receive messages from AMPS. In this case, the transport is `tcp`. For connections to transports that use the Secure Sockets Layer (SSL), use `tcps`. For connections to AMPS over a Unix domain socket, use `unix`. * _Host Address_ - Defines the destination on the network where the AMPS instance receives messages. The format of the address is dependent on the transport. For `tcp` and `tcps`, the address consists of a host name and port number. In this case, the host address is `localhost:9007`. For `unix` domain sockets, a value for hostname and port must be provided to form a valid URI, but the content of the hostname and port are ignored, and the file name provided in the **path** parameter is used instead (by convention, many connection strings use `localhost:0` to indicate that this is a local connection that does not use TCP/IP). * _Protocol_ - Sets the format in which AMPS receives commands from the client. Most code uses the default `amps` protocol, which sends header information in JSON format. AMPS supports the ability to develop custom protocols as extension modules, and AMPS also supports legacy protocols for backward compatibility. * _MessageType_ - Specifies the message type that this connection uses. This component of the connection string is required if the protocol accepts multiple message types and the transport is configured to accept multiple message types. If the protocol does not accept multiple message types, this component of the connection string is optional, and defaults to the message type specified in the transport. Legacy protocols such as `fix`, `nvfix` and `xml` only accept a single message type, and therefore do not require or accept a message type in the connection string. As an example, a connection string such as: ``` tcp://localhost:9007/amps/json ``` would work for programs connecting from the local host to a `Transport` configured as follows: ```xml showLineNumbers ... any-tcp tcp 9007 amps ... ``` See the [Configuring Transports](/docs/amps-user-guide/transports/configuring-transports) section in the _AMPS User Guide_ for more information on configuring transports. ## Using zlib Compression The AMPS C++ client supports enabling zlib compression by adding the `compression=zlib` URI parameter to the connection string. For example: ``` tcp://localhost:9007/amps/json?compression=zlib ``` No server-side configuration changes are required. The client enables zlib compression for the connection based on the URI parameter. If the connection string already contains other URI parameters, add `compression=zlib` using `&`: ``` tcp://localhost:9007/amps/json?=&compression=zlib ``` --- # Content Filtering One of the most powerful features of AMPS is content filtering. With content filtering, filters based on message content are applied at the server, so that your application and the network are not utilized by messages that are uninteresting for your application. For example, if your application is only displaying messages from a particular user, you can send a content filter to the server so that only messages from that particular user are sent to the client. The [Filtering Subscriptions by Content](/docs/amps-user-guide/pub-sub/content) under the [AMPS User Guide](/docs/amps-user-guide/) provides full details on content filtering. The [AMPS Expressions](/docs/amps-user-guide/amps-expressions) section of the AMPS User Guide describes the basic syntax of AMPS expressions, including content filters, and the [AMPS Functions](/docs/amps-user-guide/amps-functions) section describes functions that are available for use in filters. To apply a content filter to a subscription, simply pass it into the `client.subscribe()` call: ```cpp showLineNumbers for (auto message : ampsClient.subscribe("messages", 0, "/sender = 'mom'")) { // process messages from mom } ``` In this case, the application will receive messages where the `sender` field equals `mom`. In this example, we have passed in a content filter `/sender = 'mom'`. This will result in the server only sending us messages, from the `messages` topic, that have the sender field equal to `mom` in the message. For example, the AMPS server will send the following message, where `/sender` is `mom`: ```json showLineNumbers { "sender" : "mom", "text" : "Happy Birthday!", "reminder" : "Call me Thursday!" } ``` The AMPS server will not send a message with a different `/sender` value: ```json showLineNumbers { "sender" : "henry dave", "text" : "Things do not change; we change." } ``` --- # Controlling Blocking with Command Timeout The named convenience methods and the `Command` class provide a `timeout` setting that specifies how long the command should wait to receive a `processed` acknowledgment from AMPS. This can be helpful in cases where it is important for the caller to limit the amount of time to block waiting for AMPS to acknowledge the command. If the AMPS client does not receive the processed acknowledgment within the specified time, the client sends an `unsubscribe` command to the server to cancel the command and throws an exception. Acknowledgments from AMPS are processed by the client receive thread on the same socket as data from AMPS. This means that any other data previously returned (such as the results of a large query) must be consumed before the acknowledgment can be processed. An application that submits a set of SOW queries in rapid succession should set a timeout that takes into account the amount of time required to process the results of the previous query. --- # AMPS Programming: Working with Commands The AMPS clients provide named convenience methods for core AMPS functionality. These named methods work by creating messages and sending those messages to AMPS. All communication with AMPS occurs through messages. You can use the `Command` object to customize the messages that AMPS sends. This is useful for more advanced scenarios where you need precise control over AMPS, in cases where you need to use an earlier version of the client to communicate with a more recent version of AMPS, or in cases where a named method is not available. ## Understanding AMPS Messages AMPS messages are represented in the client as `AMPS.Message` objects. The `Message` object is generic, and can represent any type of AMPS message, including both outgoing and incoming messages. This section includes a brief overview of elements common to AMPS command messages. Full details of commands to AMPS are provided in the *AMPS Command Reference* (linked at the bottom of this page). All AMPS command messages contain the following elements: - **Command** - The *command* tells AMPS how to interpret the message. Without a command, AMPS will reject the message. Examples of commands include `publish`, `subscribe`, and `sow`. - **CommandId** - The *command ID*, together with the name of the client, uniquely identifies a command to AMPS. The command ID can be used later on to refer to the command or the results of the command. For example, the command ID for a `subscribe` message becomes the identifier for the subscription. The AMPS client provides a command ID when the command requires one and no command ID is set. Most AMPS messages contain the following fields: - **Topic** - The *topic* that the command applies to, or a regular expression that identifies a set of topics that the command applies to. For most commands, the topic is required. Commands such as `logon`, `start_timer`, and `stop_timer` do not apply to a specific topic, and do not need this field. - **Ack Type** - The *ack type* tells AMPS how to acknowledge the message to the client. Each command has a default acknowledgment type that AMPS uses if no other type is provided. - **Options** - The `options` are a comma-separated list of options that affect how AMPS processes and responds to the message. Beyond these fields, different commands include fields that are relevant to that particular command. For example, SOW queries, subscriptions, and some forms of SOW deletes accept the **Filter** field, which specifies the filter to apply to the subscription or query. As another example, publish commands accept the **Expiration** field, which sets the SOW expiration for the message. For full details on the options available for each command and the acknowledgment messages returned by AMPS, see the *AMPS Command Reference*. ## Creating and Populating the Command To create a command, you simply construct a command object of the appropriate type: ```cpp AMPS::Command command("sow"); ``` Once created, you set the appropriate fields on the command. For example, the following code creates a SOW query, setting the command, topic and filter for the query: ```cpp showLineNumbers AMPS::Command command("sow") .setTopic("messages-sow") .setFilter("/id > 20"); ``` When sent to AMPS using the `execute()` method, AMPS performs a SOW query from the topic `messages-sow` using a filter of `/id > 20`. The results of sending this message to AMPS are no different than using the form of the `sow` method that sets these fields. ## Using Execute Once you've created a command, use the `execute` method to send the command to AMPS. The `execute` method returns a `MessageStream` that provides response messages. The `executeAsync` method sends the command to AMPS, waits for a `processed` acknowledgment, then returns. Messages are processed on the client background thread. For example, the following snippet sends the command created above: ```cpp client.execute(command); ``` You can also provide a message handler to receive acknowledgments, statistics, or the results of subscriptions and SOW queries. The AMPS client maintains a background thread that receives and processes incoming messages. The call to `executeAsync` returns on the main thread as soon as AMPS acknowledges the command as having been processed, and messages are received and processed on the background thread: ```cpp showLineNumbers void handleMessages(const AMPS::Message& m, void* user_data) { /* print acknowledgment type and reason for sample purposes.*/ std::cout << m.getAckType() << " : " << m.getReason() << std::endl; } ... client.executeAsync(command, AMPS::MessageHandler(handleMessages, NULL)); ... ``` While this message handler simply prints the ack type and reason for sample purposes, message handlers in production applications are typically designed with a specific purpose. For example, your message handler may fill a work queue, or check for success and throw an exception if a command failed. ### Using Execute To Publish Notice that the `publish` command does not typically provide return results other than acknowledgment messages, so there is little need for a message handler with a `publish` command. To send a `publish` command, use the `executeAsync()` method with a default-constructed message handler. With a default-constructed message handler, AMPS does not enter the message handler in the internal routing table, which improves efficiency for commands that do not expect a response: ```cpp client.executeAsync(publishCmd, AMPS::MessageHandler()); ``` A default-constructed message handler has an empty implementation and does not receive acknowledgments. To detect write failures, set the `FailedWriteHandler` on the client. ## AMPS Command Cookbook The [AMPS Command Reference](/docs/amps-command-reference) includes information on which fields and options to set on commands to get a specific result. The reference includes both reference information and a [Command Cookbook](/docs/amps-command-reference/cookbook) that provides a concise guide for commonly-used commands. --- # Providing Credentials to AMPS When a client logs on to AMPS, the client sends AMPS a username and password. The username is derived from the URI, using the standard syntax for providing a user name in a URI, for example, `tcp://JohnDoe:@server:port/amps/messagetype` to include the user name `JohnDoe` in the request. For a given user name, the password is provided by an `Authenticator`. The AMPS client distribution includes a `DefaultAuthenticator` that simply returns the password, if any, provided in the URI. A `logon()` command that does not specify an `Authenticator` will use an instance of `DefaultAuthenticator`. If your authentication system requires a different authentication token, you can implement an `Authenticator` that provides the appropriate token. ## Providing Credentials in a Connection String When using the `DefaultAuthenticator`, the AMPS clients support the standard format for including a username and password in a URI, as shown below: ```bash tcp://user:password@host:port/protocol/message_type ``` When provided in this form, the default authenticator provides the username and password specified in the URI. If you have implemented another authenticator, that authenticator controls how passwords are provided to the AMPS server. --- # Delta Publish To delta publish, you use the `delta_publish` command as follows: ```cpp showLineNumbers /* assumes that client is connected and logged on */ String msg = ... ; // obtain changed fields here client.deltaPublish("myTopic", msg); ``` The message that you provide to AMPS must include the fields that the topic uses to generate the SOW key. Otherwise, AMPS will not be able to identify the message to update. For SOW topics that use a User-Generated SOW Key, use the `Command` form of `delta_publish` to set the `SowKey`. ```cpp showLineNumbers /* assumes that client is connected and logged on */ String msg = ... ; // obtain changed fields here String key = ... ; // obtain user-generated SOW key Command cmd("delta_publish"); cmd.setTopic("delta_topic"); cmd.setSowKey(key); cmd.setData(msg); /* Execute the delta publish. Use an empty * a message handler since any failure acks will * be routed to the FailedWriteHandler */ client.executeAsync(cmd,MessageHandler()); ``` The [AMPS User Guide](/docs/amps-user-guide) section on making [Incremental Message Updates](/docs/amps-user-guide/delta-publish) describes how the AMPS server processes the `delta_publish` command. --- # Delta Subscribe To delta subscribe, you simply use the `delta_subscribe` command as follows: ```cpp showLineNumbers // assumes that client is connected and logged on Command cmd("delta_subscribe"); cmd.setTopic("delta_topic"); cmd.setFilter("/thingIWant = 'true'"); for (auto m : client.execute(cmd)) { // Delta messages arrive here } ``` As described in the [AMPS User Guide](/docs/amps-user-guide) section on [Receiving Only Updated Fields](/docs/amps-user-guide/delta-subscribe), messages provided to a delta subscription will contain the fields used to generate the SOW key and any changed fields in the message. Your application is responsible for choosing how to handle the changed fields. --- # Delta Publish and Subscribe Delta messaging in AMPS has two independent aspects: - **Delta Subscribe** - Allows subscribers to receive just the fields that are updated within a message. - **Delta Publish** - Allows publishers to update and add fields within a message by publishing only the updates into the SOW. This chapter describes how to create delta publish and delta subscribe commands using the AMPS C++ client. For a discussion of this capability, how it works, and how message types support this capability see the [AMPS User Guide](/docs/amps-user-guide). --- # Detecting Write Failures The `publish` methods in the C++ client deliver the message to be published to AMPS and then return immediately, without waiting for AMPS to return an acknowledgment. Likewise, the `sowDelete` methods request deletion of SOW messages, and return before AMPS processes the message and performs the deletion. This approach provides high performance for operations that are unlikely to fail in production. However, this means that the methods return before AMPS has processed the command, without the ability to return an error in the event that the command fails. The AMPS C++ client provides a `FailedWriteHandler` that is called when the client receives an acknowledgment that indicates a failure to persist data within AMPS. To use this functionality, you implement the `FailedWriteHandler` interface, construct an instance of your new class, and register that instance with the `setFailedWriteHandler()` function on the client. When an acknowledgment returns that indicates a failed write, AMPS calls the registered handler method with information from the acknowledgment message, supplemented with information from the client publish store if one is available. Your client can log this information, present an error to the user, or take whatever action is appropriate for the failure. If your application needs to know whether publishes succeeded and are durably persisted, the following approach is recommended: - Set a `PublishStore` on the client. This will ensure that messages are retransmitted if the client becomes disconnected before the message is acknowledged *and* request `persisted` acknowledgments for messages. - Install a `FailedWriteHandler`. In the event that AMPS reports an error for a given message, that event will be reported to the `FailedWriteHandler`. - Call `publishFlush()` and verify that all messages are persisted before the application exits. When no `FailedWriteHandler` is registered, acknowledgments that indicate errors in persisting data are treated as unexpected messages and routed to the `LastChanceMessageHandler`. In this case, AMPS provides only the acknowledgment message and does not provide the additional information from the client publish store. --- # Disconnect Handling Every distributed system will experience occasional disconnections between one or more nodes. The reliability of the overall system depends on an application’s ability to efficiently detect and recover from these disconnections. Using the AMPS C/C++ client’s disconnect handling, you can build powerful applications that are resilient in the face of connection failures and spurious disconnects. --- # Error Handling In every distributed system, the robustness of your application depends on its ability to recover gracefully from unexpected events. The AMPS client provides the building blocks necessary to ensure your application can recover from the kinds of errors and special events that may occur when using AMPS. --- # Examples The AMPS C++ Client includes a set of example programs that provide simple demonstrations of client functionality. The sample archive is available that includes a set of samples and a configuration file for AMPS: **[cpp-examples.zip](./examples/cpp-examples.zip)** :::tip Examples may need to be updated with the IP address or DNS name of the host running AMPS unless you are running both the samples and the AMPS server on the same system. ::: The archive includes a simple makefiles. It does not include the C++ client distribution, which can be downloaded from the [60East C++ developer page](https://www.crankuptheamps.com/documentation/client-apis/cpp/). The samples archive includes samples such as: | Sample Name | Demonstrates | |---------------|---------------| | `amps_subscribe.cpp` | Simple subscriber to an adhoc topic. | | `amps_publish.cpp` | Simple publisher to an adhoc topic. | | `amps_publish_sow.cpp` | Simple publisher targeting a topic in the State of the World | | `amps_query_sow.cpp` | Point in time query of a topic in the State of the World | | `amps_sow_and_subscribe.cpp` | Point in time query of and ongoing subscription to a topic in the State of the World | | `amps_sow_and_subscribe_with_oof.cpp` | Point in time query of and ongoing subscription to a topic in the State of the World. This subscription also requests out of focus notifications if a message is deleted or no longer matches the subscription | | `amps_publish_for_replay.cpp` | Simple publisher targeting a topic in the transaction log | | `amps_subscribe_with_replay.cpp` | Subscriber requesting a replay from the transaction log (bookmark subscribe) | | `amps_publish_queue.cpp` | Simple publisher targeting a queue topic (which must also be in the transaction log) | | `amps_consume_queue.cpp` | Subscriber that consumes from a queue | | `amps_fix_builder_publisher.cpp` | Subscriber that uses the provided convenience class to create FIX messages | | `amps_fix_shredder_subscriber.cpp` | Subscriber that uses the provided convenience class to parse FIX messages | | `amps_nvfix_builder_publisher.cpp` | Subscriber that uses the provided convenience class to create NVFIX messages | | `amps_nvfix_shredder_subscriber.cpp` | Subscriber that uses the provided convenience class to parse NVFIX messages | | `amps_publish_composite.cpp` | Publisher that uses the provided convenience class to create a composite message | | `amps_subscribe_composite.cpp` | Subscriber that uses the provided convenience class consume composite messages | --- # Exception Handling and Asynchronous Message Processing When using asynchronous message processing, exceptions thrown from the message handler are silently absorbed by the AMPS C++ client by default. The AMPS C++ client allows you to register an exception listener to detect and respond to these exceptions. When an exception listener is registered, AMPS will call the exception listener with the exception. See the section on [Unhandled Exceptions](unhandled-exceptions) for details. --- # Exception Types Each method in AMPS documents the kinds of exceptions that it can throw. The following table details each of the exception types thrown by AMPS. They are all declared in the `AMPS` namespace, and all of these types publicly inherit from class `std::runtime_error`. | Exception | When | Notes | | ---------------------------------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AlreadyConnectedException` | Connecting | Thrown when `connect()` is called on a `Client` that is already connected. | | `AMPSException` | Anytime | Base class for all AMPS exceptions. | | `AuthenticationException` | Connecting | Indicates an authentication failure occurred on the server. | | `BadFilterException` | Subscribe or Query | This typically indicates a syntax error in a filter expression. | | `BadRegexTopicException` | Subscribe or Query | Indicates a malformed regular expression was found in the topic name. | | `BadSowKeyException` | Subscribing, Publishing, Deleting | Raised when a command uses an invalid SOW key. | | `CommandException` | Anytime | Base class for all exceptions relating to commands sent to AMPS. | | `ConnectionException` | Anytime | Base class for all exceptions relating to the state of the AMPS connection. | | `ConnectionRefusedException` | Connecting | The connection was actively refused by the server. Validate that the server is running, that network connectivity is available, and the settings on the client match those on the server. | | `DisconnectedException` | Anytime | No connection is available when AMPS needed to send data to the server or the user's disconnect handler threw an exception. | | `DuplicateLogonException` | Connecting | A client tried to logon after already logging on. | | `InvalidBookmarkException` | Subscribe or Query | Command specifies an invalid bookmark. | | `InvalidOptionsException` | Subscribe or Query | Command specifies invalid options. | | `InvalidOrderByException` | Query | Command specifies an invalid `orderby` clause. | | `InvalidSubIdException` | Subscribe or Query | Command specifies an invalid `subid`. | | `InvalidTopicException` | Subscribe or Query | The topic is not configured for the requested operation. For example, a `sow` command was issued for a topic that is not in the SOW or a bookmark subscribe was issued for a topic that is not recorded in the transaction log. | | `InvalidURIException` | Connecting | The URI string provided to `connect()` was formatted improperly. | | `LogonRequiredException` | Anytime | A client attempted to execute a command before calling logon. | | `MessageStreamFullException` | Internal client use | Indicates that a `MessageStream` has reached its max depth and can't enqueue another message. | | `MissingFieldsException` | Subscribe or Query | Thrown when a command is missing required fields. | | `NameInUseException` | Connecting | The client name (specified when instantiating `Client`) is already in use on the server. | | `NotEntitledException` | Connecting, Subscribe or Query | An authenticated client attempted to access a resource to which the user has not been granted proper entitlements. | | `PublishException` | Publishing | A client attempted to publish an invalid message or some other error occurs with the message. Requested operation. | | `PublishStoreGapException` | Connecting | The client attempted to logon to a server that appears to be missing messages from this client that are no longer in the publish store. | | `ReconnectMaximumExceededException` | Connecting | The maximum allowed time for attempting to connect to the server has been exceeded. | | `RetryOperationException` | Anytime | An error occurred that caused processing of the last command to be aborted. Try issuing the command again. | | `StoreException` | Publishing, Subscribing | Thrown when a publish store or bookmark store experiences some internal error (such as lack of resources or permissions), or is in an improper state for the requested operation. | | `SubidInUseException` | Subscribing | Indicates a subscription has been placed with the same subscription ID. | | `SubscriptionAlreadyExistsException` | Subscribing | A subscription has been requested using the same `CommandId` as another subscription. Create a unique `CommandId` for every subscription. | | `TimedOutException` | Anytime | A timeout occurred waiting for a response to a command. | | `TransportTypeException` | Connecting | Thrown when a transport type was selected in the URI that is unknown to AMPS. | | `UnknownException` | Anytime | Thrown when an internal error occurs. Contact AMPS support immediately. | | `UsageException` | Changing the properties of an object. | Thrown when the object is not in a valid state for setting the properties. For example, some properties of a `Client` (such as the `BookmarkStore` used) cannot be changed while that client is connected to AMPS. | --- # Exceptions Generally speaking, when an error occurs that prohibits an operation from succeeding, AMPS will throw an exception. AMPS exceptions universally derive from `AMPS::AMPSException`, so by catching `AMPSException`, you will be sure to catch anything AMPS throws. For example: ```cpp showLineNumebrs void ReadAndEvaluate(Client& client) { /* read a new payload from the user */ string payload; getline(cin, payload); /* write a new message to AMPS */ if (!payload.empty()) { try { client.publish("UserMessage", string("{ \"message\" : \"data\" }")); } catch (const AMPSException& exception) { cerr << "An AMPS exception occurred: "<< exception.toString() << endl; } } } ``` In this example, if an error occurs, the program writes the error to `stderr` and the `publish()` command fails. However, client is still usable for continued publishing and subscribing. When the error occurs, the exception is written to the console, converting the exception to a string via the `toString()` method. AMPS exception types vary based on the nature of the error that occurs. In your program, if you would like to handle certain kinds of errors differently than others, you can catch the appropriate subclass of `AMPSException` to detect those specific errors and do something different. ```cpp showLineNumbers string CreateNewSubscription(Client& client) { string id; string topicName; while (id.empty()) { topicName = AskUserForTopicName(); try { /* If an error occurs when setting up the subscription, whether or not to * try again depends on the subclass of AMPSException that is thrown. If a * BadRegexTopicException is thrown, it means that a bad regular expression * was supplied during subscription. In this case, we would like to give the * user a chance to correct the issue. */ id = client.subscribe(bind(HandleMessage, placeholders::_1), topicName, 5000); } catch(const BadRegexTopicException& ex) { /* This line indicates that the program catches the BadRegexTopicException * and displays a specific error to the user indicating the topic name or * expression was invalid. By not returning from the function in this catch * block, the while loop runs again and the user is asked for another topic * name. */ DisplayError("Error: bad topic name or regular " + "expression '" + topicName +"'. " + "The error was: " + ex.toString()); } /* If an AMPS exception of a type other than BadRegexTopicException is thrown by * AMPS, it is caught here. In that case, the program emits a different error * message to the user. */ catch(const AMPSException& ex) { DisplayError("Error: error setting up subscription " + "to topic " + topicName + ". The error was: " + ex.toString()); /* At this point the code stops attempting to subscribe to the client by the return * NULL statement. */ return NULL; // give up } } return id; } ``` --- # Changing the Filter on a Subscription AMPS allows you to update parameters, such as the content filter, on a subscription. When you replace a filter on the subscription, AMPS immediately begins sending only messages that match the updated filter. Notice that if the subscription was entered with a command that includes a SOW query, using the `replace` option can re-issue the SOW query (as described in the *AMPS User Guide*). To update the filter on a subscription, you create a `subscribe` command. You set the `SubscriptionId` provided on the `Command` to the identifier of the existing subscription and include the `replace` option on the `Command`. When you send the `Command`, AMPS atomically replaces the filter and sends messages that match the updated filter from that point forward. --- # Your First AMPS Program In this chapter, we will learn more about the structure and features of the AMPS C/C++ library, and build our first C/C++ program using AMPS. ## Connecting to AMPS Let’s begin by writing a simple program that connects to an AMPS server and sends a single message to a topic: ```cpp showLineNumbers #include #include int main(void) { const char* uri = "tcp://127.0.0.1:9007/amps/json"; // Construct a client with the name "examplePublisher". AMPS::Client ampsClient("examplePublisher"); try { // connect to the server and log on ampsClient.connect(uri); ampsClient.logon(); // publish a JSON message ampsClient.publish("messages", R"({ "message" : "Hello, World!" ,)" R"("client" : 1 })"); } catch (const AMPS::AMPSException& e) { std::cerr << e.what() << std::endl; exit(1); } return 0; } ``` In the example above, we show the entire program. Future examples will isolate one or more specific portions of the code. The next section describes how to build and run the application and explains the code in further detail. ### Build and run To build the program that you've created: - Create a new `.cpp` file and use your c compiler to build it, making sure the `amps-c++-client/include` directory is in your compiler’s `include` path - Link to the `libamps.a` or `amps.lib` static libraries. - Additionally, link to any operating system libraries required by AMPS; a full list may be found by examining the Makefile and project files in the `samples` directory. If the message is published successfully, there is no output to the console. We will demonstrate how to create a subscriber to receive messages in [Subscriptions](subscriptions) section. ### Examining the code: Let us revisit the code we listed earlier: ```cpp showLineNumbers /* These are the include files required for an AMPS C++ Client. The * first is . This header includes everything needed to * compile C++ programs for AMPS. The next include is the Standard C++ Library * , necessary due to use of std::cerr and std::endl. */ #include #include int main() { /* The URI to use to connect to AMPS. The URI consists of the transport, * the address, and the protocol to use for the AMPS connection. In this case, * the transport is tcp, the address is 127.0.0.1:9007, and the protocol is * amps. In this case, AMPS is configured to allow any message type on that * transport, so we specify json in the URI to let AMPS know which message * type this connection will use. Even though a transport that uses the * amps protocol can accept multiple message types, each connection must specify * the exact message type that connection will use. Check with the person who * manages the AMPS instance to get the connection string to use for your programs. */ const char* uri = "tcp://127.0.0.1:9007/amps/json"; /* This is where we first interact with AMPS by instantiating an AMPS::Client * object. Client is the class used to connect to and interact with an AMPS * server. We pass the string "exampleClient" as the clientName. This name * will be used to uniquely identify this client to the server. Errors relating * to this connection will be logged with reference to this name, and AMPS uses * this name to help detect duplicate messages. AMPS enforces uniqueness for * client names when a transaction log is configured, and it is good practice * to always use unique client names. */ AMPS::Client ampsClient("exampleClient"); /* Here we open a try block. AMPS C++ classes throw exceptions to indicate * errors. For the remainder of our interactions with AMPS, if an * error occurs, the exception thrown by AMPS will be caught and handled * in the exception handler below. */ try { /* At this point, we establish a valid AMPS network connection and * can begin to use it to publish and subscribe to messages. In this * example, we use the URI specified earlier in the file. If any errors * occur while attempting to connect to AMPS, the connect() method will * throw an exception. */ ampsClient.connect(uri); /* The AMPS logon() command connects to AMPS and creates a named * connection. This version of the logon() command uses a DefaultAuthenticator, * which uses the credentials provided in the URI. Without credentials, the * client logs on to AMPS anonymously. AMPS versions 5.0 and later * require a logon() command in the default configuration. * * If you need to use a different authentication scheme, implement an * Authenticator and pass that Authenticator to this command. */ ampsClient.logon(); /* publish a JSON message * Here, a single message is published to AMPS on the messages topic, * containing the data Hello world. This data is placed into an XML * message and sent to the server. Upon successful completion of this * function, the AMPS client has sent the message to the server, and * subscribers to the messages topic will receive this Hello world message. */ ampsClient.publish("messages", R"({ "message" : "Hello, World!" ,)" R"( "client" : 1 })"); } /* Error handling begins with the catch block. All exceptions thrown by * AMPS derive from AMPSException classes. More specific exceptions may * be caught to handle certain conditions, but catching * AMPSException& allows us to handle all AMPS errors in one * place. In this example, we print out the error to the console and exit * the program. */ catch (const AMPS::AMPSException& e) { std::cerr << e.what() << std::endl; exit(1); } /* At this point we return from main() and our ampsClient object falls * out of scope. When this happens AMPS automatically disconnects from the * server and frees all of the client resources associated with the * connection. In the AMPS C++ client, objects are reference-counted, * meaning that you can safely copy a client, for example, and destroy copies * of client without worrying about premature closure of the server connection * or memory leaks. */ return 0; } ``` :::tip ### About Authentication When a client logs on to AMPS, the client sends AMPS a username and password. The username is derived from the URI, using the standard syntax for providing a user name in a URI, for example, `tcp://JohnDoe:@server:port/amps/messagetype` to include the user name `JohnDoe` in the request. For a given user name, the password is provided by an `Authenticator`. The AMPS client distribution includes a `DefaultAuthenticator` that simply returns the password, if any, provided in the URI. A `logon()` command that does not specify an `Authenticator` will use an instance of `DefaultAuthenticator`. If your authentication system requires a different authentication token, you can implement an `Authenticator` that provides the appropriate token. ::: --- # Using a Heartbeat to Detect Disconnection The AMPS client includes a heartbeat feature to help applications detect disconnection from the server within a predictable amount of time. Without using a heartbeat, an application must rely on the operating system to notify the application when a disconnect occurs. For applications that are simply receiving messages, it can be impossible to tell whether a socket is disconnected or whether there are simply no incoming messages for the client. When you set a heartbeat, the AMPS client sends a heartbeat message to the AMPS server at a regular interval, and expects a response from the server within the specified amount of time. If the operating system reports an error on send, or if there is no activity received from the server within the specified amount of time, the AMPS client considers the server to be disconnected. Likewise, the server will ensure that traffic is sent to the client at the specified interval, using heartbeat messages when no other traffic is being sent to the client. If, after sending a heartbeat message, no traffic from the client arrives within a period twice the specified interval, the server will consider the client to be disconnected or nonresponsive. The AMPS client processes heartbeat messages on the client receive thread, which is the thread used for asynchronous message processing. If your application uses asynchronous message processing and occupies the thread for longer than the heartbeat interval, the client may fail to respond to heartbeat messages in a timely manner and may be disconnected by the server. --- # High Availability The AMPS C++ Client provides an easy way to create highly-available applications using AMPS, via the `HAClient` class. `HAClient` derives from `Client` and offers the same methods, but also adds protection against network, server, and client outages. Using `HAClient` allows applications to automatically: - Recover from temporary disconnects between client and server. - Failover from one server to another when a server becomes unavailable. Since the `HAClient` automatically manages failover and reconnection, 60East recommends using the `HAClient` for applications that need to: - Automatically reconnect and resume work in the case of disconnection. - Ensure no messages are lost or duplicated after a reconnect or failover. - Persist messages and the current state of a bookmark subscription on disk for protection against client failure. You can choose how your application uses `HAClient` features. For example, you might need automatic reconnection, but have no need to resume subscriptions or republish messages. The high availability behavior in `HAClient` is provided by implementations of defined interfaces. You can combine different implementations provided by 60East to meet your needs, and implement those interfaces to provide your own policies. Some of these features require specific configuration settings on your AMPS instance(s). This chapter mentions these features and describes how to use them from the AMPS C++ client. You can find full documentation for these settings and server features in the [AMPS User Guide](/docs/amps-user-guide). ## Overview of HAClient `HAClient` derives from `Client` and offers the same methods for sending commands to AMPS and receiving messages from AMPS. The `HAClient` differs from the `Client` in two ways: - The `HAClient` automatically installs a disconnect handler that reconnects to AMPS and resumes active (asynchronous) subscriptions. The disconnect handler optionally replays `publish` and `sow_delete` messages that have not been acknowledged by AMPS, using a `PublishStore`. The disconnect handler can optionally resume replays from the transaction log at a point that guarantees no messages are skipped and no duplicates are delivered to the application, using a `BookmarkStore`. - The `HAClient` includes the infrastructure needed for client failover, including a list of connection strings and their associated authentication mechanisms (provided by the `ServerChooser`), and options for controlling backoff behavior for reconnects (provided by the `DelayStrategy`). As a result, the `HAClient` provides a `connectAndLogon()` function for establishing a connection to AMPS, rather than treating these as independent steps that an application must manage itself. If your application needs to automatically reconnect to AMPS, 60East recommends using the `HAClient` and the automatically provided disconnect handler rather than using a `Client` or replacing the `HAClient` default disconnect handler. ## Reconnection with HAClient The most important difference between `Client` and `HAClient` is that `HAClient` automatically provides a reconnect handler. This description provides a high-level framework for understanding the components involved in failover with the `HAClient`. The components are described in more detail in the following sections. The `HAClient` reconnect handler performs the following steps when reconnecting: 1. Calls the `ServerChooser` to determine the next URI to connect to and the authenticator to use for that connection. If the connection fails, calls `get_error` on the `ServerChooser` to get a description of the failure, sends an exception to the exception listener, and stops the reconnection process. 2. Calls the `DelayStrategy` to determine how long to wait before attempting to reconnect, and waits for that period of time. 3. Connects to the AMPS server. If the connection fails, calls `reportFailure` on the `ServerChooser` and begins the process again. 4. Logs on to the AMPS server. If the connection fails, calls `reportFailure` on the `ServerChooser` and begins the process again. 5. Calls `reportSuccess` on the `ServerChooser`. 6. Receives the bookmark for the last message that the server has persisted. Discards any older messages from the `PublishStore`. 7. Republishes any messages in the `PublishStore` that have not been persisted by the server. 8. Re-establishes subscriptions using the `SubscriptionManager` for the client. For bookmark subscriptions, the reconnect handler uses the `BookmarkStore` for the client to determine the most recent bookmark, and re-subscribes with that bookmark. For subscriptions that do not use a bookmark, the `SubscriptionManager` simply re-enters the subscription, meaning that it is entered at the point at which the `HAClient` reconnects. The `ServerChooser`, `DelayStrategy`, `PublishStore`, `SubscriptionManager`, and `BookmarkStore` are all extension points for the `HAClient`. You can adapt the failover and recovery behavior by setting a different object for the behavior you want to customize on the `HAClient` or by providing your own implementation. For example, the convenience methods in the previous section customize the behavior of the `PublishStore` and `BookmarkStore` by providing either memory-backed or file-backed stores. ## Choosing Store Durability If your application needs reliable publish to AMPS, install a `PublishStore` in the `HAClient`. If your application needs to resume replays from the transaction log, install a `BookmarkStore` in the `HAClient`. These stores provide the following capabilities: - A *bookmark store* tracks received messages, and is used to resume subscriptions that replay from the transaction log. - A *publish store* tracks published messages, and is used to ensure that messages are persisted in AMPS. The AMPS C++ client provides a memory-backed version of each store and a file-backed version of each store. An `HAClient` can use either a memory backed store or a file backed store for protection. Each method provides resilience to different failures, as described below: - *Memory-backed stores* provide recovery after disconnection from AMPS by storing messages and bookmarks in your process' address space. This is the highest performance option for working with AMPS in a highly available manner. The trade-off with this method is there is no protection from a crash or failure of your client application. If your application is terminated prematurely or, if the application terminates at the same time as an AMPS instance failure or network outage, then messages may be lost or duplicated. The state of bookmark replays will be lost when the application shuts down. Messages in the publish store when the application shuts down will not be maintained through a restart, so the application will not be able to attempt any necessary redelivery when the application restarts. A memory-backed store should only be used by one instance of a client at a time. - *File-backed stores* provide recovery after client failure or shutdown and disconnection from AMPS by storing messages and bookmarks on disk. To use this protection method, the `createFileBacked` convenience method requests additional arguments for the two files that will be used for both bookmark storage and message storage. If these files exist and are non-empty (as they would be after a client application is restarted), the `HAClient` loads their contents and ensures synchronization with the AMPS server once connected. The performance of this option depends heavily on the speed of the device on which these files are placed. When the files do not exist (as they would the first time a client starts on a given system), the `HAClient` creates and initializes the files. In this case the client does not have a point at which to resume the subscription or messages to republish. A store file should only be used by one instance of a client at a time. When using file backed bookmark stores, 60East recommends periodically removing unneeded entries by calling the `prune()` method. The precise strategy that your application uses to call `prune()` depends on the nature of the application. Most applications call `prune()` when the application exits. There are two basic strategies that applications follow while the application runs: - Install a resize handler and call `prune()` after a specified number of resize operations, or when the store reaches a specific size. - Call `prune()` after a specific number of messages are processed (for example, every 10,000 messages received or every 1,000 updates completed). Regardless of the strategy, it is best to call `prune()` when the application is otherwise idle, since the `prune()` call rewrites the log file. The store interface is public, and an application can create and provide a custom store as necessary. While clients provide convenience methods for creating file-backed and memory-backed `HAClient` objects with the appropriate stores, you can also create and set the stores in your application code. The AMPS C++ client also includes default stores, which implement the appropriate interface, but do not actually persist messages. Starting in 5.3.2.0, the AMPS client contains a recovery point adapter interface to make it easy to add a custom persistence layer to a bookmark store. The distribution includes a recovery point adapter that can store bookmark recovery information in an AMPS SOW topic. The `HAClient` provides convenience methods for creating clients and setting stores. You can also construct an `HAClient` and set whichever store implementations you choose. In this example, we create several clients. The first client uses memory stores for both bookmarks and publishes. The second client uses files for both bookmarks and publishes. The third client uses a file for bookmarks. The third client does not set a store for publishes, which means that AMPS provides the default store (and no outgoing messages are stored). The final client does not specify any stores, so has no persistence for published messages or bookmark subscriptions, but can take advantage of the automatic failover and reconnection in the `HAClient`. ```cpp showLineNumbers /* Memory publish store, memory bookmark store */ HAClient memoryClient = HAClient::createMemoryBacked("lessImportantMessages"); /* File-backed publish store, file-backed bookmark store */ HAClient diskClient = HAClient::createFileBacked("moreImportantMessages", "/mnt/fastDisk/moreImportantMessages.outgoing", "/mnt/fastDisk/moreImportantMessages.incoming"); /* Default publish store, file-backed bookmark store */ HAClient subscriberClient("subscriber"); subscriberClient.setBookmarkStore( new LoggedBookmarkStore("my_app.bookmark")); /* Default publish store, default bookmark store Failover behavior and resubscription only */ HAClient streamReader("streamReader"); ``` :::tip While this chapter presents the built-in file and memory-based stores, the AMPS C/C++ Client provides open interfaces that allow development of custom persistent message stores. To fully control recovery behavior, you can implement the `Store` and `BookmarkStore` interfaces in your code, and then pass instances of those to `setPublishStore()` or `setBookmarkStore()` methods in your `Client`. You can also implement the `RecoveryPointAdapter` interface to easily add a custom storage mechanism to one of the 60East-provided bookmark store implementations. Instructions on developing a custom store are beyond the scope of this document; please refer to the *AMPS Client HA Whitepaper* for more information. ::: ### Using the SOW Recovery Point Adapter The AMPS client also includes the ability to use a SOW topic to store bookmark state for a bookmark store. This can be a useful option in a situation where an application needs a persistent bookmark store, but does not have the ability to store a file on the filesystem, or where an application has a bookmark file, but wants to have the ability to resume the subscription if the file is lost or damaged, or if the application is started on a system that does not have access to the file. To use the SOW topic recovery point adapter, you create a bookmark store of the type you would like to use for the `Client`, passing an adapter when you construct the store. You then set this bookmark store as the store for the `Client` to use. The constructor for the SOW recovery adapter allows you to customize the topic name and field names used to store the recovery point information in AMPS. As with the `RecoveryPointAdapter` interface in general, it is possible to customize the behavior of the SOW recovery point adapter by overriding the provided methods. This section describes how to use the adapter with the default settings. Should you need to change the behavior of the class, you would adjust the guidance in this section accordingly. (For example, if you override methods to produce a message with a different set of keys or a different message format, you would update the topic definition accordingly). ### AMPS Topic Configuration To store recovery point state in AMPS, the AMPS instance that will store the recovery point state must define a `SOW/Topic` to hold the recovery point data. By default, the adapter uses a topic named `/ADMIN/bookmark_store` of `json` message type, with the `/clientName` and `/subId` fields as keys, similar to the following definition: ```xml showLineNumbers /ADMIN/bookmark_store json /clientName /subId ``` You must include this definition, or an equivalent definition, in the configuration file for the AMPS instance that will host the recovery point. If you define a topic with a different configuration (for example, different key names, a different topic name or a different message type), you must ensure that the adapter that you create uses the same parameters as those configured on the server. ### Constructing a Client for the Adapter The AMPS SOW Recovery Point Adapter requires a `Client` or `HAClient` connected to the instance that contains the SOW topic. The Adapter will use this client to recover bookmark state and store bookmarks in AMPS. Notice that this client **must not** be a client that the Adapter is keeping state for. This must be a completely separate client instance, otherwise the client may deadlock while updating the store. The client must be connected and logged in to the instance that contains the SOW topic, using the message type defined for the topic. ### Capacity Planning and Store Sizing When an application uses a file-backed store, it is important to make sure that there is enough space available on the file system to be able to manage the store. For logged bookmark stores, an application needs to keep a bookmark record for each message received, each message discarded, and the persisted acknowledgments delivered by the server approximately once a second. Each bookmark entry consumes roughly 70 bytes of storage *plus* the length of the subscription ID for the subscription receiving the message. The logged bookmark store retains entries until an application explicitly calls `prune()`. The capacity needed for a logged bookmark store will depend on the strategy that the application uses for pruning the file. For a file-backed publish store, the application needs to be able to store published messages until the AMPS server that the publisher is connected to acknowledges those messages as persisted. The volume of messages that needs to be stored depends on the failover policy for the server -- that is, the maximum amount of time that the server will allow a downstream instance to fail to acknowledge a message before the server downgrades that connection to `async` acknowledgment. By default, AMPS does not downgrade connections: this policy must be set explicitly using the AMPS actions. As an example, if the server is configured to downgrade connections that are more than 120 seconds behind, then -- for disaster recovery -- the application must have the capacity to store 120 seconds of published messages at peak publishing load. However, unlike the logged bookmark store, a file-backed publish store removes messages from the store and reuses the space once AMPS has acknowledged the message. ## Connections and the Server Chooser Unlike `Client`, the `HAClient` attempts to keep itself connected to an AMPS instance at all times, by automatically reconnecting or failing over when it detects that the client is disconnected. When you are using the `Client` directly, your disconnect handler usually takes care of reconnection. `HAClient`, on the other hand, provides a disconnect handler that automatically reconnects to the current server or to the next available server. To inform the `HAClient` of the addresses of the AMPS instances in your system, you pass a `ServerChooser` instance to the `HAClient`. `ServerChooser` acts as a smart enumerator over the servers available: `HAClient` calls `ServerChooser` methods to inquire about what server should be connected, and calls methods to indicate whether a given server succeeded or failed. The AMPS C/C++ Client provides a simple implementation of `ServerChooser`, called `DefaultServerChooser`, that provides very simple logic for reconnecting. This server chooser is most suitable for basic testing, or in cases where an application should simply rotate through a list of servers. For most applications, you implement the `ServerChooser` interface yourself for more advanced logic, such as choosing a backup server based on your network topology, or limiting the number of times your application should try to reconnect to a given address. In either case, you must provide a `ServerChooser` to `HAClient` and then call `connectAndLogon()` to create the first connection. ```cpp showLineNumbers HAClient myClient = HAClient::createMemoryBacked("myClient"); /* primary.amps.xyz.com is the primary AMPS instance, and * secondary.amps.xyz.com is the secondary */ ServerChooser chooser(new DefaultServerChooser()); chooser.add("tcp://primary.amps.xyz.com:12345/fix"); chooser.add("tcp://secondary.amps.xyz.com:12345/fix"); myClient.setServerChooser(chooser); myClient.connectAndLogon(); ... myClient.disconnect(); ``` Similar to `Client`, `HAClient` remains connected to the server until `disconnect()` is called. Unlike `Client`, `HAClient` provides a built-in disconnect handler that automatically attempts to reconnect to your server if it detects a disconnect, and, if that server cannot be connected, fails over to the next server provided by the `ServerChooser`. In this example, the call to `connectAndLogon()` attempts to connect and log in to `primary.amps.xyz.com`, and returns if that is successful. If it cannot connect, it tries `secondary.amps.xyz.com`, and continues trying servers from the `ServerChooser` until a connection is established. Likewise, if it detects a disconnection while the client is in use, `HAClient` attempts to reconnect to the server it was most recently connected with, and, if that is not possible, it moves on to the next server provided by the `ServerChooser`. ### Setting a Reconnect Delay and Timeout You can control the amount of time between reconnection attempts and set a total amount of time for the `HAClient` to attempt to reconnect. The AMPS C++ Client includes an interface for managing this behavior called the `ReconnectDelayStrategy`. Two implementations of this interface are provided with the client: - `FixedDelayStrategy` provides the same delay each time the `HAClient` tries to reconnect. - `ExponentialDelayStrategy` provides an exponential backoff until a connection attempt succeeds. To use either of these classes, you simply create an instance with appropriate parameters, and install that instance as the delay strategy for the `HAClient`. For example, the following code sets up a reconnect delay that starts at 200ms and increases the delay by 1.5 times after each failure. The strategy allows a maximum delay between connection attempts of 5 seconds, and will not retry longer than 60 seconds. ```cpp showLineNumbers HAClient theClient = HAClient::createMemoryBacked("demo"); theClient.setReconnectDelayStrategy( new ExponentialDelayStrategy(200, 5000, 1.5, 6000,0) ); ``` ### Implementing a Server Chooser As described above, you provide the `HAClient` with connection strings to one or more AMPS servers using a `ServerChooser`. The purpose of a `ServerChooser` is to provide information to the `HAClient`. A `ServerChooser` does not manage the reconnection process, and should not call methods on the `HAClient`. A `ServerChooser` has two required responsibilities to the `HAClient`: - Tells the `HAClient` the connection string for the server to connect to. If there are no servers, or the `ServerChooser` wants the connection to fail, the `ServerChooser` returns an empty string. To provide this information, the `ServerChooser` implements the `getCurrentURI()` method. - Provides an `Authenticator` for the current connection string. This is especially important for installations where different servers require different credentials or authentication tokens must be reset after each connection attempt. To provide the authenticator, the `ServerChooser` implements the `getCurrentAuthenticator()` method. The `HAClient` calls the `getCurrentURI()` and `getCurrentAuthenticator()` methods each time it needs to make a connection. Each time a connection succeeds, the `HAClient` calls the `reportSuccess()` method of the `ServerChooser`. Each time a connection fails, the `HAClient` calls the `reportFailure()` method of the `ServerChooser`. The `HAClient` does not require the `ServerChooser` to take any particular action when it calls these methods. These methods are provided for the `HAClient` to do internal maintenance, logging, or record keeping. For example, an `HAClient` might keep a list of available URIs with a current failure count, and skip over URIs that have failed more than 5 consecutive times until all URIs in the list have failed more than 5 consecutive times. When the `ServerChooser` returns an empty string from `getCurrentURI()`, indicating that no servers are available for connection, the `HAClient` calls `getError()` method on the `ServerChooser` and includes the string returned by `getError()` in the generated exception. ## Heartbeats and Failure Detection Use of the `HAClient` allows your application to quickly recover from detected connection failures. By default, connection failure detection occurs when AMPS receives an operating system error on the connection. This system may result in unpredictable delays in detecting a connection failure on the client, particularly when failures in network routing hardware occur, and the client primarily acts as a subscriber. The heartbeat feature of the AMPS client allows connection failure to be detected quickly. Heartbeats ensure that regular messages are sent between the AMPS client and server on a predictable schedule. The AMPS client and server both assume disconnection has occurred if there is no other activity and these regular heartbeats cease, ensuring disconnection is detected in a timely manner. To use the heartbeat feature, call the `setHeartbeat` method on `Client` or `HAClient`: ```cpp showLineNumbers HAClient client = HAClient::createMemoryBacked("importantStuff"); ... client.setHeartbeat(3); client.connectAndLogon(); ... ``` Method `setHeartbeat` takes one parameter: the heartbeat interval. The heartbeat interval specifies the periodicity of heartbeat messages sent by the server: the value `3` indicates messages are sent on a three-second interval. If the client receives no messages in a six second window (two heartbeat intervals), the connection is assumed to be dead, and the `HAClient` attempts reconnection. An additional variant of `setHeartbeat` allows the idle period to be set to a value other than two heartbeat intervals. (The server, however, will always consider a connection to be closed after two heartbeat intervals without any traffic.) Notice that, for `HAClient`, `setHeartbeat` must be called *before* the client is connected. For `Client`, `setHeartbeat` must be called *after* the client is connected. :::warning Heartbeats are serviced on the receive thread created by the AMPS client. Your application must not block the receive thread for longer than the heartbeat interval, or the application is subject to being disconnected. ::: ## Considerations for Publishers Publishing with an `HAClient` is nearly identical to regular publishing; you simply call the `publish()` method with your message’s topic and data. The AMPS client sends the message to AMPS, and then returns from the `publish()` call. For maximum performance, the client does not wait for the AMPS server to acknowledge that the message has been received. When an `HAClient` uses a publish store (other than the `DefaultPublishStore`), the publish store retains a copy of each outgoing message and requests that AMPS acknowledge that the message has been persisted. The AMPS server acknowledges messages back to the publisher. Acknowledgments can be delivered for multiple messages at periodic intervals (for topics recorded in the transaction log) or after each message (for topics that are not recorded in the transaction log). When an acknowledgment for a message is received, the `HAClient` removes that message from the bookmark store. When a connection to a server is made, the `HAClient` automatically determines which messages from the publish store (if any) the server has not processed, and replays those messages to the server once the connection is established. For reliable publishers, the application must choose how best to handle application shutdown. For example, it is possible for the network to fail immediately after the publisher sends the message, while the message is still in transit. In this case, the publisher has sent the message, but the server has not processed it and acknowledged it. During normal operation, the `HAClient` will automatically connect and retry the message. On shutdown, however, the application must decide whether to wait for messages to be acknowledged, or whether to exit. Publish store implementations provide an `unpersistedCount()` method that reports the number of messages that have not yet been acknowledged by the AMPS server. When the `unpersistedCount()` reaches `0`, there are no unpersisted messages in the local publish store. For the highest level of safety, an application can wait until the `unpersistedCount()` reaches `0`, which indicates that all of the messages have been persisted to the instance that the application is connected to, and the synchronous replication destinations configured for that instance. When a synchronous replication destination goes offline, this approach will cause the publisher to wait to exit until the destination comes back online or until the destination is downgraded to asynchronous replication. For applications that are shut down periodically for short periods of time (for example, applications that are only offline during a weekly maintenance window), another approach is to use the `publishFlush()` method to ensure that messages are delivered to AMPS, and then rely on the connection logic to replay messages as necessary when the application restarts. For example, the following code flushes messages to AMPS, then warns if not all messages have been acknowledged: ```cpp showLineNumbers HAClient pub = HAClient.createMemoryBacked("importantStuff"); ... pub.connectAndLogon(); std::string topic = "loggedTopic"; std:string data = ...; for (size_t i = 0; i < MESSAGE_COUNT; i++) { pub.publish(topic, data); } /* We think we are done, but the server may not * have received or acknowledged the messages yet. * Wait until the server has received all messages. * The program could also specify a timeout in this * command to avoid blocking forever if the * network is down or all servers are offline. */ pub.publishFlush(); /* Print warning to the console if messages have * been published but not yet acknowledged as persisted. */ if (pub.getPublishStore().unpersistedCount() > 0) { printf("all messages have been published," " but not all have been persisted."); } pub.disconnect(); ``` In this example, the client sends each message immediately when `publish()` is called. If AMPS becomes unavailable between the final `publish()` and the `disconnect()`, or one of the servers that the AMPS instance replicates to is offline, the client may not have received a persisted acknowledgment for all of the published messages. For example, if a message has not yet been persisted by all of the servers in the replication fabric that are connected with synchronous replication, AMPS will not have acknowledged the message. Before shutting down the client, the code does two things: - First, the code flushes messages to the server to ensure that all messages have been delivered to AMPS. - Next, the code checks to see if all of the messages in the publish store have been acknowledged as persisted by AMPS. If the messages have not been acknowledged, they will remain in the publish store file and will be published to AMPS, if necessary, the next time the application connects. An application may choose to loop until `unpersistedCount()` returns `0`, or (as we do in this case) simply warn that AMPS has not confirmed that the messages are fully persisted. The behavior you choose in your application should be consistent with the high-availability guarantees your application needs to provide. :::warning AMPS uses the name of the `HAClient` to determine the origin of messages. For the AMPS server to correctly identify duplicate messages, each instance of an application that publishes messages must use a distinct name. That name must be consistent across different runs of the application. ::: If your application crashes or is terminated, some published messages may not have been persisted in the AMPS server. If you use the file-based store (in other words, the store created by using `HAClient.createFileBacked()`), then the `HAClient` will recover the messages, and once logged on, correlate the message store to what the AMPS server has received, re-publishing any missing messages. This occurs automatically when `HAClient` connects, without any explicit consideration in your code, other than ensuring that the same file name is passed to `createFileBacked()` if recovery is desired. :::warning AMPS provides persisted acknowledgment messages for topics that do not have a transaction log enabled; however, the level of durability provided for topics with no transaction log is minimal. Learn more about transaction logs in the *AMPS User Guide*. ::: ## Detecting Failover Ahead of Replication AMPS replication provides two different acknowledgment modes for outgoing replication links from an instance: - For a link in `sync` acknowledgment mode, a message must be successfully acknowledged by the downstream instance of AMPS before this instance of AMPS will acknowledge the message. - For a link in `async` acknowledgment mode, this link is not considered for acknowledging the message. In this mode, the downstream side of the replication link may not have received or processed the message at the time that the publisher receives an acknowledgment. As described in the *AMPS User Guide*, a publisher must not failover from one instance of AMPS to another instance when any link between those instances uses `async` acknowledgment *unless* replication is certain to have reached that instance. (For example, if replication is taking a maximum of 1.2 seconds between the instances and the publisher has been disconnected for 30 seconds, all messages from that publisher will have been replicated). To help detect a situation where a publisher may be "jumping ahead" of messages that it has published, but which have not yet been replicated, the AMPS client allows an application to consider it to be an error to make a connection to a server that has not received messages previously published by the application. To enable this behavior, set the `setErrorOnPublishGap()` method to set this property on the `PublishStore` in use for the client. When this property is set, the client will consider it to be an error to connect to a server that has not received messages previously published by the client, and consider the connection to have failed. Notice that an application that uses this approach may need to handle situations where no server has received the message, particularly if the replication configuration uses automated replication downgrade. ## Considerations for Subscribers `HAClient` provides two important features for applications that subscribe to one or more topics: re-subscription, and a bookmark store to track the correct point at which to resume a bookmark subscription. ### Resubscription with Asynchronous Message Processing Any asynchronous subscription placed using an `HAClient` is automatically reinstated after a disconnect or a failover. These subscriptions are placed in an in-memory `SubscriptionManager`, which is created automatically when the `HAClient` is instantiated. Most applications will use this built-in subscription manager, but for applications that create a varying number of subscriptions, you may wish to implement `SubscriptionManager` to store subscriptions in a more durable place. Note that these subscriptions contain no message data, but rather simply contain the parameters of the subscription itself (for instance, the command, topic, message handler, options, and filter). When a re-subscription occurs, the AMPS C++ Client re-executes the command as originally submitted, including the original topic, options, and so on. AMPS sends the subscriber any messages for the specified topic (or topic expression) that are published after the subscription is placed. For a `sow_and_subscribe` command, this means that the client re-issues the full command, including the SOW query as well as the subscription. :::tip A `sow` command is a point-in-time query. It isn't added to the subscription manager, and isn't restarted if a disconnection happens in the middle of a query. A `sow_and_subscribe` is a subscription, and is added to the subscription manager. ::: ### Resubscription with Synchronous Message Processing The `HAClient` (starting with the AMPS C++ Client version 4.3.1.1) does not track synchronous message processing subscriptions in the `SubscriptionManager`. The reason for this is to preserve conventional iterator behavior. That is, once the `MessageStream` indicates that there are no more elements to iterate (for example, because the connection has closed), the `MessageStream` will not suddenly produce more elements. To re-subscribe when the `HAClient` fails over, you can simply re-issue the subscription. For example, the snippet below re-issues a `subscribe` command when the message stream ends: ```cpp showLineNumbers bool still_need_to_process = true; while (still_need_to_process) { try { for ( auto message : client.subscribe("messages")) { /* process messages here */ /* check condition on still_need_to_process */ if (!still_need_to_process) break; } /* end of stream: for a subscribe this means * that the connection is likely closed, or * the program broke out of the loop */ } catch(...) /* for production, you would catch specific errors */ { /* log error as appropriate */ } } ``` ### Bookmark Stores In cases where it is critical not to miss a single message, it is important to be able to resume a subscription at the exact point that a failure occurred. In this case, simply recreating a subscription isn't sufficient. Even though the subscription is recreated, the subscriber may have been disconnected at precisely the wrong time, and will not see the message. To ensure delivery of every message from a topic or set of topics, the AMPS `HAClient` includes a `BookmarkStore` that, combined with the bookmark subscription and transaction log functionality in the AMPS server, ensures that clients receive any messages that might have been missed. The client stores the bookmark associated with each message received, and tracks whether the application has processed that message; if a disconnect occurs, the client uses the `BookmarkStore` to determine the correct resubscription point, and sends that bookmark to AMPS when it re-subscribes. AMPS then replays messages from its transaction log from the point after the specified bookmark, thus ensuring the client is completely up-to-date. `HAClient` helps you to take advantage of this bookmark mechanism through the `BookmarkStore` interface and `bookmarkSubscribe()` method on `Client`. When you create subscriptions with `bookmarkSubscribe()`, whenever a disconnection or failover occurs, your application automatically re-subscribes to the message after the last message it processed. `HAClients` created by `createFileBacked()` additionally store these bookmarks on disk, so that the application can restart with the appropriate message if the client application fails and restarts. To take advantage of bookmark subscriptions, do the following: - Ensure the topic(s) to be subscribed to are included in a transaction log. See the *AMPS User Guide* for information on how to specify the contents of a transaction log. - Use `bookmarkSubscribe()` instead of `subscribe()` when creating a subscription, and decide how the application will manage subscription identifiers (SubIds). If you are using a `Command` object, you can simply provide a bookmark on that object. - Use the `BookmarkStore.discard()` method in message handlers to indicate when a message has been fully processed by the application, that is, when the application does not need to receive the message again if the application fails over. The following example creates a bookmark subscription against a transaction-logged topic, and fully processes each message as soon as it is delivered: ```cpp showLineNumbers HAClient client = HAClient::createFileBacked("theClient", "/logs/theClient.publishLog", "/logs/theClient.subscribeLog"); namespace MyMessageHandler { public void invoke(const Message& message, void* data) { ... client.getBookmarkStore().discard(message); ... } } std::string commandID = client.executeAsync(Command("subscribe") .setTopic("myTopic") .setSubscriptionId("MySubId") .setBookmark(AMPS::Client::BOOKMARK_RECENT()), AMPS::MessageHandler(MyMessageHandler::invoke,(void*)(&client))); ``` In this example, the client is a file-backed client, meaning that arriving bookmarks will be stored in a file (`theClient.subscribeLog`). Storing these bookmarks in a file allows the application to restart the subscription from the last message processed, in the event of either server or client failure. :::info For optimum performance, it is critical to discard every message once its processing is complete. If a message is never discarded, it remains in the bookmark store. During re-subscription, `HAClient` always restarts the bookmark subscription with the oldest undiscarded message, and then filters out any more recent messages that have been discarded. If an old message remains in the store, but is no longer important for the application’s functioning, the client and the AMPS server will incur unnecessary network, disk, and CPU activity. ::: In the example above, all parameters after the bookmark are optional. However, all options before — and including the bookmark — are required when creating a `bookmarkSubscribe()`. The last parameter, `subId`, specifies an identifier to be used for this subscription. Passing `NULL` causes `HAClient` to generate one and return it, like most other `Client` functions. However, if you wish to resume a subscription from a previous point after the application has terminated and restarted, the application must pass the same subscription ID as during its previous run. Passing a different subscription ID bypasses any recovery mechanisms, creating an entirely new subscription. When you use an existing subscription ID, the `HAClient` locates the last-used bookmark for that subscription in the local store, and attempts to re-subscribe from that point. The `subId` is also required to be unique when used within a single client, but can be the same in different clients. Internally, AMPS tracks subscriptions in each client, thus each identifier for each subscription within a client must be unique. The same `subId` can be reused across unique clients simultaneously without causing problems. Below are the different bookmark types that can be used to enable different recovery strategies for an application: - `Client::BOOKMARK_NOW()` specifies that the subscription should begin from the moment the server receives the subscription request. This results in the same messages being delivered as if you had invoked `subscribe()` instead, except that the messages will be accompanied by bookmarks. This is also the behavior that results if you supply an invalid bookmark. - `Client::BOOKMARK_EPOCH()` specifies that the subscription should begin from the beginning of the AMPS transaction log (that is, the first entry in the oldest journal file for the transaction log). - `Client::BOOKMARK_RECENT()` specifies that the subscription should begin from the last-used message in the associated `BookmarkStore`, or, if this subscription has not been seen before, to begin with `EPOCH`. This is the most common value for this parameter, and is the value used in the preceding example. By using `BOOKMARK_RECENT`, the application automatically resumes from wherever the subscription left off, taking into account any messages that have already been processed and discarded. When the `HAClient` re-subscribes after a disconnection and reconnection, it always uses `BOOKMARK_RECENT`, ensuring that the continued subscription always begins from the last message discarded before the disconnect, so that no messages are missed. ## Conclusion With only a few changes, most AMPS applications can take advantage of the `HAClient` and associated classes to become more highly-available and resilient. Using the `PublishStore`, publishers can ensure that every message published has actually been persisted by AMPS. Using `BookmarkStore`, subscribers can make sure that there are no gaps or duplicates in the messages received. `HAClient` makes both kinds of applications more resilient to network and server outages, as well as temporary issues. By utilizing the file-based `HAClient`, clients can recover their state after an unexpected termination or crash. Though `HAClient` provides useful defaults for the `Store`, `BookmarkStore`, `SubscriptionManager`, and `ServerChooser`, you can customize any or all of these to the specific needs of your application and architecture. --- # Obtaining and Installing the AMPS C / C++ Client ## Obtaining the Client The AMPS C/C++ client is available as a download from the [60East Technologies](https://www.crankuptheamps.com/develop/) website. Download the client from the site, then install it on your development computer. The client is packaged into a single file, `amps-c++-client-.tar.gz`, where `` is replaced by the version of the client (such as `amps-c++-client-5.3.0.zip`). In the following examples, the version number is omitted from the filename. Once expanded, the `amps-c++-client` directory will be created, containing sources, samples and makefiles for the C++ client. You’re welcome to locate this directory anywhere that seems convenient; but for the remainder of this book, we’ll simply refer to this directory as the `amps-c++-client` directory. ## Explore the Client The client is organized into a number of directories that you’ll be using through this book. Understanding this organization now will save you time in the future. The top level directories are: ### src Sources and makefile for the AMPS C++ client library. ### include Location of `include` files for C and C++ programs. When building your own program, you’ll add the `include` directory to your `include` path. ### samples Getting started with a new C/C++ library can be challenging. For your reference, we provide a number of small samples, along with a makefile. ## Build the Client After unpacking the `amps-c++-client` directory, you must build the client library for your platform. ### Linux To build on Linux, change to the `amps-c++-client` directory and, from a command prompt, type: ```bash make ``` to make a static library, or ```bash SHARED=1 make ``` to make a shared object. ### Windows On Windows, from a Visual Studio Command Prompt, change to the `amps-c++-client` directory and type: ```bash msbuild ``` Upon successful completion, the AMPS libraries and samples are built in the `lib` and `samples` directories, respectively. ### macOS To build on macOS, change to the `amps-c++-client` directory and, from a command prompt, type: ```bash cmake ``` to make a static library, or ```bash SHARED=1 cmake ``` to make a shared object. ## Test Connectivity to AMPS Before writing programs in AMPS, make sure connectivity to your AMPS development instance is working from your AMPS development environment. Launch a terminal window and change the directory to the AMPS directory in your AMPS installation and use `spark` to test connectivity to your server. For example: ```bash ./bin/spark ping -type fix -server localhost:9004 ``` If `spark` returns an error, verify that your AMPS server is running and that there is no firewall blocking access (including local firewalls between a host instance and virtual machine or container). Without connectivity to AMPS, you will be unable to make best use of this guide. --- # Introduction ## About the C/C++ Client The C/C++ client package includes both the AMPS C++ client and a basic C client that uses only C language features. # C & C++ Support Matrix This version of the AMPS C++ client supports the following operating systems and features: | Feature | Linux x64 / aarch64 | Windows x64 | OSX x64 /aarch64 | | --------------------------------- | ------------------------- | ----------- | ---------------------- | | Incredible performance | X | X | X | | Publish and subscribe | X | X | X | | State of the World (SOW) queries | X | X | X | | Topic and content filtering | X | X | X | | Atomic SOW query and subscribe | X | X | X | | Transaction log replay | X | X | X | | Historical SOW query | X | X | X | | Beautiful documentation | X | X | X | | HA: automatic failover | X | X | X | | HA: durable publish and subscribe | X | X | X | This version of the AMPS C++ client has been tested with the following compilers and versions. Other compilers or versions may work, but have not been tested by 60East: - Linux: gcc 4.8 or later - Windows: Visual Studio versions under current mainstream support - OSX: clang --- # Managing Disconnection The `HAClient` class, included with the AMPS C++ client, contains a disconnect handler and other features for building highly-available applications. The `HAClient` includes features for managing a list of failover servers, resuming subscriptions, republishing in-flight messages, and other functionality that is commonly needed for high availability. 60East recommends using the `HAClient` for automatic reconnection wherever possible, as the HAClient disconnect handler has been carefully crafted to handle a wide variety of edge cases and potential failures. If an application needs to reconnect or fail over, use an `HAClient`, and the AMPS client library will automatically handle failover and reconnection. You control which servers the client fails over to using an implementation of the `ServerChooser` interface, and control the timing of the failover using an implementation of the `ReconnectDelayStrategy` interface. :::info For most applications, the combination of the `HAClient` disconnect handler and a `ConnectionStateListener` gives you the ability to monitor disconnections and add custom behavior at the appropriate point in the reconnection process. ::: If you need to add custom behavior to the failover (such as logging, resetting an internal cache, refreshing credentials and so on), the `ConnectionStateListener` class allows your application to be notified and take action when disconnection is detected and at each stage of the reconnection process. To extend the behavior of the AMPS client during reconnection, implement a `ConnectionStateListener` and `add` it to the set of active listeners using the `addConnectionStateListener` function. --- # Managing SOW Contents AMPS allows applications to manage the contents of the SOW by explicitly deleting messages that are no longer relevant. For example, if a particular delivery van is retired from service, the application can remove the record for the van by deleting the record for the van. The client provides the following methods for deleting records from the SOW. - `sowDelete` - Accepts a filter, and deletes all messages that match the filter. - `sowDeleteByKeys` - Accepts a set of SOW keys as a comma-delimited string and deletes messages for those keys, regardless of the contents of the messages. A SOW key is provided in the header of a SOW message, and is the internal identifier AMPS uses for that SOW message. - `sowDeleteByData` - Accepts a message, and deletes the record that would be updated by that message. The most efficient way to remove messages from the SOW is to use `sowDeleteByKeys` or `sowDeleteByData`, since those options allow AMPS to exactly target the message or messages to be removed. Many applications use `sowDelete`, since this is the most flexible method for removing items from the SOW when the application does not have information on the exact messages to be removed. Regardless of the command used, AMPS sends an OOF message to all subscribers who have received updates for the messages removed, as described in the previous section. The simple form of the `sowDelete` command returns a `MessageStream` that receives the response. The response is an acknowledgment message that contains information on the delete command. For example, the following snippet simply prints informational text with the number of messages deleted: ```cpp showLineNumbers for (auto msg : client.sowDelete("sow_topic", "/id in (42, 64, 37)")) { std::cout << "Got a " << msg.getCommand() << " message containing " << msg.getAckType() << ": deleted " << msg.getMatches() << " entries." << std::endl; } ``` You can also use `client.execute` to send a SOW delete command. As with the other SOW methods, the client provides an asynchronous versions of the SOW delete commands that require a message handler to be invoked: ```cpp showLineNumbers void HandleSOWDelete(const Message& message) { std::cout << "Got a " << msg.getCommand() << " message containing " << msg.getAckType() << ": deleted " << msg.getMatches() << " entries." << std::endl; } .... client.execute_async(Command("sow_delete") .setTopic("sow_topic") .setFilter("/id in (42, 64, 37)"), bind(HandleSOWDelete, placeholders::_1)); ``` Acknowledging messages from a queue uses a form of the `sow_delete` command that is only supported for queues. Acknowledgment is discussed in the [Using Queues](queues) chapter in this guide. --- # Manual Acknowledgement 60East generally recommends that applications use an `ack()` method to acknowledge messages during normal processing. This approach works properly from within a message handler, provides batching support as described elsewhere in this chapter, and is generally both easier to code and more efficient. However, in some situations, you may need to manually acknowledge messages in the queue. This is most common when an application needs to operate on all messages with certain characteristics, rather than acknowledging individual messages. For example, an application that is doing updates to an order may want cancel an order by both publishing a cancellation and immediately expiring all other messages in the queue for that order. With manual acknowledgment, that application can use a filter to remove all previous updates for that order, then publish the cancellation. To manually acknowledge processed messages and remove the messages from the queue, applications use the `sow_delete` command. To remove specific messages from the queue, provide the bookmarks of those messages. To remove messages that match a given filter, provide the filter. Notice that AMPS only supports using a bookmark with `sow_delete` when removing messages from a queue, not when removing records from a SOW. For example, given a `Message` object to acknowledge and a client, the code below acknowledges the message. ```cpp showLineNumbers void acknowledgeSingle(const Client& client, const Message& message) { Command acknowledge("sow_delete"); acknowledge.setTopic(message.getTopic()) .setBookmark(message.getBookmark()); client.executeAsync(acknowledge, MessageHandler()); } ``` In the above listing the program creates a `sow_delete` command, specifies the topic and the bookmark, and then sends the command to the server. While this method works, creating and sending an acknowledgment for each individual message can be inefficient if your application is processing a large volume of messages. Rather than acknowledging each message individually, your application can build a comma-delimited list of bookmarks from the processed messages and acknowledge all of the messages at the same time. In this case, it's important to be sure that the number of messages you wait for is less than the maximum backlog -- the number of messages your client can have unacknowledged at a given time. Notice that both automatic acknowledgment and the helper method on the `Message` object take the maximum backlog into account. When constructing a command to acknowledge queue messages, AMPS allows an application to specify a filter rather than a set of bookmarks. AMPS interprets this as the client requesting acknowledgment of all messages that match the filter. (This may include messages that the client has not received, subject to the `Leasing` model for the queue.) As a more typical example of manual acknowledgment, the code below expires all messages for a given `id` that have a status other than `cancel`. An application might do this to halt processing of an order that it is about to cancel: ```cpp showLineNumbers void removePending(const Client& client, const std::string& orderId) { Command acknowledge("sow_delete"); acknowledge.setTopic(message.getTopic()) .setFilter("/id = '" + orderId +"' and /status != 'cancel'") .setOptions("expire"); client.executeAsync(acknowledge, MessageHandler()); } ``` In the above listing the program specifies a topic and a filter to use to find the messages that should be removed. In this case, the program also provides the `expire` option to indicate that the messages have been removed from the queue rather than successfully processed (of course, whether this is the correct behavior for a canceled order depends on the expected message flow for your application). Notice that, as described in the section on multithreading, this method of acknowledging a message should not be used from a message handler unless the `sow_delete` is sent from a different client than the client that called the message handler. Instead, 60East recommends using the `ack()` function from within a message handler. --- # Understanding Message Objects So far, we have seen that subscribing to a topic involves working with objects of `AMPS::Message` type. A `Message` represents a single message to or from an AMPS server. Messages are received or sent for every client/server operation in AMPS. ## Header Properties There are two parts of each message in AMPS: a set of headers that provide metadata for the message, and the data that the message contains. Every AMPS message has one or more header fields defined. The precise headers present depend on the type and context of the message. There are many possible fields in any given message, but only a few are used for any given message. For each header field, the `Message` class contains a distinct property that allows for retrieval and setting of that field. For example, the `Message.get_command_id()` function corresponds to the `commandId` header field, the `Message.get_batch_size()` function corresponds to the `BatchSize` header field, and so on. For more information on these header fields, consult the [AMPS User Guide](/docs/amps-user-guide) and [AMPS Command Reference](/docs/amps-command-reference). The `Message` object contains several different accessors for header fields. For retrieving the value of fields on a message: * The `getXxx()` functions return a `Field`. The `Field` contains pointers to the underlying buffer in the `Message`. Data is not copied until either `deepCopy()` is called or the `Field` is converted to another format (such as constructing a `std::string` from the `Field`). The value of the `Field` is only valid for the lifetime of the message unless it is copied using `deepCopy()`. * The `getRawXxx()` functions take a pointer and a length. The pointer is assigned to the first byte of the value in the underlying buffer in the `Message`. The length is set to the length of the value. To assign values to a field in a message, the `Message` object provides several different options. The differences between these options are a matter of managing the memory for the value. * The `setXxx()` functions copy the value provided into the specified header. * The `assignXxx()` functions set the value of the specified header, avoiding a copy if possible. When this function is used, the `Message` may refer directly to the data passed in. That data should not change or be deallocated while the `Message` is in use. (Notice, though, that a copy of the message produced using `deepCopy` *will* copy the data and can be used safely even if the original data is changed or deallocated.) 60East does not recommend attempting to parse header fields from the raw data of the message, nor does 60East recommend attempting to manipulate the fields of the message without using the accessor methods. In AMPS, fields sometimes need to be set to a unique identifier value. For example, when creating a new subscription, or sending a manually constructed message, you’ll need to assign a new unique identifier to multiple fields such as `CommandId` and `SubscriptionId`. For this purpose, `Message` provides `newXxx()` methods for each field that generates a new unique identifier and sets the field to that new value. ## getData() Method Access to the data section of a message is provided via the `getData()` method. The `data` contains the unparsed data in the message, returned as a series of bytes (a `string` or `const char *`). Your application code parses and works with the data. The AMPS C++ client contains a collection of helper classes for working with message types that are specific to AMPS (for example, FIX, NVFIX, and AMPS composite message types). For message types that are widely used, such as JSON or XML, you can use whichever library you typically use in your environment. ## Message Field Reference The [AMPS Command Reference](/docs/amps-command-reference) contains a full description of which fields are available and which fields are returned in response to specific commands. --- # Performance Tips and Best Practices This chapter presents tips and techniques for writing high-performance applications with AMPS. This section presents principles and approaches that describe how to use the features of AMPS and the AMPS client libraries to achieve high performance and reliability. Specific techniques (for example, the details on how to write a message handler) are described in other parts of the AMPS documentation and referenced here. Other techniques require information specific to the application (for example, determining the minimum set of information required in a message), and are best done as part of your application design. All of the recommendations in this section are general guidelines. There are few, if any, universal rules for performance: at times, a design decision that is absolutely necessary to meet the requirements for an application might reduce performance somewhat. For example, your application might involve sending large binary data that cannot be incrementally updated. That application will use more bandwidth per message than an application that sends 100-byte messages with fields that can be incrementally updated. However, since the application depends on being able to deliver the binary payloads, this difference in bandwidth consumption is a part of the requirements for the application, not a design decision that can be optimized. ## Measure Performance and Set Goals The most important tools for creating high performance applications that use AMPS are clear goals and accurate measurement. Without accurate measurement, it's impossible to know whether a particular change has improved performance or not. Without clear goals, it's difficult to know whether a given result is sufficient, or whether you need to continue improving performance. 60East recommends that your measurements include baseline metrics for the part of your message processing that does not involve AMPS. As an example, imagine your task is to reduce the amount of time that elapses between when an order is sent and when the processed response is received from 100ms in total to 85ms in total. To achieve this reduction, you might first measure the processing that your application performs on the order. If that processing consumes 65ms, the most effective optimization may be to improve the order processing. On the other hand, if processing an order consumes 15ms, then optimizing message delivery or network utilization may be the most effective way to meet your goals. When measuring performance, simulate your production environment as closely as possible. For example, AMPS is highly parallelized, so sending a pattern of subscriptions and publishes from a single test client that would normally come from 20 clients will produce a very different performance profile. Likewise, AMPS can typically perform at rates that fill the available bandwidth. Performance measured on a 1GbE connection may be very different than performance measured over a 10GbE connection. Consider the characteristics of your data, and the number of messages you expect to store and process. A 1GB data set consisting of 1 million records will perform differently than a 1GB data set consisting of 10 million records, or a 1GB data set consisting of 100 records. When collecting information about performance, 60East recommends enabling persistence for the Statistics Database (`stats.db`), so you can easily collect historical data on both AMPS and the operating system. For example, a dip in performance correlated with high CPU and memory usage at the same time each day may be correlated with other activity on the system (such as cron jobs or close of business processing). In a situation like that, where the performance reduction is based on factors external to the AMPS application, the overall system metrics captured in `stats.db` can help you re-create the external state and understand the state of the system as a whole. AMPS collects the statistics in memory by default, and persisting that data into a database does not typically have a measurable effect on performance itself, but makes measuring and tuning performance much easier. For performance testing, 60East recommends using dedicated hardware for AMPS to eliminate the effects of other processes. If dedicated hardware is not available and other processes are consuming resources, 60East recommends disabling AMPS NUMA tuning to ensure that AMPS threads do not unnecessarily compete with other processes during performance tuning. ## Use HAClient and Heartbeating Where Appropriate Not every application that uses AMPS requires high availability and the ability to automatically fail over if connectivity is lost or an instance of AMPS is offline. For applications that do need automatic reconnection, 60East strongly recommends using the `HAClient` and setting heartbeating for the client to effectively detect disconnection. When using the `HAClient` and heartbeating, there are two important guidelines to follow: - Do not replace the disconnect handler on the `HAClient`. The disconnect handler is responsible for reconnection, resubscription, and so on. If you need to detect disconnection, use a connection state listener. - Set the interval for heartbeating to approximately one-half the time that the application can tolerate interruption in message flow. Notice that it's not possible for the `HAClient` to tell the difference between an interruption in message flow caused by a server going offline and interruptions caused by an increase in latency due to network saturation or so on, so the interval should be somewhat larger than the highest expected latency between AMPS and the application. Last, but not least, if the application uses asynchronous message handling, the interval should also be set to a value larger than the maximum amount of time expected for the message handler to process a single message. ## Simplify Message Format and Contents AMPS supports a wide range of message types, and is capable of filtering and processing large and complex messages. For many applications, the simplicity of being able to use messages that contain the full information is the most important consideration. For other applications, however, achieving the minimum possible latency and the maximum possible network utilization is important enough to warrant choosing a simplified message format. To simplify message contents, carefully consider the information that downstream processors require. If a downstream process will not use information in the message, there is no need to send the information. For example, consider an application that provides orders from a UI. In such an application, the object that represents the order often contains information relevant to the local state of the application that is not relevant to a downstream system. Rather than simply serializing the full object, your application may perform better if you serialize only the fields that a downstream system will take action on. To simplify message format, choose the simplest format that can convey the information that your application needs. The general principle is that the simpler the message format is, the more quickly AMPS and client libraries can parse messages of that type. Likewise, the more complicated the structure of each message is, the more work is required to parse the message. For the highest levels of performance, 60East recommends keeping the message structure simple and preferring message formats such as NVFIX, BFlat, or flattened JSON (structured as key/value pairs) as compared with more complicated formats such as XML or BSON. ## Measure Serialization and Deserialization When creating baseline performance numbers, measure serialization and deserialization performance independent of the AMPS server or client libraries. This can help you to: - Understand the baseline performance of creating and processing message data under ideal conditions (that is, where there is no application processing, networking, routing, etc. involved). - Easily compare the application-side performance of different message formats or different message layouts within a single format. When testing this performance, it is helpful to use data similar to the data that the application will actually process during a business day, at the volumes the application would typically process. This will help you understand the performance of serialization and deserialization for this specific application. For example, a library for working with a given message format might be less efficient when processing messages with a large number of string fields in a deeply-nested structure, but your application might exchange only numeric data in a relatively flat structure. Likewise, the library for a given format could be efficient for processing a small number of fields, but have lower performance for a message type with hundreds of fields. As with all performance testing, the more closely the test environment matches the actual data and volumes of a production environment, the more helpful those measurements will be for understanding system performance. ## Use Content Filtering Where Possible AMPS content filtering helps your application perform better by ensuring that your application only receives the messages that it needs. Wherever possible, we recommend using content filtering to precisely specify which messages your application needs. In particular, if at any point your application is receiving a message, parsing the message, and then determining whether to act on the message or not, 60East recommends using content filters to ensure that your application only receives messages that it needs to act on. ## Use Asynchronous Message Processing The synchronous message processing interface is straightforward, and presents a convenient interface for getting started with AMPS. However, the `MessageStream` used by the synchronous interface makes a full copy of each message and provides it from the background reader thread to the thread that consumes the message. This memory overhead and synchronization between the reader thread and consumer thread happens regardless of whether the application needs all of the header fields in the message or even processes the message. The `MessageStream` also does not take into account the speed at which your program is consuming messages, and will read messages into memory as fast as the network and processor allow. If your application cannot consume messages at wire speed, this can lead to increasing memory consumption as the application falls further behind the `MessageStream`. Most applications see improved performance by using a `MessageHandler`. With this approach, the `MessageHandler` does minimal work. If more extensive processing is needed, the `MessageHandler` dispatches the work to another thread: but it does this only when the work is necessary, and it only saves the part of the message needed to accomplish the work. ## Use Hash Indexes Where Possible for SOW Queries When querying a SOW, hash indexes on SOW topics are supported for exact matching on string data as described in the *AMPS User Guide*. A hash index can perform many times faster than a parallel query. If the query pattern for your application can take advantage of hash indexes, 60East recommends creating those hash indexes on your SOW topics. More recent versions of AMPS can use hash indexes for a wider variety of filters. When planning your queries, review the SOW queries section of the *AMPS User Guide* for the version you are using for guidelines on the optimizations available in that version. ## Use a Failed Write Handler and Exception Listener In many cases, particularly during the early stages of development, performance problems can point to defects in the application. Even after the application is tuned, monitoring for failure is important to keep applications running smoothly. 60East recommends always installing a failed write handler if your application is publishing messages. This will help you to quickly identify cases where AMPS is rejecting publishes due to entitlement failures, message type mismatches, or other similar problems. 60East recommends always installing an exception listener if your application is using asynchronous message processing. This will help you to identify and correct any problems with your message handler. An exception listener should typically log the message received and return. If recovery is needed, the listener should set a flag for another thread to process rather than attempting to recover on the thread that calls the exception listener. ## Reduce Bandwidth Requirements In many applications that use AMPS, network bandwidth is the single most important factor in overall performance. Your application can use bandwidth most efficiently by reducing message size. For example, rather than serializing an entire object, you might serialize only the fields that the remote process needs to act on, as mentioned above. Likewise, rather than sending one message that contains a collected set of information that processors will need to extract, consider sending a message in the units that processors will work with. This can reduce bandwidth to processors substantially. For example, rather than sending a single message with all of the activity for a single customer over a given period of time (such as a trading day), consider breaking out the record into the individual transactions for the customer. ### Tune Batch Size for SOW Queries As described in the section on [SOW Batch Size](/docs/amps-user-guide/sow-queries/batching-query-results), tuning the batch size for SOW queries can improve overall performance by improving network utilization. In addition, because the AMPS header is only parsed once per batch, a larger batch size can dramatically improve processing performance for smaller messages. The AMPS clients default to a batch size of `10`. This provides generally good performance for most transactional messages (such as order records or inventory records). For large messages, particularly messages greater than a megabyte in size, a batch size of `1` may reduce memory pressure in the client and improve performance. With smaller messages (for example, message sizes of a few hundred bytes), 60East recommends measuring performance with larger batch sizes such as `50` or `100`. For large messages, reducing the batch size may improve overall performance by requiring less memory consumption on the AMPS server. ### Conflate Fast-Changing Information If your data source publishes information faster than your clients need to consume it, consider using a conflated topic. For example, in a system that presents a user interface and displays fast-moving data, it is common for the data to change at a rate faster than the user interface can format and render the data. In this case, a conflated topic can both reduce bandwidth and simplify processing in the user interface. ### Minimize Bandwidth for Updates If your application uses a SOW and processes frequent updates, consider using delta publish and delta subscribe to reduce the size of the messages transmitted. These features are designed to minimize bandwidth while still providing full-fidelity data streams. ### Conflate Queue Acknowledgments The AMPS clients include the ability to conflate acknowledgments back to AMPS as queue messages are processed. Using these features, with an appropriate `max_backlog`, can reduce the amount of network traffic required for acknowledgments. ### Use a Transaction Log When Monitoring Publish Failures When a topic is not covered by a transaction log, AMPS returns acknowledgment messages for every publish that requests one. This ensures that each message is acknowledged, even when AMPS has no persistent record of the messages in the topic. However, acknowledging each message requires more network traffic for each publish message. When a topic is covered by a transaction log, AMPS conflates persisted acknowledgments. Conflation is possible in this case because AMPS has a full record of the messages and does not have to store additional state to conflate the acknowledgments. With conflated acknowledgments, AMPS will send a success acknowledgment periodically that covers all messages up to that point. If a message fails, AMPS immediately sends the conflated success acknowledgment for all previous messages and the failure acknowledgment for the failed message. ### Combine Conflation and Deltas In many cases, using an approach that combines delta publishes to a SOW with delta subscriptions to a conflated topic can dramatically reduce bandwidth to the application with no loss of information. ## Limit Unnecessary Copies One of the most effective ways to increase performance is to limit the amount of data copied within your application. For example, if your message handler submits work to a set of processors that only use the `Data` and `Bookmark` from a `Message`, create a data structure that holds only those fields and copy that information into instances of that data structure rather than copying the entire `Message`. While this approach requires a few extra lines of code, the performance benefits can be substantial. When publishing messages to AMPS, avoid unnecessary copies of the data. For example, if you have the data in a byte array, use the `publish` methods that use a byte array rather than converting the data to a string unnecessarily. Likewise, if you have the data in the form of a string, avoid converting it to a byte array where possible. ## Manage Publish Stores When using a publish store, the Client holds messages until they are acknowledged as persisted by AMPS, as determined by the replication configuration for the AMPS instance. In the event that an instance with `sync` replication goes offline, the publish store for the Client will grow, since the messages are not being fully persisted. To avoid this problem, 60East recommends that an instance that uses `sync` replication always configure Actions to automatically downgrade the replication link if the remote instance goes offline for a period of time, and upgrade the link when the remote instance comes back online. Further, 60East recommends that, where possible, a publisher is provisioned with enough storage to hold its complete publish stream for the amount of time that a destination may be offline or unavailable without downgrading from `sync` replication to `async` replication. For example, if the server considers a downstream system to be unreachable if it has not acknowledged a replicated message in 60 seconds, and the server checks this threshold every 10 seconds, then a publisher should plan that, at any time, the publisher may need to retain approximately 70 seconds worth of published messages. This is calculated as the 60 seconds threshold that the server has established for a destination to run behind, plus the 10 second interval at which the server checks whether the destination is within the threshold. Also notice that, with a configuration like this, a downstream replication destination could run as much as 59 seconds behind indefinitely. A publisher should be provisioned to be able to run effectively in a "worst case" (or nearly "worst case") scenario for an extended period of time. See the *High Availability and Replication* chapter in the *AMPS User Guide* for more information on replication, sync and async acknowledgment modes, and the Actions used to manage replication. ## Use the Server Logs to Help Troubleshoot When troubleshooting problems with an application that uses AMPS, the server-side logs often provide the most helpful information. For example, `trace` level logging shows the data that is flowing through AMPS. Log messages at `info` level show events as incoming connections, commands from clients, and so on. When questions arise about how the server and application interact, the server logs often contain the information. 60East recommends that an AMPS instance used for development and testing log at `trace` level, and that a server used for production log at `info` level, with the ability to log at `trace` level when necessary for investigating any problems that may arise. When a command does not have the expected result, or an application reports an error, the fastest way to understand the problem is often to review the `trace` level logging for the instance. See the *AMPS User Guide* for details on configuring logging and common patterns for searching for information in AMPS logs. ## Work with 60East as Necessary 60East offers performance advice adapted for your specific usage through your support agreement. Once you've set your performance goals, worked through the general best practices and applied the practices that make sense for your application, 60East can help with detailed performance tuning, including recommendations that are specific to your use case and performance needs. --- # Performing SOW Queries To begin, we will look at a simple example of issuing a SOW query. ```cpp showLineNumbers for (auto message : ampsClient.sow("orders" ,"/symbol == 'ROL'")) { if (message.getCommand() == "group_begin" ) { std::cout << "Receiving messages from the SOW." << std::endl ; } else if (message.getCommand() == "group_end") { std::cout << "Done receiving messages from SOW." << std::endl; } else { std::cout << "Received message: " << message.getData () << std::endl; } } ``` In the listing above, the program invokes `ampsClient.sow()` to initiate a SOW query on the `orders` topic, for all entries that have a symbol of ’ROL’. The SOW query is requested with a batch size of 100, meaning that AMPS will attempt to send 100 messages at a time as results are returned. As the query executes, each matching entry in the topic at the time of the query is returned. Messages containing the data of matching entries have a `Command` of value `sow`, so as those arrive, we write them to the console. AMPS sends a "group_begin" message before the first SOW result, and a "group_end" message after the last SOW result. When the SOW query is complete, the `MessageStream` completes iteration and the loop completes. There's no need to explicitly break out of the loop. As with subscribe, the sow function also provides an asynchronous version. In this case, you provide a message handler that will be called on a background thread: ```cpp showLineNumbers void HandleSOW(const Message& message) { if (message.getCommand() == "sow") { cout << message.getData() << endl; } } void ExecuteSOWQuery(Client client) { Command command("sow"); command.setTopic("orders") .setFilter("/symbol='ROL'") .setBatchSize(100); client.executeAsync(command, bind(HandleSOW, placeholders::_1)); } ``` In the listing above, the `ExecuteSOWQuery()` function invokes `client.sow()` to initiate a SOW query on the orders topic, for all entries that have a symbol of `ROL`. The SOW query is requested with a batch size of 100, meaning that AMPS will attempt to send 100 messages at a time as results are returned. As the query executes, the `HandleSOW()` method is invoked for each matching entry in the topic. Messages containing the data of matching entries have a `Command` of `sow`, so as those arrive, we write them to the console. --- # Acknowledgment Batching The AMPS C++ client automatically batches acknowledgments when either of the convenience methods is used. Batching acknowledgments reduces the number of round-trips to AMPS, reducing network traffic and improving overall performance. AMPS sends the batch of acknowledgments when the number of acknowledgments exceeds a specified size, or when the amount of time since the last batch was sent exceeds a specified timeout. You can set the number of messages to batch and the maximum amount of time between batches: ```cpp client.setAckBatchSize(10); // Send batch after 10 messages client.setAckTimeout(1000); // ... or 1 second ``` The AMPS C++ client is aware of the subscription backlog for a subscription. When AMPS returns the acknowledgment for a subscription that contains queues, AMPS includes information on the subscription backlog for the subscription. If the batch size is larger than the subscription backlog, the AMPS C++ client adjusts the requested batch size to match the subscription backlog. 60East recommends tuning the batch size to improve application performance. A value of 1/3 of the smallest `max_backlog` value is a good initial starting point for testing. 60East does not recommend setting the batch size larger than 1/2 of the `max_backlog` value without testing the setting to ensure that the application does not run out of messages to process. --- # Samples of Working With a Queue The C++ client includes the following samples that demonstrate how to query a topic in the SOW. |Sample Name |Demonstrates | |-----------------------------------|--------------------------------------| |`amps_publish_queue.cpp`|Publishing messages to a queue topic. | |`amps_consume_queue.cpp`|Consuming messages from a queue topic.| --- # Using Queues AMPS message queues provide a high-performance way of distributing messages across a set of workers. The _AMPS User Guide_ describes AMPS [Queues](/docs/amps-user-guide/queues) in detail, including the features of AMPS referred to in this chapter. This chapter does not describe message queues in detail, but instead explains how to use the AMPS C++ client with message queues. To publish messages to a message queue, publishers simply publish to any topic that is collected by the queue. There is no difference between publishing to a queue and publishing to any other topic, and a publisher does not need to be aware that the topic will be collected into a queue. Subscribers must be aware that they are subscribing to a queue, and acknowledge messages from the queue when the message is processed. --- # Content Filtering --- # Regular Expression Subscriptions Regular Expression (Regex) subscriptions allow a regular expression to be supplied in the place of a topic name. When you supply a regular expression, it is as if a subscription is made to every topic that matches your expression, including topics that do not yet exist at the time of creating the subscription. To use a regular expression, simply supply the regular expression in place of the topic name in the `subscribe` command. For example: ```cpp showLineNumbers std::string subscriptionId = client.executeAsync( Command("subscribe").setTopic("orders.*"), MessageHandler(myHandlerFunction, NULL)); ... /* The myHandlerFunction is a global function that is invoked by the AMPS client * whenever a matching message is received. The first parameter, message, is * a reference to an AMPS Message object that contains the data and headers * of the received message. The second parameter, userData, is set to whatever * value was provided in the MessageHandler constructor -- NULL in this example. */ void myHandlerFunction(const Message& message, void* userData) { std::cout << message.getTopic() << ": " << message.getData() << std::endl; } ``` In this example, messages on topics `orders-north-america`, `orders-europe`, and `new-orders` would match the regular expression. Messages published to any of those topics will be sent to our `message_handler` function. As in the example, you can use the `getTopic()` function to determine the actual topic of the message sent to the function. --- # Returning a Message to the Queue A subscriber can also explicitly release a message back to the queue. AMPS returns the message to the queue, and redelivers the message just as though the lease had expired. To do this, the subscriber sends a `sow_delete` command with the bookmark of the message to release and the `cancel` option. When using automatic acknowledgments and the asynchronous API, AMPS will cancel a message if an exception is thrown from the message handler. To return a message to the queue, you can build a `sow_delete` acknowledgment using the `Command` class, or pass an option to the `ack()` method on the message. |Option |Result | |-----------------------|----------------------------------------------| |`cancel` |Returns the message to the queue. | |`expire` |Immediately expire the message from the queue.| For example, to return a message to a queue, call `ack()` on the message and pass the `cancel` option. ```cpp message.ack("cancel"); ``` --- # Samples of Querying a Topic in the SOW The C++ client includes the following samples that demonstrate how to query a topic in the SOW. |Sample Name |Demonstrates | |---------------------------------|--------------------------------------------------| |`amps_publish_sow.cpp`|Publishing messages to a SOW topic. | |`amps_query_sow.cpp` |Querying messages from a SOW topic. | --- # Samples of SOW and Subscribe The C++ client includes the following samples that demonstrate how to query a topic in the SOW and enter a subscription to receive updates to the topic. |Sample Name |Demonstrates | |---------------------------------------|-------------------------------------------------------------------------------------------------| |`amps_publish_sow.cpp` |Publishing messages to a SOW topic.| |`amps_sow_and_subscribe.cpp`|Querying messages from a SOW topic and entering a subscription to receive updates.| --- # SOW and Subscribe Imagine an application that displays real-time information about the position and status of a fleet of delivery vans. When the application starts, it should display the current location of each of the vans along with their current status. As vans move around the city and post other status updates, the application should keep its display up to date. Vans upload information to the system by posting message to a van `location` topic, configured with a key of `van_id` on the AMPS server. In this application, it is important to not only stay up-to-date on the latest information about each van, but to ensure all of the active vans are displayed as soon as the application starts. Combining a SOW with a subscription to the topic is exactly what is needed, and that is accomplished by the `Client.sowAndSubscribe()` method. Now we will look at an example: ```cpp showLineNumbers /* processSOWMessage * * Processes a message during SOW query. Returns * true if the SOW query is complete (group_end command), * false otherwise. */ bool processSOWMessage(const AMPS::Message& message) { if (message.getCommand() == "group_begin") { std::cout << "Receiving messages from the SOW." << std::endl; } else if (message.getCommand() == "group_end") { std::cout << "Done receiving messages from SOW." << std::endl; return true; } else { std::cout << "SOW message: " << message.getData() << std::endl; addVan(message); } return false; } /* processSubscriptionMessage * * Process messages received on a subscription, after the SOW * query is complete. */ void processSubscribeMessage(const AMPS::Message& message) { if (message.getCommand() == "oof") { std::cout << "OOF : " << message.getReason() << " message to remove : " << message.getData() << std::endl; removeVan(message); } else { std::cout << "New or updated message: " << message.getData() << std::endl; addOrUpdateVan(message); } } ... void doSowAndSubscribe(AMPS::Client& ampsClient) { bool sowDone = false; std::cerr << "about to subscribe..." << std::endl; /* We issue a sowAndSubscribe() to begin receiving information about all of the * open orders in the system for the symbol ROL. These orders are now are returned * as Messages whose Command returns SOW. * * Notice here that we specified true for the oofEnabled parameter. * Setting this parameter to true causes us to receive Out-of-Focus("OOF") * messages for the topic. OOF messages are sent when an entry that was sent * to us in the past no longer matches our query. This happens when an entry * is removed from the SOW cache via a sowDelete() operation, * when the entry expires (as specified by the expiration time on the message * or by the configuration of that topic on the AMPS server), * or when the entry no longer matches the content filter * specified. In our case, when an order is processed or canceled * (or if the symbol changes), a Message is sent with Command set to OOF. * The content of that message is the message sent previously. * We use OOF messages to remove orders from our display as they are * completed or canceled. */ for (auto message : ampsClient.execute(Command("sow_and_subscribe") .setTopic("van_location") .setFilter("/status = 'ACTIVE'") .setBatchSize(100) .setOptions("oof"))) { if (sowDone == false) { sowDone = processSOWMessage(message); } else { processSubscribeMessage(message); } } } ``` Now we will look at an example that uses the asynchronous form of `sowAndSubscribe`: ```cpp showLineNumbers // handleMessage // // Handles messages for both SOW query and subscription. void processSOWMessage(const AMPS::Message& message) { if (message.getCommand() == "group_begin") { std::cout << "Receiving messages from the SOW." << std::endl; return; } else if (message.getCommand() == "group_end") { std::cout << "Done receiving messages from SOW." << std::endl; return true; } else if (message.getCommand() == "oof") { std::cout << "OOF : " << message.getReason() << " message to remove : " << message.getData() << std::endl; removeVan(message); } else { std::cout << "New or updated message: " << message.getData() << std::endl; addOrUpdateVan(message); } } ... std::string trackVanPositions(AMPS::Client& ampsClient) { std::cerr << "about to subscribe..." << std::endl; return ampsClient.executeAsync( Command("sow_and_subscribe") .setTopic("van_location") .setFilter("/status = 'ACTIVE'") .setBatchSize(100) .setOptions("oof"), bind(processSOWMessage(placeholders::_1)); } ``` In the listing above, the `trackVanPositions` function invokes `sowAndSubscribe` to begin tracking vans, and returns the subscription ID. The application can later use this to unsubscribe. The two forms have the same result. However, one form performs processing on a background thread, and blocks the client from receiving messages while that processing happens, while the other form processes messages on the calling thread and allows the background thread to continue to receive messages while processing occurs. In both cases, the application receives and processes the same messages. --- # State of the World The AMPS State of the World (SOW) allows you to automatically keep and query the latest information about a topic on the AMPS server, without building a separate database. Using SOW lets you build impressively high-performance applications that provide rich experiences to users. The AMPS C++ client lets you query SOW topics and subscribe to changes with ease. AMPS SOW topics can be used as a current value cache to provide the most recently published value for each record, as a key/value object store, as the source for an aggregate or conflated topic, or all of the above uses. For more information on State of the World topics, see the [AMPS User Guide](/docs/amps-user-guide) --- # Subscriptions Messages published to a topic on an AMPS server are available to other clients via a subscription. Before messages can be received, a client must subscribe to one or more topics on the AMPS server so that the server will begin sending messages to the client. The server will continue sending messages to the client until the client unsubscribes, or the client disconnects. With content filtering, the AMPS server will limit the messages sent only to those messages that match a client-supplied filter. In this chapter, you will learn how to subscribe, unsubscribe, and supply filters for messages using the AMPS C/C++ client. ## Subscribing Subscribing to an AMPS topic takes place by calling `Client.subscribe()`. Here is a short example showing the simplest way to subscribe to a topic (error handling and connection details are omitted for brevity): ```cpp showLineNumbers Client client(...); client.connect(...); /* Here we have created or received a Client that is properly connected to an AMPS server. */ client.logon(); /* Here we subscribe to the topic messages. We do not provide a filter, so AMPS * does not content-filter the subscription. Although we don't use the object * explicitly here, the subscribe function returns a MessageStream object that we * iterate over. If, at any time, we no longer need to subscribe, we can break out * of the loop. When we break out of the loop, the MessageStream goes out of scope, * the MessageStream destructor runs, and the AMPS client sends an unsubscribe * command to AMPS. */ for (auto message : client.subscribe("messages")) { /* Within the body of the loop, we can process the message as we need to. In this * case, we simply print the contents of the message. */ std :: cout << "Received message: " << message.getData () << std :: endl ; } ``` AMPS creates a background thread that receives messages and copies them into a `MessageStream` that you iterate over. This means that the client application as a whole can continue to receive messages while you are doing processing work. The simple method described above is provided for convenience. The AMPS C++ client provides convenience methods for the most common form of the AMPS commands. The client also provides an interface that allows you to have precise control over the command. Using that interface, the example above becomes: ```cpp showLineNumbers Client client(...); client.connect(...); /* Here we have created or received a Client that is properly connected to an AMPS server. */ client.logon(); /* Here we subscribe to the topic messages. We do not provide a filter, so AMPS * does not content-filter the subscription. Although we don't use the object * explicitly here, the execute function returns a MessageStream object that we * iterate over. If, at any time, we no longer need to subscribe, we can break out * of the loop. When we break out of the loop, the MessageStream goes out of scope, * the MessageStream destructor runs, and the AMPS client sends an unsubscribe * command to AMPS. * * Here we create a command object for the subscribe command, specifying the topic * messages. */ for (auto message : ampsClient.execute(Command("subscribe").setTopic("messages"))) { std :: cout << "Received message: "<< message.getData () << std :: endl; } ``` The `Command` interface allows you to precisely customize the commands you send to AMPS. For flexibility and ease of maintenance, 60East recommends using the `Command` interface (rather than a named method) for any command that will receive messages from AMPS. For publishing messages, there can be a slight performance advantage to using the named commands where possible. --- # Synchronous Message Processing As mentioned [earlier](subscriptions.md), one way for an application to receive messages is to have the AMPS C++ client return a `MessageStream` object that can be used to iterate over the results of the command. The `MessageStream` object makes copies of the incoming messages. When there is no message available, the `MessageStream` will block. A `MessageStream` will only remain active while the client that produced it is connected. If the client disconnects, the `MessageStream` will continue to provide any messages that have not yet been consumed, then throw an exception. The advantages of using a `MessageStream` that it provides a simple processing model, that receiving messages from a `MessageStream` does not block the client receive thread (see [Understanding Threading](understanding-threading-section.md) ) and that a copy of the message is automatically made for the application. In return for these advantages, a `MessageStream` has higher overhead than [Asynchronous Message Processing](async.md), it will not be resumed if the client disconnects, and, by default, it will use as much memory as necessary to hold messages coming from the AMPS server. --- # Understanding Threading The first time a command causes an instance of the `Client` or `HAClient` to connect to AMPS (typically, the `logon()` command), the client creates a thread that runs in the background. This background thread is responsible for processing incoming messages from AMPS, which includes both messages that contain data and acknowledgments from the server. When you call a command on the AMPS client, the command typically waits for an acknowledgment from the server and then returns. (The exception to this is `publish`. For performance, the `publish` command does not wait for an acknowledgment from the server before returning.) In the simple case, using synchronous message processing, the client provides an internal handler function that populates the `MessageStream`. The client receive thread calls the internal handler function, which makes a deep copy of the incoming message and adds it to the `MessageStream`. The `MessageStream` is used on the calling thread, so operations on the `MessageStream` do not block the client receive thread. When using asynchronous message processing, AMPS calls the handler function from the client receive thread. Message handlers provided for *asynchronous* message processing must be aware of the following considerations: - The client creates one client receive thread at a time, and the lifetime of the thread lasts for the lifetime of the connection to the AMPS server. A message handler that is only provided to a single client will only be called from a single thread at a time. If your message handler will be used by multiple clients, then multiple threads will call your message handler. In this case, you should take care to protect any state that will be shared between threads. Notice that if the client connection fails (or is closed), and the client reconnects, the client will create a different thread for the new connection. - For maximum performance, do as little work in the message handler as possible. For example, if you use the contents of the message to update an external database, a message handler that adds the relevant data to an update queue, that is processed by a different thread, will typically perform better than a message handler that does this update during the message handling. - While your message handler is running, the thread that calls your message handler is no longer receiving messages. This makes it easier to write a message handler because you know that no other messages are arriving from the same subscription. However, this also means that you cannot use the same client that called the message handler to send commands to AMPS. Acknowledgments from AMPS cannot be processed and your application will deadlock waiting for the acknowledgment. Instead, enqueue the command in a work queue to be processed by a separate thread or use a different client object to submit the commands. - The AMPS client resets and reuses the `Message` provided to this function between calls. This improves performance in the client, but means that if your handler function needs to preserve information contained within the message, you must copy the information (either by making a copy of the entire message or copying the required fields) rather than just saving the message object. Otherwise, the AMPS client cannot guarantee the state of the object or the contents of the object when your program goes to use it. Likewise, a message handler should not modify the `Message` -- this will result in modifying the message provided to other handlers (including handlers internal to the AMPS client). --- # Unexpected Messages The AMPS C++ client handles most incoming messages and takes appropriate action. Some messages are unexpected or occur only in very rare circumstances. The AMPS C++ client provides a way for clients to process these messages. Rather than providing handlers for all of these unusual events, AMPS provides a single handler function for messages that can't be handled during normal processing. Your application registers this handler by setting the `UnhandledMessageHandler` for the client. This handler is called when the client receives a message that can't be processed by any other handler. This is a rare event, and typically indicates an unexpected condition. For example, if a client publishes a message that AMPS cannot parse, AMPS returns a failure acknowledgment. This is an unexpected event, so AMPS does not include an explicit handler for this event, and failure acknowledgments are received in the method registered as the `UnhandledMessageHandler`. Your application is responsible for taking any corrective action needed. For example, if a message publication fails, your application can decide to republish the message, publish a compensating message, log the error, stop publication altogether, or any other action that is appropriate. --- # Unhandled Exceptions In the AMPS C++ client, exceptions can occur that are not thrown to the main thread of the application. For example, when an exception is thrown from a message handler running on a background thread, AMPS does not automatically propagate that exception to the main thread. Instead, AMPS provides the exception to an unhandled exception handler if one is specified on the client. The unhandled exception handler receives a reference to the exception object, and takes whatever action is necessary. Typically, this involves logging the exception or setting an error flag that the main thread can act on. Notice that AMPS C++ client only catches exceptions that derive from `std::exception`. If your message handler contains code that can throw exceptions that do not derive from `std::exception`, 60East recommends catching these exceptions and throwing an equivalent exception that derives from `std::exception`. If your application will attempt to recover from an exception thrown on the background processing thread, your application should set a flag and attempt recovery on a *different* thread than the thread that called the exception listener. :::tip At the point that the AMPS client calls the exception listener, it has handled the exception. Your exception listener must not rethrow the exception (or wrap the exception and throw a different exception type). ::: For example, the unhandled exception handler below takes a `std::ostream`, and logs information from each exception to that `std::ostream`. ```cpp showLineNumbers class ExceptionLogger : public AMPS::ExceptionListener { private: std::ostream& os_; public: ExceptionLogger() : os_(std::cout) {} ExceptionLogger(std::ostream& os) : os_(os) {} virtual void exceptionThrown(const std::exception& e) const { os_ << e.what() << std::endl; } } ``` --- # Ending Subscriptions The AMPS server continues a subscription until the client explicitly ends the subscription (that is, *unsubscribes*) or until the connection to the client is closed. With the synchronous interface, AMPS automatically unsubscribes to the topic when the destructor for the `MessageStream` runs. You can also explicitly call the `close()` method on the `MessageStream` object to remove the subscription. In the asynchronous interface, when a subscription is successfully made, messages will begin flowing to the message handler, and the `subscribe()` or `executeAsync()` call will return a string for the subscription id that serves as the identifier for this subscription. A `Client` can have any number of active subscriptions, and this subscription id is how AMPS designates messages intended for this particular subscription. To unsubscribe, we simply call `unsubscribe` with the subscription identifier: ```cpp showLineNumbers Client client = ...; // Register asynchronous subscription std::string subId = client.executeAsync( Command("subscribe").setTopic("messages"), MessageHandler(myHandlerFunction, NULL)); ... other work here ... client.unsubscribe(subId); ``` In this example, as in the previous section, we use the `client.executeAsync()` method to create a subscription to the `messages` topic. When our application is done listening to this topic, it unsubscribes by passing in the `subId` returned by `subscribe()`. After the subscription is removed, no more messages will flow into our `myHandlerFunction()`. When an application calls `unsubscribe()`, the client sends an explicit `unsubscribe` command to AMPS. The AMPS server removes that subscription from the set of subscriptions for the client, and stops sending messages for that subscription. On the client side, the client unregisters the subscription so that the `MessageStream` or `MessageHandler` for that subscription will no longer receive messages for that subscription. Notice that calling `unsubscribe` does not destroy messages that the server has already sent to the client. If there are messages on the way to the client for this subscription, the AMPS client must consume those messages. If a `LastChanceMessageHandler` is registered, the handler will receive the messages. Otherwise, they will be discarded since no message handler matches the subscription ID on the message. --- # Utility Classes The AMPS C++ client includes a set of utilities and helper classes to make working with AMPS easier. ## Composite Message Types The client provides a pair of classes for creating and parsing composite message types: - `CompositeMessageBuilder` allows you to assemble the parts of a composite message and then serialize them in a format suitable for AMPS. - `CompositeMessageParser` extracts the individual parts of a composite message type. For more information regarding composite message types, refer to the [Message Types]/docs/amps-user-guide/message-types) chapter in the [AMPS User Guide](/docs/amps-user-guide). ### Building Composite Messages To build a composite message, create an instance of `CompositeMessageBuilder`, and populate the parts. The `CompositeMessageBuilder` copies the parts provided, in order, to the underlying message. The builder simply writes to an internal buffer with the appropriate formatting, and does not allow you to update or change the individual parts of a message once they've been added to the builder. The snippet below shows how to build a composite message that includes a JSON part, constructed as a string, and a binary part consisting of the bytes from a standard `vector`. ```cpp showLineNumbers std::string json_part("{\"data\":\"sample\"}"); std::vector data; /* Populate data */ ... /* Create the payload for the composite message. */ AMPS::CompositeMessageBuilder builder; builder.append(json_part.str()); builder.append(reinterpret_cast(data.data()), data.size() * sizeof(double)); /* Send the message */ std::string topic("messages"); ampsClient.publish(topic.c_str(), topic.length(), builder.data(), builder.length()); ``` ### Parsing Composite Messages To parse a composite message, create an instance of `CompositeMessageParser`, then use the `parse()` method to parse the message provided by the AMPS client. The `CompositeMessageParser` gives you access to each part of the message as a sequence of bytes. For example, the following snippet parses and prints messages that contain a JSON part and a binary part that contains an array of doubles. ```cpp showLineNumbers for (auto message : ampsClient.subscribe("messages")) { parser.parse(message); /* First part is JSON */ std::string json_part = std::string(parser.getPart(0)); /* Second part is the raw bytes for a vector */ AMPS::Field binary = parser.getPart(1); std::vector vec; double *array_start = (double*)binary.data(); double *array_end = array_start + (binary.len() / sizeof(double)); vec.insert(vec.end(), array_start, array_end); /* Print the contents of the message */ std::cout << "Received message with " << parser.size() << " parts" << std::endl << "\t" << json_part << std::endl; for (auto d : vec) std::cout << d << " "; std::cout << std::endl; } ``` Notice that the receiving application is written with explicit knowledge of the structure and content of the composite message type. ### Composite Message Builder and Composite Message Parser Samples The C++ client distribution contains the following samples to demonstrate `CompositeMessageBuilder` and `CompositeMessageParser`. |Sample Name |Demonstrates | |-----------------------------------------|--------------------------------------------------------------------------------------| |`amps_publish_composite.cpp` |Creating and publishing a composite message using `CompositeMessageBuilder`| |`amps_subscribe_composite.cpp`|Receiving and parsing a composite message using `CompositeMessageParser`| ## NVFIX Messages The client provides a pair of classes for creating and parsing NVFIX message types: - `NVFIXBuilder` allows you to assemble an NVFIX message and then serialize it in a format suitable for AMPS. - `NVFIXShredder` extracts the individual fields of an NVFIX message type. ### Building NVFIX Messages To build an NVFIX message, create an instance of `NVFIXBuilder`, then add the fields of the message using `append()`. `NVFIXBuilder` copies the fields provided, in order, to the underlying message. The builder simply writes to an internal buffer with the appropriate formatting, and does not allow you to update or change the individual fields of a message once they've been added to the builder. The snippet below shows how to build an NVFIX message and publish it to the AMPS client. ```cpp showLineNumbers /* Construct a client with the name "NVFIXPublisher". */ AMPS::Client ampsClient("NVFIXPublisher"); /* Construct a simple NVFIX message. */ AMPS::NVFIXBuilder builder; /* Add data to the builder */ builder.append("Test", "data"); builder.append("More", "stuff"); /* Display the data */ std::cout << builder.getString() << std::endl; try { /* Connect to the server and log on */ ampsClient.connect(uri); ampsClient.logon(); /* Publish message to the topic messages */ std::string topic("messages"); ampsClient.publish(topic, builder.getString()); } catch (const AMPS::AMPSException& e) { std::cerr << e.what() << std::endl; exit(1); } ``` ### Parsing NVFIX Messages To parse an NVFIX message, create an instance of `NVFIXShredder`, then use the `toMap()` method to parse the message provided by the AMPS client. The `NVFIXShredder` gives you access to each field of the message in a map. The snippet below shows how to parse and print an NVFIX message. ```cpp showLineNumbers /* Create a client with the name "NVFIXSubscriber" */ AMPS::Client ampsClient("NVFIXSubscriber"); try { /* Connect to the server and log on */ ampsClient.connect(uri); ampsClient.logon(); /* Subscribe to the messages topic. * * This overload of the subscribe method returns a MessageStream * that can be iterated over. When the MessageStream destructor * runs, the destructor unsubscribes. */ /* Set up the shredder */ AMPS::NVFIXShredder shredder; for (auto message : ampsClient.subscribe("messages")) { /* Shred the data to a map */ auto subscription = shredder.toMap(message.getData()); /* Display the data */ for (auto iterator = subscription.begin(); iterator != subscription.end(); ++iterator) { std::cout << iterator->first << " " << iterator->second << std::endl; } } } catch (const AMPS::AMPSException& e) { std::cerr << e.what() << std::endl; exit(1); } ``` ### NVFIX Builder and Shredder Samples The C++ client distribution contains the following samples to demonstrate `NVFIXBuilder` and `NVFIXShredder`. |Sample Name |Demonstrates | |----------------------------------------------|-------------------------------------------------------------------| |`amps_nvfix_builder_publisher.cpp` |Creating and publishing a message using `NVFIXBuilder`| |`amps_nvfix_builder_subscriber.cpp`|Receiving a message and parsing it using `NVFIXShredder`| ## FIX Messages The client provides a pair of classes for creating and parsing FIX messages: - `FIXBuilder` allows you to assemble a FIX message and then serialize them in a format suitable for AMPS. - `FIXShredder` extracts the individual fields of a FIX message. ### Building FIX Messages To build a FIX message, create an instance of `FIXBuilder`, then add the fields of the message using `append()`. `FIXBuilder` copies the fields provided, in order, to the underlying message. The builder simply writes to an internal buffer with the appropriate formatting, and does not allow you to update or change the individual fields of a message once they've been added to the builder. The snippet below shows how to build a FIX message and publish it to the AMPS client. ```cpp showLineNumbers /* Construct a client with the name "FIXPublisher". */ AMPS::Client ampsClient("FIXPublisher"); /* Construct a simple FIX message. */ AMPS::FIXBuilder builder; /* Add data to the builder */ builder.append(0, "123"); /* Display the data */ std::cout << builder.getString() << std::endl; try { /* Connect to the server and log on */ ampsClient.connect(uri); ampsClient.logon(); /* Publish message to the messages topic */ std::string topic("messages"); ampsClient.publish(topic, builder.getString()); } catch (const AMPS::AMPSException& e) { std::cerr << e.what() << std::endl; exit(1); } ``` ### Parsing FIX Messages To parse a FIX message, create an instance of `FIXShredder`, then use the `toMap()` method to parse the message provided by the AMPS client. The `FIXShredder` gives you access to each field of the message in a map. The snippet below shows how to parse and print a FIX message. ```cpp showLineNumbers /* Create a client with the name "FIXSubscriber" */ AMPS::Client ampsClient("FIXSubscriber"); try { /* Connect to the server and log on */ ampsClient.connect(uri); ampsClient.logon(); /* Subscribe to the messages topic * * This overload of the subscribe method returns a MessageStream * that can be iterated over. When the MessageStream destructor * runs, the destructor unsubscribes. */ /* Set up the shredder */ AMPS::FIXShredder shredder; for (auto message : ampsClient.subscribe("messages")) { // Shred the data to a map auto subscription = shredder.toMap(message.getData()); // Display the data for (auto iterator = subscription.begin(); iterator != subscription.end(); ++iterator) { std::cout << iterator->first << " " << iterator->second << std::endl; } } } catch (const AMPS::AMPSException& e) { std::cerr << e.what() << std::endl; exit(1); } ``` ### FIX Builder and Shredder Samples The C++ client distribution contains the following samples to demonstrate `FIXBuilder` and `FIXShredder`. |Sample Name |Demonstrates | |--------------------------------------------|-----------------------------------------------------------------| |`amps_fix_builder_publisher.cpp` |Creating and publishing a message using `FIXBuilder`| |`amps_fix_builder_subscriber.cpp`|Receiving a message and parsing it using `FIXShredder`| --- # Welcome to the AMPS C#/.NET Client This guide provides information you need to get started with the AMPS C#/.NET client. It focuses specifically on the client and does not cover AMPS itself in detail. For an overview of AMPS and instructions on setting up your development environment, see the [Introduction to AMPS](/docs/intro-guide/intro) guide. :::tip This guide assumes that you have a development environment for C# and access to an AMPS server using the configuration provided with the C# samples (in the full source distribution of the client). ::: --- # Acknowledgment Batching The AMPS C# client automatically batches acknowledgments when either of the convenience methods is used. Batching acknowledgments reduces the number of round-trips to AMPS, reducing network traffic and improving overall performance. AMPS sends the batch of acknowledgments when the number of acknowledgments exceeds a specified size, or when the amount of time since the last batch was sent exceeds a specified timeout. You can set the number of messages to batch and the maximum amount of time between batches, as shown below: ``` client.setAckBatchSize(10); // Send batch after 10 messages client.setAckTimeout(1000); // ... or 1 second ``` The AMPS C# client is aware of the subscription backlog for a subscription. When AMPS returns the acknowledgment for a subscription that contains queues, AMPS includes information on the subscription backlog for the subscription. If the batch size is larger than the subscription backlog, the AMPS C# client adjusts the requested batch size to match the subscription backlog. 60East recommends tuning the batch size to improve application performance. A value of 1/3 of the smallest `max_backlog` value is a good initial starting point for testing. 60East does not recommend setting the batch size larger than 1/2 of the `max_backlog` value without testing the setting to ensure that the application does not run out of messages to process while the acknowledgment is being sent to AMPS. --- # Acknowledging Messages For each message delivered on a subscription, AMPS counts the message against the subscription backlog until the message is explicitly acknowledged. In addition, when a queue specifies `at-least-once` delivery, AMPS retains the message in the queue until the message expires or until the message has been explicitly acknowledged and removed from the queue. From the point of view of the AMPS server, acknowledgment is implemented as a `sow_delete` from the queue with the bookmarks of the messages to remove. The AMPS C# client provides several ways to make it easier for applications to create and send the appropriate `sow_delete`. ## Automatic Acknowledgment The AMPS client allows you to specify that messages should be automatically acknowledged. When this mode is on, AMPS acknowledges the message automatically in the following cases: - *Asynchronous Message Processing Interface* - The message handler returns without throwing an exception. - *Synchronous Message Processing Interface* - The application requests the next message from the `MessageStream`. AMPS batches acknowledgments created with this method, as described in the following section. To enable automatic acknowledgment, use the `setAutoAck()` method. ```csharp client.setAutoAck(true); // enable AutoAck ``` ## Message Convenience Method The AMPS C# client provides a convenience method, `ack()`, on delivered messages. When the application is finished with the message, the application simply calls `ack()` on the message. (This, in turn, provides the topic and bookmark to the `ack()` method of the client that received the message.) For messages that originated from a queue with `at-least-once` semantics, this adds the bookmark from the message to the batch of messages to acknowledge. For other messages, this method has no effect. ``` message.ack(); // Add this message to the next // acknowledgment batch. ``` --- # Advanced Topics ## C# Client Compatibility AMPS clients are available for many languages. Many AMPS customers write clients using a variety of languages, often both Java and C#. While Java and C# are fundamentally different languages, they share enough syntax that it can be straightforward to port code between the two, and especially from Java to C#. To aid in conversion from Java to C# (and from C# to Java), the C# client has a number of features that make it a little easier to bring code from Java to C#, as described below: - Java-style getters and setters - `getXXX()` / `setXXX()` - are provided corresponding to properties on the `Message` class. For example, given a variable message of type `Message`, the code: `string userName = message.UserName` and `string userName = message.getUserName()` are equivalent. - C# parameters that take lambda functions also take an interface type. The AMPS Java client defines interfaces such as `ClientMessageHandler` that your application implements, with a single `invoke()` method that is called when an event occurs. In C#, the AMPS client uses lambda functions and delegates to provide equivalent functionality. However, the same `*Handler` interfaces exist in C#, and instead of passing a lambda function, you may also implement these interfaces and pass in derived classes. While doing so would be inconvenient in C#, providing this symmetry allows your Java and C# to be ported interchangeably. - Java-style method name conventions are used throughout AMPS. In .NET, method names often begin with a capitalized first letter (e.g. `Connect()` instead of `connect()`). However, the C# AMPS client retains the capitalization style of the Java client where possible, making porting straightforward. ## Strong Naming Starting with the 5.0 release of the C# client, the included Release build of the C# client is strong-named. The build files included with the client do not produce a strong-named assembly. To prevent misidentification of assemblies, 60East does not ship the key used to strong-name the assembly, and has removed references to the key from the build files included with the client. What this means is that, if you build your own version of the assembly, you must provide your own strong-name key and update the build process to reference that key. For more information on strong naming assemblies, visit the link [https://msdn.microsoft.com/en-us/library/wd40t7ad(v=vs.110).aspx](https://msdn.microsoft.com/en-us/library/wd40t7ad(v=vs.110).aspx) to see the MSDN article. ## SSL Certificates and the C# Client The AMPS C# client uses the standard .NET mechanisms for creating an SSL connection. This means that you manage certificates stores and trust chains for the AMPS C# client as you would for any other .NET application. For information on creating SSL certificates for testing, visit the link [https://msdn.microsoft.com/en-us/library/ms733813(v=vs.110).aspx](https://msdn.microsoft.com/en-us/library/ms733813(v=vs.110).aspx) to see the MSDN article. ## Transport Filtering The AMPS C# client offers the ability to filter incoming and outgoing messages in the format they are sent and received on the network. This allows you to inspect or modify outgoing messages before they are sent to the network, and incoming messages as they arrive from the network. To create a transport filter, you implement the interface `TransportFilter`, construct an instance of the filter class, and install the filter with the `setTransportFilter` method on the transport. The AMPS C# client does not validate any changes made by the transport filter. This interface is most useful for application debugging or transport development. The client includes a sample filter, `TransportTraceFilter`, that simply writes incoming and outgoing buffers to a `TextWriter`. Notice that the `TransportFilter` function is called with the verbatim contents of data received from AMPS. This means that, for incoming data, the function may not be called precisely on message boundaries, and that the binary length encoding used by the client and server will be presented to the transport filter. ## Working with Messages and Byte Buffers The AMPS C# client allows you to publish messages that contain data from byte buffers. When working with byte buffers in AMPS, it's best to follow the simple conventions outlined below. AMPS provides overloaded `publish()` methods that allow you to publish messages from various formats. For example, to publish a message with data from a byte buffer, you must first provide the data as a `byte[]`, the position of the data, and the length of the data. Also, the message topic, to which the message will be published, must also be provided as a `byte[]` along with its position and length. The example below shows how to serialize an object into a byte buffer, then publish the message to AMPS using the `publish()` method. ```csharp showLineNumbers ... // create the topic string, and decode it to a byte[] var topic = "messages"; byte[] topicBytes = System.Text.Encoding.UTF8.GetBytes(topic.ToCharArray()); // create the payload and construct the composite CompositeMessageBuilder builder = new CompositeMessageBuilder(); builder.append(data.getBytes(), 0, data.getBytes().Length); // set the topic to messages, and create a field for the builder // to extract the buffer string topic = "messages"; // construct the payload object Employee emp; // create the BinaryFormatter used to serialize the object BinaryFormatter bf = new BinaryFormatter(); using (var memStream = new MemoryStream()) { // serialize the object to a MemoryStream bf.Serialize(memStream, emp); // set the byte of the message to a field field.set(ms.GetBuffer(), 0, ms.GetBuffer().Length); } // publish to the "messages" topic using byte buffers client.publish(topicBytes, 0, topicBytes.Length, myField.buffer, myField.position, myField.length); ... ``` AMPS also provides a way to access the raw bytes of a message when subscribing to those messages. The method `getDataRaw()` returns a `Field` that is composed of a byte buffer, position of the data in the buffer, and length. This data could then be deserialized and converted to an object for further use. The example below shows how to access the raw bytes of a message, and then shows how to deserialize the bytes of that message to a arbitrary C# object. ```csharp showLineNumbers ... // store the message raw data to a variable var rawBytes = message.getDataRaw(); // create the container object for the data Employee emp; using (var memStream = new MemoryStream()) { // construct the BinaryFormatter that will parse // the MemoryStream data to an object var formatter = new BinaryFormatter(); // write, to the MemoryStream, the message raw data memStream.Write(rawBytes.buffer, rawBytes.position, rawBytes.length); memStream.Seek(0, SeekOrigin.Begin); // deserialize the MemoryStream back to the original object emp = bf.Deserialize(memStream); } ... ``` --- # Assembly Deployment Once your application is built, you will need to think about how to deploy it to additional computers. With your application’s dependency on `AMPS.Client.dll`, you need to include `AMPS.Client.dll` along with your application. The most straightforward way to accomplish this is to install `AMPS.Client.dll` into the same folder as your `.exe` file. For example, if you distribute your executable in a zip file that users are expected to unpack, simply include `AMPS.Client.dll` assemblies into that zip file. When your executable runs, Windows will attempt to load `AMPS.Client.dll` from the same directory as your executable, and if it is not found, your executable will fail to run. If your organization develops and deploys many AMPS applications and would like more centralized control over the maintenance of these AMPS client deployments, consider installing `AMPS.Client.dll` into the *Global Assembly Cache* ("GAC"). The GAC allows you to share one copy of an assembly — like the AMPS client — across many applications on a computer. This technique requires that the assembly have a strong name, and that you use an installer that places `AMPS.Client.dll` into the GAC. Installing the assembly in the GAC is not recommended unless many applications will share an AMPS client. For more information on the GAC, follow the link [http://msdn.microsoft.com/en-us/library/yf1d93sz.aspx](http://msdn.microsoft.com/en-us/library/yf1d93sz.aspx) to the Microsoft Developer Network documentation on the GAC. You are now able to develop and deploy an application in C# that publishes messages to AMPS. In the following chapters, you will learn how to subscribe to messages, use content filters, work with SOW caches and fine-tune messages that you send. --- # Asynchronous Message Processing The AMPS C# client also supports an asynchronous interface. In this case, you add a message handler to the call to the subscribe. The client returns the command ID of the command submitted to AMPS, and returns once the server has acknowledged that the command has been processed. As messages arrive, AMPS calls your message handler directly on the background thread. This can be an advantage for some applications. For example, if your application is highly multithreaded and copies message data to a work queue processed by multiple threads, there may be a performance benefit to enqueuing work directly from the background thread. See the [Understanding Threading](understanding-threading-section.md) section for a discussion of threading considerations, including considerations for message handlers. As with the simple interface, the AMPS client provides both convenience interfaces and interfaces that use a `Command` object. The following example shows how to use the asynchronous interface. ```csharp showLineNumbers class MyApp { public static void Main() { // Here, we create a Client. We protect the Client in a // using block so that the connection and subscriptions // are properly cleaned up when dispose() is called. using(Client client = new Client("subscribe")) { client.connect("tcp://127.0.0.1/9007/amps"); client.logon(); Command command = new Command("subscribe").setTopic("messages"); // Here, we call the executeAsync() method, specifying the command and // the message handler to invoke with messages received in response to // the command. CommandId subscriptionId = c.executeAsync(command, (message) => Console.WriteLine(message.Data)); // (message) => Console.WriteLine(message.Data) is a lambda function // that acts as our message handler. When a message is received, this // lambda function is invoked, and in this case, the Data property from // message is printed to the screen. Message is of type AMPS.Client.Message. } } } ``` :::info In the asynchronous interface, the AMPS client resets and reuses the message object provided to this lambda function between calls. This improves performance in the client, but means that if your handler function needs to preserve information contained within the message, you must copy the information rather than just saving the message object. Otherwise, the AMPS client cannot guarantee the state of the object or the contents of the object when your program goes to use it. ::: --- # Backlog and Smart Pipelining AMPS queues are designed for high-volume applications that need minimal latency and overhead. One of the features that helps performance is the *subscription backlog* feature, which allows applications to receive multiple messages at a time. The subscription backlog sets the maximum number of unacknowledged messages that AMPS will provide to the subscription. When the subscription backlog is larger than `1`, AMPS delivers additional messages to a subscriber before the subscriber has acknowledged the first message received. This technique allows subscribers to process messages as fast as possible, without ever having to wait for messages to be delivered. The technique of providing a consistent flow of messages to the application is called *smart pipelining*. ### Subscription Backlog The AMPS server determines the backlog for each subscription. An application can set the maximum backlog that it is willing to accept with the `max_backlog` option. Depending on the configuration of the queue (or queues) specified in the subscription, AMPS may assign a smaller backlog to the subscription. If no `max_backlog` option is specified, AMPS uses a `max_backlog` of `1` for that subscription. In general, applications that have a constant flow of messages perform better with a `max_backlog` setting higher than `1`. The reason for this is that, with a backlog greater than `1`, the application can always have a message waiting when the previous message is processed. Setting the optimum `max_backlog` is a matter of understanding the messaging pattern of your application and how quickly your application can process messages. To request a `max_backlog` for a subscription, you explicitly set the option on the subscribe command, as shown below: ```csharp showLineNumbers Command cmd = new Command("subscribe"); cmd.setTopic("my_queue").setOptions("max_backlog=10"); ``` --- # Before You Start Welcome to developing applications with AMPS, the Advanced Message Processing System from 60East Technologies! These guides will help you learn how to develop applications using AMPS. Before getting started with this guide, it is important to have a good understanding of the following topics: * *Developing Applications in C# (or your .NET language of choice)* To be successful using this guide, and developing applications with AMPS, you will need to have a working knowledge of the language you are developing in. * *AMPS Concepts* This guide focuses on using the AMPS client libraries and how those libraries work with the AMPS server. Before working through this guide, we recommend reading the [Introduction to AMPS](/docs/intro-guide/intro) guide. Detailed explanations of the AMPS server behavior are in the [AMPS Server Documentation](/). ## Version Compatibility The AMPS C# client is compatible with .NET Framework 4.6.2 and later versions, as .NET Framework 4.6.2 is the earliest 4.x version still supported by Microsoft. The client can also be used with .NET Standard 2.0 and later, following Microsoft's schedule for supported versions. ## Setting Up a Development Instance You will need an installed and running AMPS server to use the product as well. You can read the sample programs without a running server, but you will get the most out of this guide by running the programs against a working server. Instructions for starting an instance of AMPS are available in the [Introduction to AMPS](/docs/intro-guide/intro) guide. :::tip The AMPS server runs on x64 Linux. The [Introduction to AMPS](/docs/intro-guide/intro) and [AMPS FAQ](/faq) contain information on how to run an AMPS server on a development system that does not run Linux. ::: --- # Client Identification AMPS uses the name of the client as a session identifier and as part of the identifier for messages originating from that client. For this reason, when a transaction log is enabled in the AMPS instance (that is, when the instance is recording a sequence of publishes and attempting to eliminate duplicate publishes), an AMPS instance will only allow one application with a given client name to connect to the instance. When a transaction log is present, AMPS **requires** the client name for a publisher to be: - Unique within a set of replicated AMPS instances - Consistent from invocation to invocation *if* the publisher will be publishing the same *logical* stream of messages If publishers do not meet this contract (for example, if the publisher changes its name and publishes the same messages, or if a different publisher uses the same session name), message loss or duplication can happen. 60East recommends always using consistent, unique client names. For example, the client name could be formed by combining the application name, an identifier for the host system, and the ID of the user running the application. A strategy like this provides a name that will be different for different users or on different systems, but consistent for instances of the application that should be treated as equivalent to the AMPS system. Likewise, if a publisher is sending a completely independent stream of messages (for example, a microservice that sends a different, unrelated sequence of messages each time it connects to AMPS), there is no need for a publisher to retain the same name each time it starts. However, if a publisher is resuming a stream of messages (as in the case when using a file-backed publish store), that publisher **must** use the same client name, since the publisher is resuming the session. --- # Client-Side Conflation In many cases, applications that use SOW topics only need the current value of a message at the time the message is processed, rather than processing each change that led to the current value. On the server side, AMPS provides *conflated topics* to meet this need. Conflated topics are described in more detail in the *AMPS User Guide*, and require no special handling on the client side. In some cases, though, it's important to conflate messages on the client side. This can be particularly useful for applications that do expensive processing on each message, applications that are more efficient when processing batches of messages, or for situations where you cannot provide an appropriate conflation interval for the server to use. A `MessageStream` has the ability to conflate messages received for a subscription to a SOW topic, view, or conflated topic. When conflation is enabled, for each message received, the client checks to see whether it has already received an unprocessed message with the same `SowKey`. If so, the client replaces the unprocessed message with the new message. The application never receives the message that has been replaced. To enable client-side conflation, you call `conflate()` on the `MessageStream`, and then use the `MessageStream` as usual: ```csharp showLineNumbers // SOW query and subscribe MessageStream results = ampsClient.sowAndSubscribe("orders", "/symbol == 'ROL'"); // Turn on conflation results.conflate(); // Process the results foreach (Message m in results) { // Process message here } ``` Notice that if the `MessageStream` is used for a subscription that does not include `SowKeys` (such as a subscription to a topic that does not have a SOW), no conflation will occur. When using client-side conflation with delta subscriptions, bear in mind that client-side conflation replaces the whole message, and does not attempt to merge deltas. This means that updates can be lost when messages are replaced. For some applications (for example, a ticker application that simply sends delta updates that replace the current price), this causes no problems. For other applications (for example, when several processors may be updating different fields of a message simultaneously), using conflation with deltas could result in lost data, and server-side conflation is a safer alternative. --- # Connection Parameters for AMPS When specifying a URI for connection to an AMPS server, you may specify a number of transport-specific options in the parameters section of the URI connection parameters. Here is an example: ```bash tcp://localhost:9007/amps/json?tcp_nodelay=true&tcp_sndbuf=100000 ``` In this example, we have specified the AMPS instance on `localhost`, port `9007`, connecting to a transport that uses the `amps` protocol and sending JSON messages. We have also set two parameters: `tcp_nodelay`, a Boolean (true/false) parameter, and `tcp_sndbuf`, an integer parameter. Multiple parameters may be combined to finely tune settings available on the transport. Normally, you'll want to stick with the defaults on your platform, but there may be some cases where experimentation and fine-tuning will yield higher or more efficient performance. The AMPS client supports the value of `tcp` in the *scheme* component connection string for TCP/IP connections, and the value of `tcps` as the scheme for SSL encrypted connections. ## IPv6 Connections Starting with version 5.3.3.0, the AMPS client supports creating connections over both IPv4 and IPv6 protocols if supported by the underlying Operating System. By default, the AMPS client will prefer to resolve host names to IPv4 addresses, but this behavior can be adjusted by supplying the `ip_protocol_prefer` transport option, described in the table below. ## TCP and SSL Transport Options The following transport options are available for TCP connections: |Option |Description | |-------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`bind` |

    (IP address) Sets the interface to bind the outgoing socket to.

    Starting with version 5.3.3.0, both IPv4 and IPv6 addresses are fully supported for use with this parameter.

    | |`tcp_connecttimeout` |(integer) Sets the connect timeout in milliseconds. This helps enable failover in cases where an attempt to connect to a server is unresponsive without returning a failure. | |`tcp_rcvbuf` |(integer) Sets the socket receive buffer size. This defaults to the system default size. (On Linux, you can find the system default size in `/proc/sys/net/core/rmem_default`.)| |`tcp_sndbuf` |(integer) Sets the socket send buffer size. This defaults to the system default size. (On Linux, you can find the system default size in `/proc/sys/net/core/wmem_default`.)| |`tcp_nodelay` |(boolean) Enables or disables the `TCP_NODELAY` setting on the socket. By default `TCP_NODELAY` is disabled.| |`tcp_linger` |(integer) Enables and sets the `SO_LINGER` value for the socket By default, `SO_LINGER` is enabled with a value of `10`, which specifies that the socket will linger for 10 seconds.| |`tcp_keepalive` |(boolean) Enables or disables the `SO_KEEPALIVE` value for the socket. The default value for this option is true.| |`ip_protocol_prefer` |

    (string) Influence the IP protocol to prefer during DNS resolution of the host. If a DNS entry of the preferred protocol can not be found, the other non-preferred protocol will then be tried.

    If this parameter is not set, the default will be to prefer IPv4.

    If an explicit IPv4 address or IPv6 IP address is provided as the host, the format of the IP address is used to determine the IP protocol used and this setting has no effect.

    Supported Values:

    `ipv4`: Prefer an IPv4 address when resolving the host

    `ipv6`: Prefer an IPv6 address when resolving the host

    This parameter is available starting with version 5.3.3.0.

    | ## Using HTTP Preflight for Connection Upgrades Some users need to minimize the number of externally accessible ports while still allowing multiple AMPS transports to be used in environments with strict firewall policies for security reasons. To address this, AMPS supports an HTTP Preflight mechanism that enables TCP clients to share the same external port used for WebSockets. Instead of requiring a dedicated external TCP port, clients can establish a connection over an existing HTTP endpoint. This reduces the number of open network ports while maintaining full TCP/TCPS functionality. This feature works by leveraging HTTP Upgrade requests, similar to WebSockets, allowing clients to connect via an initial HTTP request before transitioning to a full TCP/TCPS session. Additionally, the HTTP preflight mechanism enables custom HTTP headers to be included in the initial handshake, making it easier to integrate with reverse proxies. To request HTTP preflight, add the `http_preflight=true` option to the connection string. For example: ``` tcp://proxy-to-amps.example.com:8020/client/amps/json?http_preflight=true ``` To learn more about HTTP Preflight, including how to enable it and configure an NGINX proxy, refer to the [HTTP Preflight](/docs/amps-user-guide/transports/http-preflight) section in the _AMPS User Guide_ and the [HTTP Preflight- Proxy Play: AMPS Unlocked](../../blog/http-preflight) blog. ## Compress Network Traffic By default, network traffic between the application and the AMPS server is not compressed. The AMPS C#/.NET client optionally supports compressing the network connection to and from AMPS. To enable this, add the `compression` option to the connection string. In this release, the AMPS C#/.NET client supports `zlib` compression to the server. Notice that compressing and decompressing data performs additional work on both the client and server side to compress and decompress the data. The goal is reducing the bandwidth required for the traffic. If bandwidth is at a premium and the data compresses well, this can produce gains in overall performance. In other situations, the benefit can be negligible or create extra latency. 60East recommends testing with your message data and volumes to determine the effect of compression for your usage. For example: ``` tcp://amps.example.com:9007/client/amps/json?compression=zlib ``` ## AMPS Additional Logon Options The connection string can also be used to pass logon parameters to AMPS. AMPS supports the following additional logon option: |Option |Description | |-----------------------|-----------------------------------------------------------------------------------------------| |`pretty` |Provide formatted representations of binary messages rather than the original message contents.| --- # Connection Strings for AMPS The AMPS clients use connection strings to determine the server, port, transport, and protocol to use to connect to AMPS. When the connection point in AMPS accepts multiple message types, the connection string also specifies the precise message type to use for this connection. Connection strings have a number of elements: ![Diagram showing the different elements of a connection string](./assets/client-connection-string-light.svg#gh-light-mode-only) ![Diagram showing the different elements of a connection string](./assets/client-connection-string-dark.svg#gh-dark-mode-only) As shown in the figure above, connection strings have the following elements: * _Transport_ - Defines the network used to send and receive messages from AMPS. In this case, the transport is `tcp`. For connections to transports that use the Secure Sockets Layer (SSL), use `tcps`. For connections to AMPS over a Unix domain socket, use `unix`. * _Host Address_ - Defines the destination on the network where the AMPS instance receives messages. The format of the address is dependent on the transport. For `tcp` and `tcps`, the address consists of a host name and port number. In this case, the host address is `localhost:9007`. For `unix` domain sockets, a value for hostname and port must be provided to form a valid URI, but the content of the hostname and port are ignored, and the file name provided in the **path** parameter is used instead (by convention, many connection strings use `localhost:0` to indicate that this is a local connection that does not use TCP/IP). * _Protocol_ - Sets the format in which AMPS receives commands from the client. Most code uses the default `amps` protocol, which sends header information in JSON format. AMPS supports the ability to develop custom protocols as extension modules, and AMPS also supports legacy protocols for backward compatibility. * _MessageType_ - Specifies the message type that this connection uses. This component of the connection string is required if the protocol accepts multiple message types and the transport is configured to accept multiple message types. If the protocol does not accept multiple message types, this component of the connection string is optional, and defaults to the message type specified in the transport. Legacy protocols such as `fix`, `nvfix` and `xml` only accept a single message type, and therefore do not require or accept a message type in the connection string. As an example, a connection string such as: ```bash tcp://localhost:9007/amps/json ``` would work for programs connecting from the local host to a `Transport` configured as follows: ```xml showLineNumbers ... any-tcp tcp 9007 amps ... ``` See the [Configuring Transports](/docs/amps-user-guide/transports/configuring-transports) section in the _AMPS User Guide_ for more information on configuring transports. ## Using zlib Compression The AMPS C# Client supports enabling zlib compression by adding the `compression=zlib` URI parameter to the connection string. For example: ``` tcp://localhost:9007/amps/json?compression=zlib ``` No server-side configuration changes are required. The client enables zlib compression for the connection based on the URI parameter. If the connection string already contains other URI parameters, add `compression=zlib` using `&`: ``` tcp://localhost:9007/amps/json?=&compression=zlib ``` --- # Content Filtering One of the most powerful features of AMPS is content filtering. With content filtering, filters based on message content are applied at the server so that your application and the network are not utilized by messages that are not relevant to your application. For example, if your application is only displaying messages from a particular user, you can send a content filter to the server so that only messages from that particular user are sent to the client. To apply a content filter to a subscription, simply pass it into the `Client.subscribe()` call: ```csharp showLineNumbers Client c = ...; CommandId subscriptionId = c.subscribe((m) => Console.WriteLine(m), "messages", "/message/sender = 'mom' ", 5000); // Timeout ``` In this example, we have passed in a content filter `/sender = 'mom'`. This will result in the server only sending us messages, from the `messages` topic, that have the sender field equal to `mom` in the message. For example, the AMPS server will send the following message, where `/sender` is `mom`: ```javascript showLineNumbers { "sender" : "mom", "text" : "Happy Birthday!", "reminder" : "Call me Thursday!" } ``` The AMPS server will not send a message with a different `/sender` value: ```javascript showLineNumbers { "sender" : "henry dave", "text" : "Things do not change; we change." } ``` --- # Controlling Blocking with Command Timeout The named convenience methods and the `Command` class provide a `timeout` setting that specifies how long the command should wait to receive a `processed` acknowledgment from AMPS. This can be helpful in cases where it is important for the caller to limit the amount of time to block waiting for AMPS to acknowledge the command. If the AMPS client does not receive the processed acknowledgment within the specified time, the client sends an `unsubscribe` command to the server to cancel the command and throws an exception. Acknowledgments from AMPS are processed by the client receive thread on the same socket as data from AMPS. This means that any other data previously returned (such as the results of a large query) must be consumed before the acknowledgment can be processed. An application that submits a set of SOW queries in rapid succession should set a timeout that takes into account the amount of time required to process the results of the previous query. --- # AMPS Programming: Working With Commands The AMPS clients provide named methods for core AMPS functionality. These named methods work by creating messages and sending those messages to AMPS. All communication with AMPS occurs through messages. You can use the `Command` object to customize the messages that the AMPS client sends. This can be useful for more advanced scenarios, where you need precise control over AMPS, in cases where you need to use an earlier version of the client to communicate with a more recent version of AMPS, or in cases where a named method is not available. ## Understanding AMPS Messages AMPS messages are represented in the client as `AMPS.Message` objects. The `Message` object is generic, and can represent any type of AMPS message, including both outgoing and incoming messages. This section includes a brief overview of elements common to AMPS command messages. Full details of commands to AMPS are provided in the *AMPS Command Reference* (linked at the bottom of this page). All AMPS command messages contain the following elements: - **Command** - The *command* tells AMPS how to interpret the message. Without a command, AMPS will reject the message. Examples of commands include `publish`, `subscribe`, and `sow`. - **CommandId** - The *command ID*, together with the name of the client, uniquely identifies a command to AMPS. The command ID can be used later on to refer to the command or the results of the command. For example, the command ID for a `subscribe` message becomes the identifier for the subscription. The AMPS client provides a command ID when the command requires one and no command ID is set. Most AMPS messages contain the following fields: - **Topic** - The *topic* that the command applies to, or a regular expression that identifies a set of topics that the command applies to. For most commands, the topic is required. Commands such as `logon`, `start_timer`, and `stop_timer` do not apply to a specific topic, and do not need this field. - **Ack Type** - The *ack type* tells AMPS how to acknowledge the message to the client. Each command has a default acknowledgment type that AMPS uses if no other type is provided. - **Options** - The `options` are a comma-separated list of options that affect how AMPS processes and responds to the message. Beyond these fields, different commands include fields that are relevant to that particular command. For example, SOW queries, subscriptions, and some forms of SOW deletes accept the **Filter** field, which specifies the filter to apply to the subscription or query. As another example, publish commands accept the **Expiration** field, which sets the SOW expiration for the message. For full details on the options available for each command and the acknowledgment messages returned by AMPS, see the *AMPS Command Reference*. ## Creating and Populating the Command To create a command, you simply allocate a message object of the appropriate type: ```csharp Command command = new Command("sow"); ``` Once created, you set the appropriate fields on the command. For example, the following code creates a SOW query, setting the command, topic and filter for the query: ```csharp Command command = new Command("sow") .setTopic("messages-sow") .setFilter("/id > 20"); ``` When sent to AMPS using the `execute()` method, AMPS performs a SOW query from the topic `messages-sow` using a filter of `/id > 20`. The results of sending this message to AMPS are no different than using the form of the `sow` method that sets these fields. ## Using Execute Once you've created a command, use the `execute` method to send the command to AMPS. The `execute` method returns a `MessageStream` that provides response messages. The `executeAsync` method sends the command to AMPS, waits for a `processed` acknowledgment, then returns. Messages are processed on the client background thread. For example, the following snippet sends the command created above: ```csharp client.execute(command); ``` This returns a `MessageStream` identical to the `MessageStream` returned by the equivalent `client.sow()` method. You can also provide a message handler to receive acknowledgments, statistics, or the results of subscriptions and SOW queries. The AMPS client maintains a background thread that receives and processes incoming messages. The call to `executeAsync` returns on the main thread as soon as AMPS acknowledges the command as having been processed, and messages are received and processed on the background thread. To send a message and use an asynchronous message handler, pass the handler and the message to `executeAsync()`. For example, the following snippet uses a lambda expression to create a simple message handler, passing that message handler and the message to `executeAsync()`. ```csharp client.executeAsync(command, (m) => Console.WriteLine(m.getAckType() + " : " + m.getReason)); ``` While this message handler simply prints the ack type and reason for sample purposes, message handlers in production applications are typically designed with a specific purpose. For example, your message handler may fill a work queue, or check for success and throw an exception if the command failed. ### Using Execute to Publish Notice that the `publish` command does not typically provide return results other than acknowledgment messages. To send a `publish` command, use the `executeAsync()` method with a `null` message handler: ```csharp client.executeAsync(publishCmd, null); ``` Since the code provides a `null` message handler, this code does not receive acknowledgments. To detect publish failures, set the `FailedWriteHandler` for the client. The `publish` methods of the AMPS C# client use the same internal implementation as `executeAsync` with a `null` message handler. ## AMPS Command Cookbook The [AMPS Command Reference](/docs/amps-command-reference) includes information on which fields and options to set on commands to get a specific result. The reference includes both reference information and a [Command Cookbook](/docs/amps-command-reference/cookbook) that provides a concise guide for commonly-used commands. --- # Providing Credentials to AMPS When a client logs on to AMPS, the client sends AMPS a username and password. The username is derived from the URI, using the standard syntax for providing a username in a URI. For example, `tcp://JohnDoe:@server:port/amps/messagetype` to include the username `JohnDoe` in the request. For a given username, the password is provided by an `Authenticator`. The AMPS client distribution includes a `DefaultAuthenticator` that simply returns the password, if any, provided in the URI. A `logon()` command that does not specify an `Authenticator` will use an instance of `DefaultAuthenticator`. If your authentication system requires a different authentication token, you can implement an `Authenticator` that provides the appropriate token. --- # Delta Publish To delta publish, you use the `delta_publish` command as follows: ```csharp showLineNumbers // assumes that client is connected and logged on String msg = ... ; // obtain changed fields here client.deltaPublish("myTopic", msg); ``` The message that you provide to AMPS must include the fields that the topic uses to generate the SOW key. Otherwise, AMPS will not be able to identify the message to update. For SOW topics that use a User-Generated SOW Key, use the `Command` form of `delta_publish` to set the `SowKey`, as shown below: ```csharp showLineNumbers // assumes that client is connected and logged on msg = ... ; // obtain changed fields here key = ... ; // obtain user-generated SOW key Command cmd("delta_publish"); cmd.setTopic("delta_topic"); cmd.setSowKey(key); cmd.setData(msg); // Execute the delta publish. Use null for // the message handler since any failure acks will // be routed to the FailedWriteHandler. client.executeAsync(cmd,null); ``` The [AMPS User Guide](/docs/amps-user-guide) section on making [Incremental Message Updates](/docs/amps-user-guide/delta-publish) describes how the AMPS server processes the `delta_publish` command. --- # Delta Subscribe To delta subscribe, you use the `delta_subscribe` command as follows: ```csharp showLineNumbers // assumes that client is connected and logged on Command cmd("delta_subscribe"); cmd.setTopic("delta_topic"); cmd.setFilter("/thingIWant = 'true'"); for (Message m in client.execute(cmd)) { // work with message here } ``` As described in the [AMPS User Guide](/docs/amps-user-guide) section on [Receiving Only Updated Fields](/docs/amps-user-guide/delta-subscribe), messages provided to a delta subscription will contain the fields used to generate the SOW key and any changed fields in the message. Your application is responsible for choosing how to handle the changed fields. --- # Delta Publish and Subscribe Delta messaging in AMPS has two independent aspects: - **Delta Subscribe** - Allows subscribers to receive just the fields that are updated within a message. - **Delta Publish** - Allows publishers to update and add fields within a message by publishing only the updates into the SOW. This chapter describes how to create delta publish and delta subscribe commands using the AMPS C# client. For a discussion of this capability, how it works, and how message types support this capability see the [AMPS User Guide](/docs/amps-user-guide). --- # Detecting Write Failures The `publish` methods in the C# client deliver the message to be published to AMPS and then return immediately, without waiting for AMPS to return an acknowledgment. Likewise, the `sowDelete` methods request deletion of SOW messages, and return before AMPS processes the message and performs the deletion. This approach provides high performance for operations that are unlikely to fail in production. However, this means that the methods return before AMPS has processed the command, without the ability to return an error in the event that the command fails. The AMPS C# client provides a `FailedWriteHandler` that is called when the client receives an acknowledgment that indicates a failure to persist data within AMPS. To use this functionality, you implement the `FailedWriteHandler` interface, construct an instance of your new class, and register that instance with the `setFailedWriteHandler()` function on the client. When an acknowledgment returns that indicates a failed write, AMPS calls the registered handler method with information from the acknowledgment message, supplemented with information from the client publish store if one is available. Your client can log this information, present an error to the user, or take whatever action is appropriate for the failure. If your application needs to know whether publishes succeeded and are durably persisted, the following approach is recommended: - Set a `PublishStore` on the client. This will ensure that messages are retransmitted if the client becomes disconnected before the message is acknowledged *and* request `persisted` acknowledgments for messages. - Install a `FailedWriteHandler`. In the event that AMPS reports an error for a given message, that event will be reported to the `FailedWriteHandler`. - Call `publishFlush()` and verify that all messages are persisted before the application exits. When no `FailedWriteHandler` is registered, acknowledgments that indicate errors in persisting data are treated as unexpected messages and routed to the `LastChanceMessageHandler`. In this case, AMPS provides only the acknowledgment message and does not provide the additional information from the client publish store. --- # Disconnect Handling Every distributed system will experience occasional disconnections between one or more nodes. The reliability of the overall system depends on an application's ability to efficiently detect and recover from these disconnections. Using the AMPS C# client's disconnect handling, you can build powerful applications that are resilient in the face of connection failures and spurious disconnects. --- # Error Handling In every distributed system, the robustness of your application depends on its ability to recover gracefully from unexpected events. The AMPS client provides the building blocks necessary to ensure your application can recover from the kinds of errors and special events that may occur when using AMPS. --- # Exception Handling and Asynchronous Message Processing When using asynchronous message processing, exceptions thrown from the message handler are silently absorbed by the AMPS C# client by default. The AMPS C# client allows you to register an exception listener to detect and respond to these exceptions. When an exception listener is registered, AMPS will call the exception listener with the exception. See the section on [Unhandled Exceptions](unhandled-exceptions.md) for details. --- # Exception Types The following table details each of the exception types thrown by AMPS. | Exception | When | Notes | | ------------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AlreadyConnectedException` | Connecting | Thrown when `connect()` is called on a Client that is already connected. | | `AMPSException` | Anytime | Base class for all AMPS exceptions. | | `AuthenticationException` | Anytime | Indicates an authentication failure occurred on the server. | | `BadFilterException` | Subscribing | This typically indicates a syntax error in a filter expression. | | `BadRegexTopicException` | Subscribing | Indicates a malformed regular expression was found in the topic name. | | `CommandException` | Anytime | Base class for all exceptions relating to commands sent to AMPS. | | `ConnectionException` | Anytime | Base class for all exceptions relating to the state of the AMPS connection. | | `ConnectionRefusedException` | Connecting | The connection was actively refused by the server. Validate that the server is running, that network connectivity is available, and the settings on the client match those on the server. | | `DisconnectedException` | Anytime | No connection is available when AMPS needed to send data to the server *or* the user's disconnect handler threw an exception. | | `InvalidTopicException` | SOW query | The topic is not configured for the requested operation. For example, a `sow` command was issued for a topic that is not in the SOW or a bookmark subscribe was issued for a topic that is not recorded in the transaction log. | | `InvalidTransportOptionsException` | Connecting | An invalid option or option value was specified in the URI. | | `InvalidURIException` | Connecting | The URI string provided to `connect()` was formatted improperly. | | `MessageTypeException` | Connecting | The class for a given transport's message type was not found in AMPS. | | `MessageTypeNotFoundException` | Connecting | The message type specified in the URI was not found in AMPS. | | `NameInUseException` | Connecting | The client name (specified when instantiating `Client`) is already in use on the server. | | `RetryOperationException` | Anytime | An error occurred that caused processing of the last command to be aborted. Try issuing the command again. | | `StreamException` | Anytime | Indicates that data corruption has occurred on the connection between the client and server. This usually indicates an internal error inside of AMPS -- contact AMPS support. | | `SubscriptionAlreadyExistsException` | Subscribing | A subscription has been requested using the same `CommandId` as another subscription. Create a unique `CommandId` for every subscription. | | `TimedOutException` | Anytime | A timeout occurred waiting for a response to a command. | | `TransportTypeException` | Connecting | Thrown when a transport type is selected in the URI that is unknown to AMPS. | | `UnknownException` | Anytime | Thrown when an internal error occurs. Contact AMPS support immediately. | --- # Exceptions Generally speaking, when an error occurs that prohibits an operation from succeeding, AMPS will throw an exception. AMPS exceptions universally derive from `AMPS.Client.Exceptions.AMPSException`, so by catching `AMPSException`, you will be sure to catch anything AMPS throws, for example: ```csharp showLineNumbers using AMPS.Client; using AMPS.Client.Exceptions; ... public static void ReadAndEvaluate(Client client) { // read a new payload from the user string payload = Console.ReadLine(); // write a new message to AMPS if (!string.IsNullOrEmpty(payload)) { try { client.publish("UserMessage", @"{ ""data"" : """ + payload + @""" }"); } catch (AMPSException exception) { Console.Error.WriteLine("An AMPS exception " + "occurred: {0}", exception); } } } ``` In this example, if an error occurs, the program writes the error to `Console.Error` and the `publish()` command fails. However, `client` is still usable for continued publishing and subscribing. When the error occurs, the exception is written to the console, which implicitly calls the exception's `ToString()` method. As with most .NET exceptions, `ToString()` will convert the exception into a string that includes a message, stacktrace and information on any "inner" exceptions (exception from outside of AMPS that caused AMPS to throw an exception). AMPS exception types vary, based on the nature of the error that occurs. In your program, if you would like to handle certain kinds of errors differently than others, then you can `catch` the appropriate subclass of `AMPSException` to detect those specific errors and do something different. ```csharp showLineNumbers public CommandId CreateNewSubscription(Client client) { CommandId id = null; // Attempt to retrieve a topic name (or regular // expression) from the user. string topicName; while (id == null) { topicName = AskUserForTopicName(); try { Command command = new Command("subscribe").setTopic(topicName); // If an error occurs when setting up the subscription, whether // or not to try again depends on the subclass of AMPSException // that is thrown. If a BadRegexTopicException is thrown, it means // that a bad regular expression was supplied during subscription. // In this case, we would like to give the user a chance to correct // the issue. id = client.executeAsync(command, (x)=>HandleMessage(x)); } catch(BadRegexTopicException ex) { DisplayError(string.Format("Error: bad topic name or regular " + "expression '{0}'. The error was: {1}", topicName, ex.Message)); // We'll ask the user for another topic. } // If an AMPS exception of a type other than BadRegexTopicException is // thrown by AMPS, it is caught here. In that case, the program emits // a different error message to the user. catch(AMPSException ex) { DisplayError(string.Format("Error: error setting up subscription " + "to topic '{0}'. The error was: {1}", topicName, ex.Message)); // At this point the code stops attempting to subscribe to the client // by the return null statement. return null; // give up } } return id; } ``` --- # Changing the Filter on a Subscription AMPS allows you to update parameters, such as the content filter, on a subscription. When you replace a filter on the subscription, AMPS immediately begins sending only messages that match the updated filter. Notice that if the subscription was entered with a command that includes a SOW query, using the `replace` option can re-issue the SOW query (as described in the *AMPS User Guide*). To update the filter on a subscription, you create a `subscribe` command. You set the `SubscriptionId` provided on the `Command` to the identifier of the existing subscription and include the `replace` option on the `Command`. When you send the `Command`, AMPS atomically replaces the filter and sends messages that match the updated filter from that point forward. ```csharp showLineNumbers // Assumes client is connected and logged on to AMPS // Enter subscription Command subscribe_cmd = new Command("sow_and_subscribe") .setTopic("orders-sow") .setSubId("A42") // Used later for replace .setFilter("/details/items/description LIKE 'puppy'"); MyHandler mh = new MyHandler(); client.executeAsync(subscribe_cmd, mh); // ... // Replace filter elsewhere in the program Command replace_cmd = new Command("sow_and_subscribe") .setTopic("orders-sow") .setSubId("A42") // A42 is the ID of the subscription to replace .setFilter("/details/items/description LIKE 'kitten'") .setOptions("replace"); client.executeAsync(replace_cmd, mh); ``` --- # Your First AMPS Program In this chapter, we will learn more about the structure and features of the AMPS C# library, and build our first C# program using AMPS. ## About the Client Library The AMPS client is packaged as a single managed assembly, `AMPS.Client.dll` You can find `AMPS.Client.dll` in the `AMPS/bin` directory of your AMPS C# client. Every .NET application you build will need to reference this assembly file, and the assembly must be deployed along with your application in order for your application to function properly. ## Connecting to AMPS Let's begin by writing a simple program that connects to an AMPS server and publishes a single message to a topic: ```csharp showLineNumbers using System; using AMPS.Client; using AMPS.Client.Exceptions; namespace AMPSBookExamples { class ConnectToAMPS { static void Main(string[] args) { using(Client client = new Client("exampleClient")) { try { client.connect("tcp://192.168.1.3:9007/amps"); client.logon(); client.publish("messages", @"{ ""message"" : ""Hello, World!"" }"); } catch (AMPSException e) { Console.Error.WriteLine(e); } } } } } ``` In the example above, we show the entire program; but future examples will isolate one or more specific portions of the code. The next section describes how to build and run the application and explains the code in further detail. ### Build and Run To build this program, create a new C# command-line project in Visual Studio and add a reference to `AMPS.Client.dll` using the "Add Reference..." option in Visual Studio. Replace the code in `Program.cs` with the code in the example above, then modify the `client.connect()` on line 14 with the address and port of your AMPS server. Now you should be able to compile and execute the code, and if the AMPS server is running, the message `Hello world` is published to the messages topic. If an error occurs, an exception will be written to the console. If the message is published successfully, there is no output to the console. We will demonstrate how to create a subscriber to receive messages in the next chapter. ### Examining the Code Let us now revisit the code we listed earlier. ```csharp showLineNumbers using System; using AMPS.Client; using AMPS.Client.Exceptions; namespace AMPSBookExamples { class ConnectToAMPS { static void Main(string[] args) { // This line creates a new Client object. Client encapsulates a // single connection to an AMPS server. Methods on Client allow // for connecting, disconnecting, publishing, and subscribing to // an AMPS server. The argument to the Client constructor, // "exampleClient", is a name chosen by the client to identify // itself to the server. Errors relating to this connection will // be logged with reference to this name, and AMPS uses this name // to help detect duplicate messages. AMPS enforces uniqueness for // client names when a transaction log is configured, and it is good // practice to always use unique client names. The using statement // ensures that the connection underlying the client is disposed of // before the program exists. Client implements the .NET IDisposable // interface, making it easy to ensure that connections are freed // when your Client is no longer in use. There is no need to disconnect // the Client when it is protected by a using statement. using(Client client = new Client("exampleClient")) { try { // At this point, we have a valid AMPS connection and can begin // to use it to publish and subscribe to messages. client.connect("tcp://192.168.1.3:9007/amps"); // This version of logon() uses the DefaultAuthenticator, which // provides the credentials in the URI (if any are present). To // use a different authentication scheme, implement an Authenticator // and pass that to this command. client.logon() // Here, we publish a single message to AMPS on the messages topic, // containing the data { "message" : "Hello, world!" }. // This JSON message is sent to the server. Upon successful completion // of this function, the AMPS client has sent the message to the server, // and subscribers to the messages topic will receive this message. client.publish("messages", @"{ ""message"" : ""Hello, World!"" }"); } catch (AMPSException e) { Console.Error.WriteLine(e); } } } } } ``` --- # Using a Heartbeat to Detect Disconnection The AMPS client includes a heartbeat feature to help applications detect disconnection from the server within a predictable amount of time. Without using a heartbeat, an application must rely on the operating system to notify the application when a disconnect occurs. For applications that are simply receiving messages, it can be impossible to tell whether a socket is disconnected or whether there are simply no incoming messages for the client. When you set a heartbeat, the AMPS client sends a heartbeat message to the AMPS server at a regular interval, and expects a response from the server within the specified amount of time. If the operating system reports an error on send, or if there is no activity received from the server within the specified amount of time, the AMPS client considers the server to be disconnected. Likewise, the server will ensure that traffic is sent to the client at the specified interval, using heartbeat messages when no other traffic is being sent to the client. If, after sending a heartbeat message, no traffic from the client arrives within a period twice the specified interval, the server will consider the client to be disconnected or nonresponsive. The AMPS client processes heartbeat messages on the client receive thread, which is the thread used for asynchronous message processing. If your application uses asynchronous message processing and occupies the thread for longer than the heartbeat interval, the client may fail to respond to heartbeat messages in a timely manner and may be disconnected by the server. --- # High Availability The AMPS C# client provides an easy way to create highly-available applications using AMPS, via the `HAClient` class. `HAClient` derives from `Client` and offers the same methods, but also adds protection against network, server, and client outages. Using `HAClient` allows applications to automatically: - Recover from temporary disconnects between client and server. - Failover from one server to another when a server becomes unavailable. Since the `HAClient` automatically manages failover and reconnection, 60East recommends using the `HAClient` for applications that need to: - Automatically reconnect and resume work in the case of disconnection. - Ensure no messages are lost or duplicated after a reconnect or failover. - Persist messages and bookmarks on disk for protection against client failure. You can choose how your application uses `HAClient` features. For example, you might need automatic reconnection, but have no need to resume subscriptions or republish messages. The high availability behavior in `HAClient` is provided by implementations of defined interfaces. You can combine different implementations provided by 60East to meet your needs, and implement those interfaces to provide your own policies. Some of these features require specific configuration settings on your AMPS instance(s). This chapter mentions these features and describes how to use them from the AMPS C# client, but you can find full documentation for these settings and server features in the *AMPS User Guide*. ## Overview of HAClient `HAClient` derives from `Client` and offers the same methods for sending commands to AMPS and receiving messages from AMPS. The `HAClient` differs from the `Client` in two ways: - The `HAClient` automatically installs a disconnect handler that reconnects to AMPS and resumes active (asynchronous) subscriptions. The disconnect handler optionally replays `publish` and `sow_delete` messages that have not been acknowledged by AMPS, using a `PublishStore`. The disconnect handler can optionally resume replays from the transaction log at a point that guarantees no messages are skipped and no duplicates are delivered to the application, using a `BookmarkStore`. - The `HAClient` includes the infrastructure needed for client failover, including a list of connection strings and their associated authentication mechanisms (provided by the `ServerChooser`), and options for controlling backoff behavior for reconnects (provided by the `DelayStrategy`). As a result, the `HAClient` provides a `connectAndLogon()` function for establishing a connection to AMPS, rather than treating these as independent steps that an application must manage itself. If your application needs to automatically reconnect to AMPS, 60East recommends using the `HAClient` and the automatically provided disconnect handler rather than using a `Client` or replacing the `HAClient` default disconnect handler. ## Reconnection with HAClient The most important difference between `Client` and `HAClient` is that `HAClient` automatically provides a reconnect handler. This description provides a high-level framework for understanding the components involved in failover with the `HAClient`. The components are described in more detail in the following sections. The `HAClient` reconnect handler performs the following steps when reconnecting: 1. Calls the `ServerChooser` to determine the next URI to connect to and the authenticator to use for that connection. If the connection fails, calls `getError` on the `ServerChooser` to get a description of the failure, sends an exception to the exception listener, and stops the reconnection process. 2. Calls the `DelayStrategy` to determine how long to wait before attempting to reconnect, and waits for that period of time. 3. Connects to the AMPS server. If the connection fails, calls `reportFailure` on the `ServerChooser` and begins the process again. 4. Logs on to the AMPS server. If the connection fails, calls `reportFailure` on the `ServerChooser` and begins the process again. 5. Calls `reportSuccess` on the ServerChooser. 6. Receives the bookmark for the last message that the server has persisted. Discards any older messages from the `PublishStore`. 7. Republishes any messages in the `PublishStore` that have not been persisted by the server. 8. Re-establishes subscriptions using the `SubscriptionManager` for the client. For bookmark subscriptions, the reconnect handler uses the `BookmarkStore` for the client to determine the most recent bookmark, and resubscribes with that bookmark. For subscriptions that do not use a bookmark, the `SubscriptionManager` simply re-enters the subscription, meaning that it is entered at the point at which the `HAClient` reconnects. The `ServerChooser`, `DelayStrategy`, `PublishStore`, `SubscriptionManager`, and `BookmarkStore` are all extension points for the `HAClient`. You can adapt the failover and recovery behavior by setting a different object for the behavior you want to customize on the `HAClient` or by providing your own implementation. For example, the convenience methods in the previous section customize the behavior of the `PublishStore` and `BookmarkStore` by providing either memory-backed or file-backed stores. ## Choosing Store Durability If your application needs reliable publish to AMPS, install a `PublishStore` in the `HAClient`. If your application needs to resume replays from the transaction log, install a `BookmarkStore` in the `HAClient`. These stores provide the following capabilities: - A *bookmark store* tracks received messages, and is used to resume subscriptions that replay from the transaction log. - A *publish store* tracks published messages, and is used to ensure that messages are persisted in AMPS. The AMPS C# Client provides a memory-backed version of each store and a file-backed version of each store. An `HAClient` can use either a memory backed store or a file backed store for protection. Each method provides resilience to different failures, as described below: - *Memory-backed stores* provide recovery after disconnection from AMPS by storing messages and bookmarks in your process' address space. This is the highest performance option for working with AMPS in a highly available manner. The trade-off with this method is there is no protection from a crash or failure of your client application. If your application is terminated prematurely or, if the application terminates at the same time as an AMPS instance failure or network outage, then messages may be lost or duplicated. The state of bookmark replays will be lost when the application shuts down. Messages in the publish store when the application shuts down will not be maintained through a restart, so the application will not be able to attempt any necessary redelivery when the application restarts. A memory-backed store should only be used by one instance of a client at a time. - *File-backed stores* protect against client failure and disconnection from AMPS by storing messages and bookmarks on disk. To use this protection method, the `createFileBacked` convenience method requests additional arguments for the two files that will be used for both bookmark storage and message storage. If these files exist and are non-empty (as they would be after a client application is restarted), the `HAClient` loads their contents and ensures synchronization with the AMPS server once connected. The performance of this option depends heavily on the speed of the device on which these files are placed. When the files do not exist (as they would the first time a client starts on a given system), the `HAClient` creates and initializes the files, and in this case the client does not have a point at which to resume the subscription or messages to republish. A store file should only be used by one instance of a client at a time. When using a file-backed bookmark store, 60East recommends periodically removing unneeded entries by calling the `prune()` method. The precise strategy that your application uses to call `prune()` depends on the nature of the application. Most applications call `prune()` when the application exits. There are two basic strategies that applications follow while the application runs: - Install a resize handler and call `prune()` after a specified number of resize operations, or when the store reaches a specific size. - Call `prune()` after a specific number of messages are processed (for example, every 10,000 messages received or every 1,000 updates completed). Regardless of the strategy, it is best to call `prune()` when the application is idle, since the `prune()` call rewrites the log file. The store interface is public, and an application can create and provide a custom store as necessary. While clients provide convenience methods for creating file-backed and memory-backed `HAClient` objects with the appropriate stores, you can also create and set the stores in your application code. The AMPS C# Client also includes default stores, which implement the appropriate interface, but do not actually persist messages. Starting in 5.3.2.0, the AMPS client contains a recovery point adapter interface to make it easy to add a custom persistence layer to a bookmark store. The distribution includes a recovery point adapter that can store bookmark recovery information in an AMPS SOW topic. The `HAClient` provides convenience methods for creating clients and setting stores. You can also construct an `HAClient` and set whichever store implementations you choose. In this example, we create several clients. The first client uses memory stores for both bookmarks and publishes. The second client uses files for both bookmarks and publishes. The third client uses a file for bookmarks. The third client does not set a store for publishes, which means that AMPS provides the default store (and no outgoing messages are stored). The final client does not specify any stores, so has no persistence for published messages or bookmark subscriptions, but can take advantage of the automatic failover and reconnection in the `HAClient`. ```csharp showLineNumbers // Memory publish store, memory bookmark store HAClient memoryClient = HAClient.createMemoryBacked("lessImportantMessages"); // File-backed publish store, file-backed bookmark store HAClient diskClient = HAClient.createFileBacked( "moreImportantMessages", "/mnt/fastDisk/moreImportantMessages.outgoing", "/mnt/fastDisk/moreImportantMessages.incoming"); // Default publish store, file-backed bookmark store HAClient subscriberClient = new HAClient("subscriber"); subscriberClient.setBookmarkStore(new LoggedBookmarkStore("/mnt/fastDisk/bookmark.store")); // Default publish store, default bookmark store // Failover behavior only HAClient streamReader = new HAClient("streamReader"); ``` :::info While this chapter presents the built-in file and memory-based stores, the AMPS C# Client provides open interfaces that allow development of custom persistent message stores. To fully control recovery behavior, you can implement the `Store` and `BookmarkStore` interfaces in your code, and then pass instances of those to `setPublishStore()` or `setBookmarkStore()` methods in your `Client`. You can also implement the `RecoveryPointAdapter` interface to easily add a custom storage mechanism to one of the 60East-provided bookmark store implementations. Instructions on developing a custom store are beyond the scope of this document; please refer to the *AMPS Client HA Whitepaper* for more information. ::: ### Using the SOW Recovery Point Adapter The AMPS client also includes the ability to use a SOW topic to store bookmark state for a bookmark store. This can be a useful option in a situation where an application needs a persistent bookmark store, but does not have the ability to store a file on the filesystem, or where an application has a bookmark file, but wants to have the ability to resume the subscription if the file is lost or damaged, or if the application is started on a system that does not have access to the file. To use the SOW topic recovery point adapter, you create a bookmark store of the type you would like to use for the `Client`, passing an adapter when you construct the store. You then set this bookmark store as the store for the `Client` to use. The constructor for the SOW recovery adapter allows you to customize the topic name and field names used to store the recovery point information in AMPS. As with the `RecoveryPointAdapter` interface in general, it is possible to customize the behavior of the SOW recovery point adapter by overriding the provided methods. This section describes how to use the adapter with the default settings. Should you need to change the behavior of the class, you would adjust the guidance in this section accordingly. (For example, if you override methods to produce a message with a different set of keys or a different message format, you would update the topic definition accordingly). ### AMPS Topic Configuration To store recovery point state in AMPS, the AMPS instance that will store the recovery point state must define a `SOW/Topic` to hold the recovery point data. By default, the adapter uses a topic named `/ADMIN/bookmark_store` of `json` message type, with the `/clientName` and `/subId` fields as keys, similar to the following definition: ```xml showLineNumbers /ADMIN/bookmark_store json /clientName /subId ``` You must include this definition, or an equivalent definition, in the configuration file for the AMPS instance that will host the recovery point. If you define a topic with a different configuration (for example, different key names, a different topic name or a different message type), you must ensure that the adapter that you create uses the same parameters as those configured on the server. ### Constructing a Client for the Adapter The AMPS SOW Recovery Point Adapter requires a `Client` or `HAClient` connected to the instance that contains the SOW topic. The Adapter will use this client to recover bookmark state and store bookmarks in AMPS. Notice that this client **must not** be a client that the Adapter is keeping state for. This must be a completely separate client instance, otherwise the client may deadlock while updating the store. The client must be connected and logged in to the instance that contains the SOW topic, using the message type defined for the topic. ### Capacity Planning and Store Sizing When an application uses a file-backed store, it is important to make sure that there is enough space available on the file system to be able to manage the store. For logged bookmark stores, an application needs to keep a bookmark record for each message received, each message discarded, and the persisted acknowledgments delivered by the server approximately once a second. Each bookmark entry consumes roughly 70 bytes of storage *plus* the length of the subscription ID for the subscription receiving the message. The logged bookmark store retains entries until an application explicitly calls `prune()`. The capacity needed for a logged bookmark store will depend on the strategy that the application uses for pruning the file. For a file-backed publish store, the application needs to be able to store published messages until the AMPS server that the publisher is connected to acknowledges those messages as persisted. The volume of messages that needs to be stored depends on the failover policy for the server -- that is, the maximum amount of time that the server will allow a downstream instance to fail to acknowledge a message before the server downgrades that connection to `async` acknowledgment. By default, AMPS does not downgrade connections: this policy must be set explicitly using the AMPS actions. As an example, if the server is configured to downgrade connections that are more than 120 seconds behind, then -- for disaster recovery -- the application must have the capacity to store 120 seconds of published messages at peak publishing load. However, unlike the logged bookmark store, a file-backed publish store removes messages from the store and reuses the space once AMPS has acknowledged the message. ## Connections and the Server Chooser Unlike `Client`, the `HAClient` attempts to keep itself connected to an AMPS instance at all times, by automatically reconnecting or failing over when it detects that the client is disconnected. When you are using the `Client` directly, your disconnect handler usually takes care of reconnection. `HAClient`, on the other hand, provides a disconnect handler that automatically reconnects to the current server or to the next available server. To inform the `HAClient` of the addresses of the AMPS instances in your system, you pass a `ServerChooser` instance to the `HAClient`. `ServerChooser` acts as a smart enumerator over the servers available: `HAClient` calls `ServerChooser` methods to inquire about what server should be connected, and also calls methods to indicate whether a given server succeeded or failed. The AMPS C# Client provides a simple implementation of `ServerChooser`, called `DefaultServerChooser`, that provides very simple logic for reconnecting. This server chooser is most suitable for basic testing, or in cases where an application should simply rotate through a list of servers. For most applications, you implement the `ServerChooser` interface yourself for more advanced logic, such as choosing a backup server based on your network topology, or limiting the number of times your application should try to reconnect to a given address. To connect to AMPS, you provide a `ServerChooser` to `HAClient` and then invoke `connectAndLogon()` to create the first connection: ```csharp showLineNumbers HAClient myClient = HAClient.createMemoryBacked( "myClient"); // primary.amps.xyz.com is the primary AMPS instance, and // secondary.amps.xyz.com is the secondary DefaultServerChooser chooser = new DefaultServerChooser(); chooser.add("tcp://primary.amps.xyz.com:12345/fix"); chooser.add("tcp://secondary.amps.xyz.com:12345/fix"); myClient.setServerChooser(chooser); myClient.connectAndLogon(); ... myClient.disconnect(); ``` Similar to `Client`, `HAClient` remains connected to the server until `disconnect()` is called. Unlike `Client`, `HAClient` automatically attempts to reconnect to your server if it detects a disconnect and, if that server cannot be connected, fails over to the next server provided by the `ServerChooser`. In this example, the call to `connectAndLogon()` attempts to connect and log in to `primary.amps.xyz.com`, and returns if that is successful. If it cannot connect, it tries `secondary.amps.xyz.com`, and continues trying servers from the `ServerChooser` until a connection is established. Likewise, if it detects a disconnection while the client is in use, `HAClient` attempts to reconnect to the server with which it was most recently connected; if that is not possible, it moves on to the next server provided by the `ServerChooser`. :::info While this chapter presents the built-in file and memory-based stores, the AMPS C# Client provides open interfaces that allow development of custom persistent message stores. You can implement the `Store` and `BookmarkStore` interfaces in your code, and then pass instances of those to `setPublishStore()` or `setBookmarkStore()` methods in your `Client`. Instructions on developing a custom store are beyond the scope of this document; please contact 60East support for more information. ::: ### Setting a Reconnect Delay and Timeout You can control the amount of time between reconnection attempts and set a total amount of time for the `HAClient` to attempt to reconnect. The AMPS C# Client includes an interface for managing this behavior called the `ReconnectDelayStrategy`. Two implementations of this interface are provided with the client: - `FixedDelayStrategy` provides the same delay each time the `HAClient` tries to reconnect. - `ExponentialDelayStrategy` provides an exponential backoff until a connection attempt succeeds. To use either of these classes, you simply create an instance, set the appropriate parameters, and install that instance as the delay strategy for the `HAClient`. For example, the following code sets up a reconnect delay that starts at 200ms and increases the delay by 1.5 times after each failure. The strategy allows a maximum delay of 5 seconds, and will not retry longer than 60 seconds. ```csharp showLineNumbers HAClient theClient = HAClient.createMemoryBacked("demo"); ExponentialDelayStrategy theStrategy = new ExponentialDelayStrategy(); theStrategy.setInitialDelay(200); theStrategy.setBackoffExponent(1.5); theStrategy.setMaximumDelay(5000); theStrategy.setMaximumRetryTime(60000); theClient.setDelayStrategy(theStrategy); ``` ### Implementing a Server Chooser As described above, you provide the `HAClient` with connection strings to one or more AMPS servers using a `ServerChooser`. The purpose of a `ServerChooser` is to provide information to the `HAClient`. A `ServerChooser` does not manage the reconnection process, and should not call methods on the `HAClient`. A `ServerChooser` has two required responsibilities to the `HAClient`: - Tells the `HAClient` the connection string for the server to connect to. If there are no servers, or the `ServerChooser` wants the connection to fail, the `ServerChooser` returns an empty string. To provide this information, the `ServerChooser` implements the `getCurrentURI()` method. - Provides an `Authenticator` for the current connection string. This is especially important for installations where different servers require different credentials or authentication tokens must be reset after each connection attempt. To provide the authenticator, the `ServerChooser` implements the `getCurrentAuthenticator()` method. The `HAClient` calls the `getCurrentURI()` and `getCurrentAuthenticator()` methods each time it needs to make a connection. Each time a connection succeeds, the `HAClient` calls the `reportSuccess()` method of the `ServerChooser`. Each time a connection fails, the `HAClient` calls the `reportFailure()` method of the `ServerChooser`. The `HAClient` does not require the `ServerChooser` to take any particular action when it calls these methods. These methods are provided for the `HAClient` to do internal maintenance, logging, or record keeping. For example, an `HAClient` might keep a list of available URIs with a current failure count, and skip over URIs that have failed more than 5 consecutive times until all URIs in the list have failed more than 5 consecutive times. When the `ServerChooser` returns an empty string from `getCurrentURI()`, indicating that no servers are available for connection, the `HAClient` calls the `getError()` method on the `ServerChooser` and includes the string returned by `getError()` in the generated exception. ## Heartbeats and Failure Detection Use of the `HAClient` allows your application to quickly recover from detected connection failures. By default, connection failure detection occurs when AMPS receives an operating system error on the connection. This system may result in unpredictable delays in detecting a connection failure on the client, particularly when failures in network routing hardware occur, and the client primarily acts as a subscriber. The heartbeat feature of the AMPS client allows connection failure to be detected quickly. Heartbeats ensure that regular messages are sent between the AMPS client and server on a predictable schedule. The AMPS client and server both assume disconnection has occurred if these regular heartbeats cease, ensuring disconnection is detected in a timely manner. To utilize the heartbeat feature, call the `setHeartbeat` method on `Client` or `HAClient`: ```csharp showLineNumbers HAClient client = HAClient.createMemoryBacked("importantStuff"); ... client.setHeartbeat(3); client.connectAndLogon(); ... ``` Method `setHeartbeat` takes one parameter: the heartbeat interval. The heartbeat interval specifies the periodicity of heartbeat messages sent by the server: the value `3` indicates messages are sent on a three-second interval. If the client receives no messages in a six-second window (two heartbeat intervals), the connection is assumed to be dead, and the `HAClient` attempts reconnection. An additional variant of `setHeartbeat` allows the idle period to be set to a value other than two heartbeat intervals. (The server, however, will always consider a connection to be broken after two heartbeat intervals without any traffic.) Notice that, for `HAClient`, `setHeartbeat` must be called *before* the client is connected. For `Client`, `setHeartbeat` must be called *after* the client is connected. :::warning Heartbeats are serviced on the receive thread created by the AMPS client. Your application must not block the receive thread for longer than the heartbeat interval, or the application is subject to being disconnected. ::: ## Considerations for Publishers Publishing with an `HAClient` is nearly identical to regular publishing; you simply call the `publish()` method with your message's topic and data. The AMPS client sends the message to AMPS, and then returns from the `publish()` call. For maximum performance, the client does not wait for the AMPS server to acknowledge that the message has been received. When an `HAClient` uses a publish store (other than the `DefaultPublishStore`), the publish store retains a copy of each outgoing message and requests that AMPS acknowledge that the message has been persisted. The AMPS server acknowledges messages back to the publisher. Acknowledgments can be delivered for multiple messages at periodic intervals (for topics recorded in the transaction log) or after each message (for topics that are not recorded in the transaction log). When an acknowledgment for a message is received, the `HAClient` removes that message from the bookmark store. When a connection to a server is made, the `HAClient` automatically determines which messages from the publish store (if any) the server has not processed, and replays those messages to the server once the connection is established. For reliable publishers, the application must choose how best to handle application shutdown. For example, it is possible for the network to fail immediately after the publisher sends the message, while the message is still in transit. In this case, the publisher has sent the message, but the server has not processed it and acknowledged it. During normal operation, the `HAClient` will automatically connect and retry the message. On shutdown, however, the application must decide whether to wait for messages to be acknowledged, or whether to exit. Publish store implementations provide an `unpersistedCount()` method that reports the number of messages that have not yet been acknowledged by the AMPS server. When the `unpersistedCount()` reaches `0`, there are no unpersisted messages in the local publish store. For the highest level of safety, an application can wait until the `unpersistedCount()` reaches `0`, which indicates that all of the messages have been persisted to the instance that the application is connected to, and the synchronous replication destinations configured for that instance. When a synchronous replication destination goes offline, this approach will cause the publisher to wait to exit until the destination comes back online or until the destination is downgraded to asynchronous replication. For applications that are shut down periodically for short periods of time (for example, applications that are only offline during a weekly maintenance window), another approach is to use the `publishFlush()` method to ensure that messages are delivered to AMPS, and then rely on the connection logic to replay messages as necessary when the application restarts. For example, the following code flushes messages to AMPS, then warns if not all messages have been acknowledged: ```csharp showLineNumbers HAClient pub = HAClient.createMemoryBacked( "importantStuff"); ... pub.connectAndLogon(); // Publish messages ... // We think we are done, but the server may not // have received or acknowledged the messages yet. // Wait until the server has received all messages. // The program could also specify a timeout in this // command to avoid blocking forever if the // network is down or all servers are offline. pub.publishFlush(); // Print warning to the console if messages have // been published but not yet acknowledged as persisted. if (pub.getPublishStore().unpersistedCount() > 0) { Console.WriteLine("all messages have been published,"+ " but not all have been persisted"); } pub.disconnect(); ``` In this example, the client sends each message immediately when `publish()` is called. If AMPS becomes unavailable between the final `publish()` and the `disconnect()`, or one of the servers that the AMPS instance replicates to is offline, the client may not have received a persisted acknowledgment for all of the published messages. For example, if a message has not yet been persisted by all of the servers in the replication fabric that are connected with synchronous replication, AMPS will not have acknowledged the message. Before shutting down the client, the code does two things: - First, the code flushes messages to the server to ensure that all messages have been delivered to AMPS. - Next, the code checks to see if all of the messages in the publish store have been acknowledged as persisted by AMPS. If the messages have not been acknowledged, they will remain in the publish store file and will be published to AMPS, if necessary, the next time the application connects. An application may choose to loop until `unpersistedCount()` returns `0`, or (as we do in this case) simply warn that AMPS has not confirmed that the messages are fully persisted. The behavior you choose in your application should be consistent with the high-availability guarantees your application needs to provide. :::warning AMPS uses the name of the `HAClient` to determine the origin of messages. For the AMPS server to correctly identify duplicate messages, each instance of an application that publishes messages must use a distinct name. That name must be consistent across different runs of the application. ::: If your application crashes or is terminated, some published messages may not have been persisted in the AMPS server. If you use the file-based store (in other words, the store created by using `HAClient.createFileBacked()`), the `HAClient` will recover the messages, and once logged on, correlate the message store to what the AMPS server has received, re-publishing any missing messages. This occurs automatically when `HAClient` connects, without any explicit consideration in your code, other than ensuring that the same file name is passed to `createFileBacked()` if recovery is desired. :::warning AMPS provides persisted acknowledgment messages for topics that do not have a transaction log enabled; however, the level of durability provided for topics with no transaction log is minimal. Learn more about transaction logs in the *AMPS User Guide*. ::: ## Considerations for Subscribers `HAClient` provides two important features for applications that subscribe to one or more topics: re-subscription, and a bookmark store to track the correct point at which to resume a bookmark subscription. ### Resubscription with Asynchronous Message Processing Any asynchronous subscription placed using an `HAClient` is automatically reinstated after a disconnect or a failover. These subscriptions are placed in an in-memory `SubscriptionManager`, which is created automatically when the `HAClient` is instantiated. Most applications will use this built-in subscription manager, but for applications that create a varying number of subscriptions, you may wish to implement `SubscriptionManager` to store subscriptions in a more durable place. Note that these subscriptions contain no message data, but rather simply contain the parameters of the subscription itself (for instance, the command, topic, message handler, options, and filter). When a re-subscription occurs, the AMPS C# Client re-executes the command as originally submitted, including the original topic, options, and so on. AMPS sends the subscriber any messages for the specified topic (or topic expression) that are published after the subscription is placed. For a `sow_and_subscribe` command, this means that the client re-issues the full command, including the SOW query as well as the subscription. :::tip A `sow` command is a point-in-time query. It isn't added to the subscription manager, and isn't restarted if a disconnection happens in the middle of a query. A `sow_and_subscribe` is a subscription, and is added to the subscription manager. ::: ### Resubscription with Synchronous Message Processing The `HAClient` (starting with the AMPS C# Client version 4.3.1.1) does not track synchronous message processing subscriptions in the `SubscriptionManager`. Once the `MessageStream` indicates that there are no more elements in the stream, you can consider the stream to be closed. The `MessageStream` does not suddenly produce more elements. To re-subscribe when the `HAClient` fails over, you can simply re-issue the subscription. For example, the snippet below re-issues the subscribe command when the message stream ends: ```csharp showLineNumbers boolean still_need_to_process = true; while (still_need_to_process == true) { MessageStream ms = client.subscribe("topic"); try { for (Message m : ms) { // process message // check condition on still_need_to_process if (still_need_to_process == false) break; } // end of stream, for a subscribe this means // that the connection is likely closed. } finally { if (ms != null) ms.close(); } } ``` ### Bookmark Stores In cases where it is critical not to miss a single message, it is important to be able to resume a subscription at the exact point that a failure occurred. In this case, simply recreating a subscription isn't sufficient. Even though the subscription is recreated, the subscriber may have been disconnected at precisely the wrong time, and will not see the message. To ensure delivery of every message from a topic or set of topics, the AMPS `HAClient` includes a `BookmarkStore` that, combined with the bookmark subscription and transaction log functionality in the AMPS server, ensures that clients receive any messages that might have been missed. The client stores the bookmark associated with each message received, and tracks whether the application has processed that message; if a disconnect occurs, the client uses the `BookmarkStore` to determine the correct resubscription point, and sends that bookmark to AMPS when it re-subscribes. AMPS then replays messages from its transaction log from the point after the specified bookmark, thus ensuring the client is completely up-to-date. `HAClient` helps you to take advantage of this bookmark mechanism through the `BookmarkStore` interface and `bookmarkSubscribe()` method on `Client`. When you create subscriptions with `bookmarkSubscribe()`, whenever a disconnection or failover occurs, your application automatically re-subscribes to the message after the last message it processed. `HAClients` created by `createFileBacked()` additionally store these bookmarks on disk, so that the application can restart with the appropriate message if the client application fails and restarts. To take advantage of bookmark subscriptions, do the following: - Ensure the topic(s) to be subscribed to are included in a transaction log. See the *AMPS User Guide* for information on how to specify the contents of a transaction log. - Use `bookmarkSubscribe()` instead of `subscribe()` when creating a `subscription()`, and decide how the application will manage subscription identifiers (SubIds). If you are using a `Command` object, you can simply provide a bookmark on that object. - Use the `BookmarkStore.discard()` method in message handlers to indicate when a message has been fully processed by the application, that is, when the application does not need to receive the message again if the application fails over. The following example creates a bookmark subscription against a transaction-logged topic, and fully processes each message as soon as it is delivered: ```csharp showLineNumbers final HAClient client = HAClient.createFileBacked( "aClient", "/logs/aClient.publishLog", "/logs/aClient.subscribeLog"); class MyMessageHandler implements MessageHandler { public void invoke(Message message) { ... client.getBookmarkStore().discard( message.getSubIdRaw(), message.getBookmarkSeqNo()); ... } } ... // Set the commandId to a previously saved GUID. Guid cmdIdGuid = new Guid("0066e1dc-9cfd-4b02-934b-2376a52cb412"); String cmdIdData = Convert.ToBase64String(cmdIdGuid.ToByteArray(), 0, 16); CommandId cmdId = new CommandId(); cmdId.set(System.Text.Encoding.UTF8.GetBytes(cmdIdData), 0, 24); Command command = new Command("subscribe") .setTopic("myTopic") .setSubId(cmdId) .setBookMark(Client.Bookmarks.MOST_RECENT); client.executeAsync(command, new MyMessageHandler()); ``` In this example, the client is a file-backed client, meaning that arriving bookmarks will be stored in a file (`Client.subscribeLog`). Storing these bookmarks in a file allows the application to restart the subscription from the last message processed, in the event of either server or client failure. :::tip For optimum performance, it is critical to discard every message once its processing is complete. If a message is never discarded, it remains in the bookmark store. During re-subscription, `HAClient` always restarts the bookmark subscription with the oldest undiscarded message, and then filters out any more recent messages that have been discarded. If an old message remains in the store, but is no longer important for the application’s functioning, the client and the AMPS server will incur unnecessary network, disk, and CPU activity. ::: The `subscriptionId` parameter specifies an identifier to be used for this subscription. Passing null, or leaving the field unset, causes `HAClient` to generate a subscription ID, like most other `Client` functions. However, if you wish to resume a subscription from a previous point after the application has terminated and restarted, the application must pass the same subscription ID as during its previous run. Passing a different subscription ID bypasses any recovery mechanisms, creating an entirely new subscription. When you use an existing subscription ID, the `HAClient` locates the last-used bookmark for that subscription in the local store, and attempts to re-subscribe from that point. Below are the different bookmark types that can be used to enable different recovery strategies for an application: - `Client.Bookmarks.NOW` specifies that the subscription should begin from the moment the server receives the subscription request. This results in the same messages being delivered as if you had invoked `subscribe()` instead, except that the messages will be accompanied by bookmarks. This is also the behavior that results if you supply an invalid bookmark. - `Client.Bookmarks.EPOCH` specifies that the subscription should begin from the beginning of the AMPS transaction log (that is, the first entry in the oldest journal file for the transaction log). - `Client.Bookmarks.MOST_RECENT` specifies that the subscription should begin from the last-used message in the associated `BookmarkStore`. Alternatively, if this subscription has not been seen before, to begin with `EPOCH`. This is the most common value for this parameter, and is the value used in the preceding example. By using `MOST_RECENT`, the application automatically resumes from wherever the subscription left off, taking into account any messages that have already been processed and discarded. When the `HAClient` re-subscribes after a disconnection and reconnection, it always uses `MOST_RECENT`, ensuring that the continued subscription always begins from the last message used before the disconnect, so that no messages are missed. ## Conclusion With only a few changes, most AMPS applications can take advantage of the `HAClient` and associated classes to become more highly-available and resilient. Using the `PublishStore`, publishers can ensure that every message published has actually been persisted by AMPS. Using `BookmarkStore`, subscribers can make sure that there are no gaps or duplicates in the messages received. `HAClient` makes both kinds of applications more resilient to network and server outages, as well as temporary issues. By using the file based `HAClient`, clients can recover their state after an unexpected termination or crash. Though `HAClient` provides useful defaults for the `Store`, `BookmarkStore`, `SubscriptionManager`, and `ServerChooser`, you can customize any or all of these to the specific needs of your application and architecture. --- # Obtaining and Installing the AMPS C#/.NET Client ## Obtaining the Client You must first download and install the client on your development computer. This can be accomplished through any of the following methods: **Use the AMPS Client executable installer** For this option, download the `amps-csharp-client-.exe`, where \ is replaced by the version of the client (such as `amps-csharp-client-3.3.0.exe`). Double-click the \*.exe file to launch the installation wizard. Once the installation completes, you will be able to find the installed client under your computer's `Program Files` directory, in a subdirectory entitled AMPS. **Unpack the AMPS Client zip file** For this option, download the `amps-csharp-client-.zip` file from the [60East Technologies](https://www.crankuptheamps.com/develop/) website or copy it from the AMPS server installation directory. Save the zip file to your development computer. Right-click the `amps-csharp-client.zip` file, and choose `Extract` to extract the contents of the zip file. You're welcome to extract the AMPS client to wherever suits your needs; we'll refer to that directory as the `AMPS` directory for the remainder of this guide. **Use nuget to install the client** For this option, use `nuget` to install the `AMPS.Client` package. This option provides less control over the installation, and does not include the source code for the client, samples, or documentation, which makes it a less-preferable option for many installations. The documentation for the client is available in the `AMPS.Docs` package. The `AMPS.Docs` package includes only documentation, and is intended for use at sites with policies that do not allow download of archives from the 60East website. ## Test Connectivity to AMPS Before writing programs using AMPS, make sure connectivity to an AMPS server from this computer is working. Launch a terminal window and change the directory to the `AMPS` directory in your AMPS installation, and use `spark` to test connectivity to your server, for example: ```bash ./bin/spark ping -type fix -server 192.168.1.2:9004 ``` If you receive an error message, verify that your AMPS server is up and running, and work with your systems administrator to determine the cause of the connectivity issues. Without connectivity to AMPS, you will be unable to make the best use of this guide. --- # Managing Disconnection The `HAClient` class, included with the AMPS C#/.NET client, contains a disconnect handler and other features for building highly-available applications. The `HAClient` includes features for managing a list of failover servers, resuming subscriptions, republishing in-flight messages, and other functionality that is commonly needed for high availability. 60East recommends using the `HAClient` for automatic reconnection wherever possible, as the HAClient disconnect handler has been carefully crafted to handle a wide variety of edge cases and potential failures. If an application needs to reconnect or fail over, use an `HAClient`, and the AMPS client library will automatically handle failover and reconnection. You control which servers the client fails over to using an implementation of the `ServerChooser` interface, and you can control the timing of the failover using an implementation of the `ReconnectDelayStrategy` interface. :::info For most applications, the combination of the `HAClient` disconnect handler and a `ConnectionStateListener` gives you the ability to monitor disconnections and add custom behavior at the appropriate point in the reconnection process. ::: If you need to add custom behavior to the failover (such as logging, resetting an internal cache, refreshing credentials and so on), the `ConnectionStateListener` class allows your application to be notified and take action when disconnection is detected and at each stage of the reconnection process. To extend the behavior of the AMPS client during reconnection, implement a `ConnectionStateListener`. --- # Managing SOW Contents AMPS allows applications to manage the contents of the SOW by explicitly deleting messages that are no longer relevant. For example, if a particular delivery van is retired from service, the application can remove the record for the van by deleting the record for the van. The client provides the following methods for deleting records from the SOW: - `sowDelete` - Accepts a filter, and deletes all messages that match the filter. - `sowDeleteByKeys` - Accepts a set of SOW keys as a comma-delimited string and deletes messages for those keys, regardless of the contents of the messages. SOW keys are provided in the header of a SOW message, and is the internal identifier AMPS uses for that SOW message. - `sowDeleteByData` - Accepts a message, and deletes the record that would be updated by that message. The most efficient way to remove messages from the SOW is to use `sowDeleteByKeys` or `sowDeleteByData`, since those options allow AMPS to exactly target the message or messages to be removed. Many applications use `sowDelete`, since this is the most flexible method for removing items from the SOW when the application does not have information on the exact messages to be removed. Regardless of the command used, AMPS sends an OOF message to all subscribers who have received updates for the messages removed, as described in the previous section. The simple form of the `sowDelete` command returns a `MessageStream` that receives the response. This response is an acknowledgment message that contains information on the delete command. For example, the following snippet simply prints informational text with the number of messages deleted: ```csharp showLineNumbers foreach (Message msg in client.SowDelete("sow_topic", "/id IN (42, 64, 37)")) { System.Console.WriteLine("Got an {0} containing {1} : " + "deleted {2} messages.", msg.Command, msg.AckType, msg.Matches); } ``` In either case, AMPS sends an OOF message to all subscribers who have received updates for the messages removed, as described in the previous section. Acknowledging messages from a queue uses a form of the `sow_delete` command that is only supported for queues. Acknowledgment is discussed in the [Using Queues](queues) chapter in this guide. --- # Manual Acknowledgment 60East generally recommends that applications use an `ack()` method to acknowledge messages during normal processing. This approach functions correctly when used within a message handler, supports batching as explained elsewhere in this chapter, and is generally both easier to code and more efficient. However, in some situations, you may need to manually acknowledge messages in the queue. This is most common when an application needs to operate on all messages with certain characteristics, rather than acknowledging individual messages. For example, an application that is doing updates to an order may want to cancel an order by both publishing a cancellation and immediately expiring all other messages in the queue for that order. With manual acknowledgment, that application can use a filter to remove all previous updates for that order, then publish the cancellation. To manually acknowledge processed messages and remove the messages from the queue, applications use the `sow_delete` command. To remove specific messages from the queue, provide the bookmarks of those messages. To remove messages that match a given filter, provide the filter. Notice that AMPS only supports using a bookmark with `sow_delete` when removing messages from a queue, not when removing records from a SOW. For example, given a `Message` object to acknowledge and a client, the code below acknowledges the message. ```csharp showLineNumbers void acknowledgeSingle(Client client, Message message) { Command acknowledge = Command("sow_delete"); acknowledge.setTopic(message.getTopic()) .setBookmark(message.getBookmark()); client.executeAsync(acknowledge, null); } ``` In the example above, the program creates a `sow_delete` command, specifies the topic and the bookmark, and then sends the command to the server. While this method works, creating and sending an acknowledgment for each individual message can be inefficient if your application is processing a large volume of messages. Rather than acknowledging each message individually, your application can build a comma-delimited list of bookmarks from the processed messages and acknowledge all of the messages at the same time. In this case, it's important to be sure that the number of messages you wait for is less than the maximum backlog -- the number of messages your client can have unacknowledged at a given time. Notice that both automatic acknowledgment and the helper method on the `Message` object take the maximum backlog into account. When constructing a command to acknowledge queue messages, AMPS allows an application to specify a filter rather than a set of bookmarks. AMPS interprets this as the client requesting acknowledgment of all messages that match the filter. (This may include messages that the client has not received, subject to the `Leasing` model for the queue.) As a more typical example of manual acknowledgment, the code below expires all messages for a given `id` that have a status other than `cancel`. An application might do this to halt processing of an order that it is about to cancel. ```csharp showLineNumbers void removePending(Client client, string orderId) { Command acknowledge = Command("sow_delete"); acknowledge.setTopic(message.getTopic()) .setFilter("/id = '" + orderId + "' and /status != 'cancel'") .setOptions("expire"); client.executeAsync(acknowledge, null); } ``` In the example above, the program specifies a topic and a filter to use to find the messages that should be removed. In this case, the program also provides the `expire` option to indicate that the messages have been removed from the queue rather than successfully processed (of course, whether this is the correct behavior for a canceled order depends on the expected message flow for your application). Notice that, as described in [Understanding Threading](understanding-threading-section.md), this method of acknowledging a message should not be used from a message handler unless the `sow_delete` is sent from a different client than the client that called the message handler. Instead, 60East recommends using the `ack()` function from within a message handler. --- # Understanding Message Objects So far, we have seen that subscribing to a topic involves working with objects of type `AMPS.Client.Message`. A `Message` represents a single message to or from an AMPS server. Messages are received or sent for every client/server operation in AMPS. ## Header Properties There are two parts of each message in AMPS: a set of headers that provide metadata for the message, and the data that the message contains. Every AMPS message has one or more header fields defined. The precise headers present depend on the type and context of the message. There are many possible fields in any given message, but only a few are used for any given message. For each header field, the `Message` class contains a distinct property that allows for retrieval and setting of that field. For example, the `Message.getCommandId()` function corresponds to the `commandId` header field, the `Message.getBatchSize()` function corresponds to the `BatchSize` header field, and so on. For more information on these header fields, consult the *AMPS User Guide* and *AMPS Command Reference*. To work with header fields, a `Message` contains `getXxx()`/`setXxx()` methods corresponding to the header fields. 60East does not recommend attempting to parse header fields from the raw data of the message. In AMPS, fields sometimes need to be set to a unique identifier value. For example, when creating a new subscription, or sending a manually constructed message, you’ll need to assign a new unique identifier to multiple fields such as `CommandId` and `SubscriptionId`. For this purpose, `Message` provides `newXxx()` methods for each field that generates a new unique identifier and sets the field to that new value. ## Data Property Access to the data section of a message is provided via the `Data` property. The `Data` property will contain the unparsed data of the message. The `Data` property returns the data as a .Net string, which is suitable for message formats that can be represented as Unicode text, such as JSON, XML, FIX, or NVFIX. For binary data, the AMPS C# client provides a `getDataRaw()` method to allow you to work with the underlying byte array in the message. See the section on [Byte Buffers](./advanced-topics.md#working-with-messages-and-byte-buffers) for details. The AMPS C# client contains a collection of helper classes for working with message types that are specific to AMPS (for example, FIX, NVFIX, and AMPS composite message types). For message types that are widely used, such as JSON or XML, you can use whichever library you typically use in your environment. ## Message Field Reference The [AMPS Command Reference](/docs/amps-command-reference) contains a full description of which fields are available and which fields are returned in response to specific commands. --- # Monitoring Connection State The AMPS client interface provides the ability to set one or more connection state listeners. A connection state listener is a callback that is invoked when the AMPS client detects a change to the connection state. A connection state listener may be called from the client receive thread. An application should not submit commands to AMPS from a connection state listener, or the application risks creating a deadlock for commands that wait for acknowledgement from the server. The AMPS client provides the following state values for a connection state listener: |State |Indicates | |--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`Connected` |

    The client has established a connection to AMPS. If you are using a `Client`, this is delivered when `connect()` is successful.

    If you are using an `HAClient`, this state indicates that the `connect` part of the connect and logon process has completed. An `HAClient` using the default disconnect handler will attempt to log on immediately after delivering this state.

    Most applications that use `Client` will attempt to log on immediately after the call to `connect()` returns.

    An application should not submit commands to AMPS from the connection state listener while the client is in this state unless the application knows that the state has been delivered from a `Client` and that the `Client` does not call `logon()`.

    | |`LoggedOn` |

    The client has successfully logged on to AMPS. If you are using a `Client`, this is delivered when `logon()` is successful.

    If you are using an `HAClient`, this state indicates that the `logon` part of the connect and logon process has completed.

    This state is delivered after the client is logged on, but before recovery of client state is complete. Recovery will continue after delivering this state: the application should not submit commands to AMPS from the connection state listener while the client is in this state if further recovery will take place.

    | |`HeartbeatInitiated`|

    The client has successfully started heartbeat monitoring with AMPS. This state is delivered if the application has enabled heartbeating on the client.

    This state is delivered before recovery of the client state is complete. Recovery may continue after this state is delivered. The application should not submit commands to AMPS from the connection state listener until the client is completely recovered.

    | |`PublishReplayed` |

    Delivered when a client has completed replay of the publish store when recovering after connecting to AMPS.

    This state is delivered when the client has a PublishStore configured.

    If the client has a subscription manager set, (which is the default for an `HAClient`), the application should not submit commands from the connection state listener until the `Resubscribed` state is received.

    | |`Resubscribed` |

    Delivered when a client has re-entered subscriptions when recovering after connecting to AMPS.

    This state is delivered when the client has a subscription manager set (which is the default for an `HAClient`). This is the final recovery step. An application can submit commands to AMPS from the connection state listener after receiving this state.

    | |`Disconnected` |The client is not connected. For an `HAClient`, this means that the client will attempt to reconnect to AMPS. For a `Client`, this means that the client will invoke the disconnect handler, if one is specified.| |`Shutdown` |The client is shut down. For an `HAClient`, this means that the client will no longer attempt to reconnect to AMPS. This state is delivered when `close()` is called on the client or when a server chooser tells the `HAClient` to stop reconnecting to AMPS.| The enumeration provided for the connection state listener also includes a value of `UNKNOWN`, for use as a default or to represent additional states in a custom `Client` implementation. The 60East implementations of the client do not deliver this state. The following table shows examples of the set of states that will be delivered during connection, in order, depending on what features of the client are set. Notice that, for an instance of the `Client` class, this table assumes that the application calls both `connect()` and `logon()`. For an `HAClient`, this table assumes that the `HAClient` is using the default `DisconnectHandler` for the `HAClient`. |Configuration |States | |----------------------------------------------------------------------|------------------------------------------------------------------------------------------------------| |

    subscription manager

    publish store

    |

    `Connected`

    `LoggedOn`

    `PublishReplayed`

    `Resubscribed`

    | |

    subscription manager

    publish store

    heartbeat set

    |

    `Connected`

    `LoggedOn`

    `HeartbeatInitiated`

    `PublishReplayed`

    `Resubscribed`

    | |subscription manager |

    `Connected`

    `LoggedOn`

    `Resubscribed`

    | |

    subscription manager

    heartbeat set

    |

    `Connected`

    `LoggedOn`

    `HeartbeatInitiated`

    `Resubscribed`

    | |(default `Client` configuration) |

    `Connected`

    `LoggedOn`

    | --- # Performance Tips and Best Practices This chapter presents tips and techniques for writing high-performance applications with AMPS. This section presents principles and approaches that describe how to use the features of AMPS and the AMPS client libraries to achieve high performance and reliability. Specific techniques (for example, the details on how to write a message handler) are described in other parts of the AMPS documentation and referenced here. Other techniques require information specific to the application (for example, determining the minimum set of information required in a message), and are best done as part of your application design. All of the recommendations in this section are general guidelines. There are few, if any, universal rules for performance: at times, a design decision that is absolutely necessary to meet the requirements for an application might reduce performance somewhat. For example, your application might involve sending large binary data that cannot be incrementally updated. That application will use more bandwidth per message than an application that sends 100-byte messages with fields that can be incrementally updated. However, since the application depends on being able to deliver the binary payloads, this difference in bandwidth consumption is a part of the requirements for the application, not a design decision that can be optimized. ## Measure Performance and Set Goals The most important tools for creating high performance applications that use AMPS are clear goals and accurate measurement. Without accurate measurement, it's impossible to know whether a particular change has improved performance or not. Without clear goals, it's difficult to know whether a given result is sufficient, or whether you need to continue improving performance. 60East recommends that your measurements include baseline metrics for the part of your message processing that does not involve AMPS. As an example, imagine your task is to reduce the amount of time that elapses between when an order is sent and when the processed response is received from 100ms in total to 85ms in total. To achieve this reduction, you might first measure the processing that your application performs on the order. If that processing consumes 65ms, the most effective optimization may be to improve the order processing. On the other hand, if processing an order consumes 15ms, then optimizing message delivery or network utilization may be the most effective way to meet your goals. When measuring performance, simulate your production environment as closely as possible. For example, AMPS is highly parallelized, so sending a pattern of subscriptions and publishes from a single test client that would normally come from 20 clients will produce a very different performance profile. Likewise, AMPS can typically perform at rates that fill the available bandwidth. Performance measured on a 1GbE connection may be very different than performance measured over a 10GbE connection. Consider the characteristics of your data, and the number of messages you expect to store and process. A 1GB data set consisting of 1 million records will perform differently than a 1GB data set consisting of 10 million records, or a 1GB data set consisting of 100 records. When collecting information about performance, 60East recommends enabling persistence for the Statistics Database (`stats.db`), so you can easily collect historical data on both AMPS and the operating system. For example, a dip in performance correlated with high CPU and memory usage at the same time each day may be correlated with other activity on the system (such as cron jobs or close of business processing). In a situation like that, where the performance reduction is based on factors external to the AMPS application, the overall system metrics captured in `stats.db` can help you re-create the external state and understand the state of the system as a whole. AMPS collects the statistics in memory by default, and persisting that data into a database does not typically have a measurable effect on performance itself, but makes measuring and tuning performance much easier. For performance testing, 60East recommends using dedicated hardware for AMPS to eliminate the effects of other processes. If dedicated hardware is not available and other processes are consuming resources, 60East recommends disabling AMPS NUMA tuning to ensure that AMPS threads do not unnecessarily compete with other processes during performance tuning. ## Use HAClient and Heartbeating Where Appropriate Not every application that uses AMPS requires high availability and the ability to automatically fail over if connectivity is lost or an instance of AMPS is offline. For applications that do need automatic reconnection, 60East strongly recommends using the `HAClient` and setting heartbeating for the client to effectively detect disconnection. When using the `HAClient` and heartbeating, there are two important guidelines to follow: - Do not replace the disconnect handler on the `HAClient`. The disconnect handler is responsible for reconnection, resubscription, and so on. If you need to detect disconnection, use a connection state listener. - Set the interval for heartbeating to approximately one-half the time that the application can tolerate interruption in message flow. Notice that it's not possible for the `HAClient` to tell the difference between an interruption in message flow caused by a server going offline and interruptions caused by an increase in latency due to network saturation or so on, so the interval should be somewhat larger than the highest expected latency between AMPS and the application. Last, but not least, if the application uses asynchronous message handling, the interval should also be set to a value larger than the maximum amount of time expected for the message handler to process a single message. ## Simplify Message Format and Contents AMPS supports a wide range of message types, and is capable of filtering and processing large and complex messages. For many applications, the simplicity of being able to use messages that contain the full information is the most important consideration. For other applications, however, achieving the minimum possible latency and the maximum possible network utilization is important enough to warrant choosing a simplified message format. To simplify message contents, carefully consider the information that downstream processors require. If a downstream process will not use information in the message, there is no need to send the information. For example, consider an application that provides orders from a UI. In such an application, the object that represents the order often contains information relevant to the local state of the application that is not relevant to a downstream system. Rather than simply serializing the full object, your application may perform better if you serialize only the fields that a downstream system will take action on. To simplify message format, choose the simplest format that can convey the information that your application needs. The general principle is that the simpler the message format is, the more quickly AMPS and client libraries can parse messages of that type. Likewise, the more complicated the structure of each message is, the more work is required to parse the message. For the highest levels of performance, 60East recommends keeping the message structure simple and preferring message formats such as NVFIX, BFlat, or flattened JSON (structured as key/value pairs) as compared with more complicated formats such as XML or BSON. ## Measure Serialization and Deserialization When creating baseline performance numbers, measure serialization and deserialization performance independent of the AMPS server or client libraries. This can help you to: - Understand the baseline performance of creating and processing message data under ideal conditions (that is, where there is no application processing, networking, routing, etc. involved). - Easily compare the application-side performance of different message formats or different message layouts within a single format. When testing this performance, it is helpful to use data similar to the data that the application will actually process during a business day, at the volumes the application would typically process. This will help you understand the performance of serialization and deserialization for this specific application. For example, a library for working with a given message format might be less efficient when processing messages with a large number of string fields in a deeply-nested structure, but your application might exchange only numeric data in a relatively flat structure. Likewise, the library for a given format could be efficient for processing a small number of fields, but have lower performance for a message type with hundreds of fields. As with all performance testing, the more closely the test environment matches the actual data and volumes of a production environment, the more helpful those measurements will be for understanding system performance. ## Use Content Filtering Where Possible AMPS content filtering helps your application perform better by ensuring that your application only receives the messages that it needs. Wherever possible, we recommend using content filtering to precisely specify which messages your application needs. In particular, if at any point your application is receiving a message, parsing the message, and then determining whether to act on the message or not, 60East recommends using content filters to ensure that your application only receives messages that it needs to act on. ## Use Asynchronous Message Processing The synchronous message processing interface is straightforward, and presents a convenient interface for getting started with AMPS. However, the `MessageStream` used by the synchronous interface makes a full copy of each message and provides it from the background reader thread to the thread that consumes the message. This memory overhead and synchronization between the reader thread and consumer thread happens regardless of whether the application needs all of the header fields in the message or even processes the message. The `MessageStream` also does not take into account the speed at which your program is consuming messages, and will read messages into memory as fast as the network and processor allow. If your application cannot consume messages at wire speed, this can lead to increasing memory consumption as the application falls further behind the `MessageStream`. Most applications see improved performance by using a `MessageHandler`. With this approach, the `MessageHandler` does minimal work. If more extensive processing is needed, the `MessageHandler` dispatches the work to another thread: but it does this only when the work is necessary, and it only saves the part of the message needed to accomplish the work. ## Use Hash Indexes Where Possible for SOW Queries When querying a SOW, hash indexes on SOW topics are supported for exact matching on string data as described in the *AMPS User Guide*. A hash index can perform many times faster than a parallel query. If the query pattern for your application can take advantage of hash indexes, 60East recommends creating those hash indexes on your SOW topics. More recent versions of AMPS can use hash indexes for a wider variety of filters. When planning your queries, review the SOW queries section of the *AMPS User Guide* for the version you are using for guidelines on the optimizations available in that version. ## Use a Failed Write Handler and Exception Listener In many cases, particularly during the early stages of development, performance problems can point to defects in the application. Even after the application is tuned, monitoring for failure is important to keep applications running smoothly. 60East recommends always installing a failed write handler if your application is publishing messages. This will help you to quickly identify cases where AMPS is rejecting publishes due to entitlement failures, message type mismatches, or other similar problems. 60East recommends always installing an exception listener if your application is using asynchronous message processing. This will help you to identify and correct any problems with your message handler. An exception listener should typically log the message received and return. If recovery is needed, the listener should set a flag for another thread to process rather than attempting to recover on the thread that calls the exception listener. ## Reduce Bandwidth Requirements In many applications that use AMPS, network bandwidth is the single most important factor in overall performance. Your application can use bandwidth most efficiently by reducing message size. For example, rather than serializing an entire object, you might serialize only the fields that the remote process needs to act on, as mentioned above. Likewise, rather than sending one message that contains a collected set of information that processors will need to extract, consider sending a message in the units that processors will work with. This can reduce bandwidth to processors substantially. For example, rather than sending a single message with all of the activity for a single customer over a given period of time (such as a trading day), consider breaking out the record into the individual transactions for the customer. ### Tune Batch Size for SOW Queries As described in the section on [SOW Batch Size](/docs/amps-user-guide/sow-queries/batching-query-results), tuning the batch size for SOW queries can improve overall performance by improving network utilization. In addition, because the AMPS header is only parsed once per batch, a larger batch size can dramatically improve processing performance for smaller messages. The AMPS clients default to a batch size of `10`. This provides generally good performance for most transactional messages (such as order records or inventory records). For large messages, particularly messages greater than a megabyte in size, a batch size of `1` may reduce memory pressure in the client and improve performance. With smaller messages (for example, message sizes of a few hundred bytes), 60East recommends measuring performance with larger batch sizes such as `50` or `100`. For large messages, reducing the batch size may improve overall performance by requiring less memory consumption on the AMPS server. ### Conflate Fast-Changing Information If your data source publishes information faster than your clients need to consume it, consider using a conflated topic. For example, in a system that presents a user interface and displays fast-moving data, it is common for the data to change at a rate faster than the user interface can format and render the data. In this case, a conflated topic can both reduce bandwidth and simplify processing in the user interface. ### Minimize Bandwidth for Updates If your application uses a SOW and processes frequent updates, consider using delta publish and delta subscribe to reduce the size of the messages transmitted. These features are designed to minimize bandwidth while still providing full-fidelity data streams. ### Conflate Queue Acknowledgments The AMPS clients include the ability to conflate acknowledgments back to AMPS as queue messages are processed. Using these features, with an appropriate `max_backlog`, can reduce the amount of network traffic required for acknowledgments. ### Use a Transaction Log When Monitoring Publish Failures When a topic is not covered by a transaction log, AMPS returns acknowledgment messages for every publish that requests one. This ensures that each message is acknowledged, even when AMPS has no persistent record of the messages in the topic. However, acknowledging each message requires more network traffic for each publish message. When a topic is covered by a transaction log, AMPS conflates persisted acknowledgments. Conflation is possible in this case because AMPS has a full record of the messages and does not have to store additional state to conflate the acknowledgments. With conflated acknowledgments, AMPS will send a success acknowledgment periodically that covers all messages up to that point. If a message fails, AMPS immediately sends the conflated success acknowledgment for all previous messages and the failure acknowledgment for the failed message. ### Combine Conflation and Deltas In many cases, using an approach that combines delta publishes to a SOW with delta subscriptions to a conflated topic can dramatically reduce bandwidth to the application with no loss of information. ## Limit Unnecessary Copies One of the most effective ways to increase performance is to limit the amount of data copied within your application. For example, if your message handler submits work to a set of processors that only use the `Data` and `Bookmark` from a `Message`, create a data structure that holds only those fields and copy that information into instances of that data structure rather than copying the entire `Message`. While this approach requires a few extra lines of code, the performance benefits can be substantial. When publishing messages to AMPS, avoid unnecessary copies of the data. For example, if you have the data in a byte array, use the `publish` methods that use a byte array rather than converting the data to a string unnecessarily. Likewise, if you have the data in the form of a string, avoid converting it to a byte array where possible. ## Manage Publish Stores When using a publish store, the Client holds messages until they are acknowledged as persisted by AMPS, as determined by the replication configuration for the AMPS instance. In the event that an instance with `sync` replication goes offline, the publish store for the Client will grow, since the messages are not being fully persisted. To avoid this problem, 60East recommends that an instance that uses `sync` replication always configure Actions to automatically downgrade the replication link if the remote instance goes offline for a period of time, and upgrade the link when the remote instance comes back online. Further, 60East recommends that, where possible, a publisher is provisioned with enough storage to hold its complete publish stream for the amount of time that a destination may be offline or unavailable without downgrading from `sync` replication to `async` replication. For example, if the server considers a downstream system to be unreachable if it has not acknowledged a replicated message in 60 seconds, and the server checks this threshold every 10 seconds, then a publisher should plan that, at any time, the publisher may need to retain approximately 70 seconds worth of published messages. This is calculated as the 60 seconds threshold that the server has established for a destination to run behind, plus the 10 second interval at which the server checks whether the destination is within the threshold. Also notice that, with a configuration like this, a downstream replication destination could run as much as 59 seconds behind indefinitely. A publisher should be provisioned to be able to run effectively in a "worst case" (or nearly "worst case") scenario for an extended period of time. See the *High Availability and Replication* chapter in the *AMPS User Guide* for more information on replication, sync and async acknowledgment modes, and the Actions used to manage replication. ## Use the Server Logs to Help Troubleshoot When troubleshooting problems with an application that uses AMPS, the server-side logs often provide the most helpful information. For example, `trace` level logging shows the data that is flowing through AMPS. Log messages at `info` level show events as incoming connections, commands from clients, and so on. When questions arise about how the server and application interact, the server logs often contain the information. 60East recommends that an AMPS instance used for development and testing log at `trace` level, and that a server used for production log at `info` level, with the ability to log at `trace` level when necessary for investigating any problems that may arise. When a command does not have the expected result, or an application reports an error, the fastest way to understand the problem is often to review the `trace` level logging for the instance. See the *AMPS User Guide* for details on configuring logging and common patterns for searching for information in AMPS logs. ## Work with 60East as Necessary 60East offers performance advice adapted for your specific usage through your support agreement. Once you've set your performance goals, worked through the general best practices and applied the practices that make sense for your application, 60East can help with detailed performance tuning, including recommendations that are specific to your use case and performance needs. --- # Using Queues AMPS message queues provide a high-performance way of distributing messages across a set of workers. The *AMPS User Guide* describes AMPS [Queues](/docs/amps-user-guide/queues) in detail, including the features of AMPS referred to in this chapter. This chapter does not describe message queues in detail, but instead explains how to use the AMPS C# client with message queues. To publish messages to a message queue, publishers simply publish to any topic that is collected by the queue. There is no difference between publishing to a queue and publishing to any other topic, and a publisher does not need to be aware that the topic will be collected into a queue. Subscribers must be aware that they are subscribing to a queue, and acknowledge messages from the queue when the message is processed. --- # Regular Expression Subscriptions Regular Expression (Regex) subscriptions allow a regular expression to be supplied in the place of a topic name. When you supply a regular expression, it is as if a subscription is made to every topic that matches your expression, including topics that do not yet exist at the time of creating the subscription. To use a regular expression, simply supply the regular expression in place of the topic name in the `subscribe()` call. For example: ```csharp showLineNumbers Client c = ...; foreach (Message msg in c.subscribe("client.*")) { Console.WriteLine("{0}:{1}", msg.Topic, msg.Data); } ``` In this example, messages on topics `client` and `client1` would match the regular expression, and those messages would all be received by our subscription. As in the example, you can use the `Topic` property to determine the actual topic of the message sent to the lambda function. --- # Returning a Message to the Queue A subscriber can also explicitly release a message back to the queue. AMPS returns the message to the queue, and redelivers the message just as though the lease had expired. To do this, the subscriber sends a `sow_delete` command with the bookmark of the message to release and the `cancel` option. When using automatic acknowledgments and the asynchronous API, AMPS will cancel a message if an exception is thrown from the message handler. To return a message to the queue, you can build a `sow_delete` acknowledgment using the `Command` class, or pass an option to the `ack()` method on the message. | Option | Result | | ---------- | ----------------------------------------------- | | `cancel` | Returns the message to the queue. | | `expire` | Immediately expires the message from the queue. | For example, to return a message to a queue, call `ack()` on the message and pass the `cancel` option. ```csharp message.ack("cancel"); ``` --- # Setting Batch Size The AMPS clients include a batch size parameter that specifies how many messages the AMPS server will return to the client in a single batch when returning the results of a SOW query. The 60East clients set a batch size of 10 by default. This batch size works well for common message sizes and network configurations. Adjusting the batch size may produce better network utilization and produce better performance overall for the application. The larger the batch size, the more messages AMPS will send to the network layer at a time. This can result in fewer packets being sent, and therefore less overhead in the network layer. The effect on performance is generally most noticeable for small messages, where setting a larger batch size will allow several messages to fit into a single packet. For larger messages, a batch size may still improve performance, but the improvement is less noticeable. In general, 60East recommends setting a batch size that is large enough to produce few partially-filled packets. Bear in mind that AMPS holds the messages in memory while batching them, and the client must also hold the messages in memory while receiving the messages. Using batch sizes that require large amounts of memory for these operations can reduce overall application performance, even if network utilization is good. For smaller message sizes, 60East recommends using the default batch size, and experimenting with tuning the batch size if performance improvements are necessary. For relatively large messages (especially messages with sizes over 1MB), 60East recommends explicitly setting a batch size of 1 as an initial value, and increasing the batch size only if performance testing with a larger batch size shows improved network utilization or faster overall performance. --- # SOW and Subscribe Imagine an application that displays real-time information about the position and status of a fleet of delivery vans. When the application starts, it should display the current location of each of the vans, along with their current status. As vans move around the city and post other status updates, the application should keep its display up to date. Vans upload information to the system by posting messages to the `van_location` topic, configured with a key of `van_id` on the AMPS server. In this application, it is important to not only stay up-to-date on the latest information about each van, but to ensure all of the active vans are displayed as soon as the application starts. Combining a SOW with a subscription to the topic is exactly what is needed, and that is accomplished by the AMPS `sow_and_subscribe` command. Now we will look at an example: ```csharp showLineNumbers private void UpdateVanPosition(Message message) { switch (message.Command) { case Message.Commands.SOW: case Message.Commands.Publish: // For each of these messages we call AddOrUpdateVan(), that presumably // adds the van to our application's display. As vans send updates to the // AMPS server, those are also received by the client because of the // subscription placed by sowAndSubscribe(). Our application does not need // to distinguish between updates and the original set of vans we found via // the SOW query, so we use addOrUpdateVan() to display the new position of // vans as well. AddOrUpdateVan(message); break; case Message.Commands.OOF: RemoveVan(message); break; } } public void SubscribeToVanLocation(Client client) { Command command = new Command("sow_and_subscribe") .setTopic("van_location") .setFilter("/status = 'ACTIVE'") .setBatchSize(100) .setOptions("oof"); // Notice here that we specified an option of "oof". Including this option causes us // to receive Out-of-Focus (OOF) messages for the topic. // OOF messages are sent when an entry that was sent to us in the past // no longer matches our query. This happens when an entry is removed from the SOW // cache via a sowDelete() operation, when the entry expires (as specified by the // expiration time on the message or by the configuration of that topic on the AMPS // server), or when the entry no longer matches the content filter specified. In // our case, if a van's status changes to something other than ACTIVE, it no longer // matches the content filter, and becomes out of focus. When this occurs, a // Message is sent with Command set to oof. We use OOF messages to remove vans from // the display as they become inactive, expire, or are deleted. foreach (Message msg in client.execute(command)) { updateVanPosition(message); } } public void addOrUpdateVan(message) { // Use information in the message to add the van or update // the van position. } public void removeVan(message) { // Use information in the message to remove information on // the van position. } ``` Now we will look at an example that uses the asynchronous form of execute to place a `sow_and_subscribe` command: ```csharp showLineNumbers private void UpdateVanPosition(Message message) { switch (message.Command) { case Message.Commands.SOW: case Message.Commands.Publish: AddOrUpdateVan(message); break; case Message.Commands.OOF: RemoveVan(message); break; } } public void SubscribeToVanLocation(Client client) { Command command = new Command("sow_and_subscribe") .setTopic("van_location") .setFilter("/status = 'ACTIVE'") .setBatchSize(100) .setOptions("oof"); client.executeAsync(command, msg => UpdateVanPosition(msg)); } ``` --- # State of the World (SOW) AMPS State of the World (SOW) allows you to automatically keep and query the latest information about a topic on the AMPS server, without building a separate database. Using SOW lets you build impressively high-performance applications that provide rich experiences to users. The AMPS C# client lets you query SOW topics and subscribe to changes with ease. ## Performing SOW Queries To begin, we will look at a simple example of issuing a SOW query. ```csharp showLineNumbers public void ExecuteSOWQuery(Client client) { foreach (Message m in client.sow("messages-sow", "/id > 20")) { if (m.Command == Message.Commands.BeginGroup) { System.Console.WriteLine("--- Begin SOW Results ---"); } if (m.Command == Message.Commands.EndGroup) { System.Console.WriteLine("--- End SOW Results ---"); } if (m.Command == Message.Commands.SOW) { System.Console.WriteLine(m.Data); } } } ``` In the example above, the `ExecuteSOWQuery()` function invokes `Client.sow()` to initiate a SOW query on the `messages-sow` topic, for all entries that have an `id` greater than 20. As the query executes, the body of the loop is invoked for each matching entry in the topic. Messages containing the data of matching entries have a `Command` of value `sow`; as those arrive, we write them to the console. AMPS sends a `group_begin` message at the beginning of the results and a `group_end` message at the end of the results. We use those messages to delimit the results of the query. As with subscribe, the sow command also provides an asynchronous version, as well as versions that accept a `Command`. For example, the listing below shows an asynchronous SOW command that specifies the *batch size*, or the maximum number of records that AMPS will return at a time. ```csharp showLineNumbers private void HandleSOW(Message message) { if (message.Command == Message.Commands.SOW) { Console.WriteLine(message.Data); } } public void ExecuteSOWQuery(Client client) { Command command = new Command(Message.Commands.SOW) .setTopic("messages-sow") .setFilter("/id > 20") .setBatchSize(100); client.executeAsync(command, message => HandleSOW(message)); } ``` In the example above, the `ExecuteSOWQuery()` function invokes `Client.executeAsync()` to initiate a SOW query on the `messages-sow` topic, for all entries that have an `id` greater than 20. The SOW query is requested with a batch size of 100, meaning that AMPS will attempt to send 100 messages at a time as results are returned. As the query executes, the `HandleSOW()` method is invoked for each matching entry in the topic. Messages containing the data of matching entries have a `Command` of value `sow`; as those arrive, we write them to the console. --- # Subscriptions Messages published to a topic on an AMPS server are available to other clients via a subscription. Before messages can be received, a client must subscribe to one or more topics on the AMPS server so that the server will begin sending messages to the client. The server will continue sending messages to the client until the client unsubscribes, or until the client disconnects. With content filtering, the AMPS server will limit the messages sent to only those messages that match a client-supplied filter. In this chapter, you will learn how to subscribe, unsubscribe, and supply filters for messages using the AMPS C# client. ## Subscribing to a Topic Subscribe to an AMPS topic by calling `Client.subscribe()`. Below is a short example (error handling and connection details are omitted for brevity): ```csharp showLineNumbers class MyApp { public static void Main() { // Here, we create a Client. We protect the Client in a using // block so that the connection and subscriptions are properly // cleaned up when dispose() is called. using(Client client = new Client("subscribe")) { client.connect("tcp://127.0.0.1/9007/amps"); client.logon(); // Here we subscribe to the topic messages. We do not provide // a filter, so the subscription receives all of the messages // published to the topic, regardless of content. The foreach // loop iterates over the messages returned by the MessageStream. // When we no longer need to subscribe, we can break out of the // loop. When the MessageStream is disposed, the client sends an // unsubscribe command to AMPS and stops receiving messages. foreach(Message m in client.subscribe("messages")) { // Within the loop, we process the message. In this case, // we simply print the contents of the message. System.Console.Writeline(m.getData()); } } } } ``` AMPS creates a background thread that receives the messages and copies them into the `MessageStream` that you iterate over. This means that the client application as a whole can continue to receive messages while you are doing processing work. The simple method described above is provided for convenience. The AMPS C# client provides convenience methods for the most common forms of the commands. The client also provides an interface that gives you precise control over the command. Using that interface, the example above becomes: ```csharp showLineNumbers class MyApp { public static void Main() { // Here, we create a Client. We protect the Client in a using // block so that the connection and subscriptions are properly // cleaned up when dispose() is called. using(Client client = new Client("subscribe")) { client.connect("tcp://127.0.0.1/9007/amps"); client.logon(); // We create a Command object to subscribe to the messages topic. Command command = new Command("subscribe").setTopic("messages"); // Here we execute the command and subscribe to the topic // messages. This works exactly the same way as the command // in the example above. We do not provide a filter, so the // subscription receives all of the messages published to the // topic, regardless of content. // The foreach loop iterates over the messages returned by // the MessageStream. When we no longer need to subscribe, we // can break out of the loop. When the MessageStream is disposed, // the client sends an unsubscribe command to AMPS and stops // receiving messages. foreach(Message m in client.execute(command)) { // Within the loop, we process the message. In this case, we // simply print the contents of the message. System.Console.WriteLine(m.getData()); } } } } ``` The `Command` interface allows you to precisely customize the commands you send to AMPS. For flexibility and ease of maintenance, 60East recommends using the `Command` interface (rather than a named method) for any command that will receive messages from AMPS. For publishing messages, there can be a slight performance advantage to using the named commands where possible. --- # Synchronous Message Processing As mentioned [earlier](subscriptions.md), one way for an application to receive messages is to have the AMPS C# client return a `MessageStream` object that can be used to iterate over the results of the command. The `MessageStream` object makes copies of the incoming messages. When there is no message available, the `MessageStream` will block. A `MessageStream` will only remain active while the client that produced it is connected. If the client disconnects, the `MessageStream` will continue to provide any messages that have not yet been consumed, then throw an exception. The advantages of using a `MessageStream` that it provides a simple processing model, that receiving messages from a `MessageStream` does not block the client receive thread (see [Understanding Threading](understanding-threading-section.md) ) and that a copy of the message is automatically made for the application. In return for these advantages, a `MessageStream` has higher overhead than [Asynchronous Message Processing](async-message-processing.md), it will not be resumed if the client disconnects, and, by default, it will use as much memory as necessary to hold messages coming from the AMPS server. --- # Understanding Threading The first time a command causes an instance of the `Client` or `HAClient` to connect to AMPS (typically, the `logon()` command), the client creates a thread that runs in the background. This background thread is responsible for processing incoming messages from AMPS, which includes both messages that contain data and acknowledgments from the server. When you call a command on the AMPS client, the command typically waits for an acknowledgment from the server and then returns. (The exception to this is `publish`. For performance, the `publish` command does not wait for an acknowledgment from the server before returning.) In the simple case, using synchronous message processing, the client provides an internal handler function that populates the `MessageStream`. The client receive thread calls the internal handler function, which makes a deep copy of the incoming message and adds it to the `MessageStream`. The `MessageStream` is used on the calling thread, so operations on the `MessageStream` do not block the client receive thread. When using asynchronous message processing, AMPS calls the handler function from the client receive thread. Message handlers provided for *asynchronous* message processing must be aware of the following considerations: - The client creates one client receive thread at a time, and the lifetime of the thread lasts for the lifetime of the connection to the AMPS server. A message handler that is only provided to a single client will only be called from a single thread at a time. If your message handler will be used by multiple clients, then multiple threads will call your message handler. In this case, you should take care to protect any state that will be shared between threads. Notice that if the client connection fails (or is closed), and the client reconnects, the client will create a different thread for the new connection. - For maximum performance, do as little work in the message handler as possible. For example, if you use the contents of the message to update an external database, a message handler that adds the relevant data to an update queue, that is processed by a different thread, will typically perform better than a message handler that does this update during the message handling. - While your message handler is running, the thread that calls your message handler is no longer receiving messages. This makes it easier to write a message handler because you know that no other messages are arriving from the same subscription. However, this also means that you cannot use the same client that called the message handler to send commands to AMPS. Acknowledgments from AMPS cannot be processed and your application will deadlock waiting for the acknowledgment. Instead, enqueue the command in a work queue to be processed by a separate thread or use a different client object to submit the commands. - The AMPS client resets and reuses the `Message` provided to this function between calls. This improves performance in the client, but means that if your handler function needs to preserve information contained within the message, you must copy the information (either by making a copy of the entire message or copying the required fields) rather than just saving the message object. Otherwise, the AMPS client cannot guarantee the state of the object or the contents of the object when your program goes to use it. Likewise, a message handler should not modify the `Message` -- this will result in modifying the message provided to other handlers (including handlers internal to the AMPS client). --- # Unexpected Messages The AMPS C# client handles most incoming messages and takes appropriate action. Some messages are unexpected or occur only in very rare circumstances. The AMPS C# client provides a way for clients to process these messages. Rather than providing handlers for all of these unusual events, AMPS provides a single handler function for messages that can't be handled during normal processing. Your application registers this handler by setting the `lastChanceMessageHandler` for the client. This handler is called when the client receives a message that can't be processed by any other handler. This is a rare event, and typically indicates an unexpected condition. For example, if a client publishes a message that AMPS cannot parse, AMPS returns a failure acknowledgment. This is an unexpected event, so AMPS does not include an explicit handler for this event, and failure acknowledgments are received in the method registered as the `lastChanceMessageHandler`. Your application is responsible for taking any corrective action needed. For example, if a message publication fails, your application can decide to republish the message, publish a compensating message, log the error, stop publication altogether, or any other action that is appropriate. --- # Unhandled Exceptions In the AMPS C# client, exceptions can occur that are not thrown to the user. For example, when an exception occurs in the process of reading subscription data from the AMPS server, the exception occurs on a thread inside of AMPS. Consider the following example: ```csharp showLineNumbers public class MyApp { ... public static void WaitToBePoked(Client client) { client.subscribe( x=>Console.WriteLine("Hey! {0} poked you!", x.UserId), "pokes", string.Format("/Pokee LIKE '{0}-.*'", System.Environment.UserName), 5000 ); Console.ReadKey(); } } ``` In this example, we set up a simple subscription to wait for messages on the `pokes` topic, whose `Pokee` tag begins with our username. When messages arrive, we print a message out to the console, but otherwise our application waits for a key to be pressed. Inside of the AMPS client, the client creates a new thread of execution that reads data from the server, and invokes message handlers and disconnect handlers when those events occur. When exceptions occur inside this thread, however, there is no caller for them to be thrown to, and by default they are ignored. In applications where it is important to deal with every issue that occurs in using AMPS, you can set an `ExceptionListener` via `Client.setExceptionListener()` that receives these otherwise unhandled exceptions. Making the modifications shown in the example below, to our previous example, will allow those exceptions to be caught and handled. In this case we are simply printing those caught exceptions out to the console. If your application will attempt to recover from an exception thrown on the background processing thread, your application should set a flag and attempt recovery on a *different* thread than the thread that called the exception listener. :::tip At the point that the AMPS client calls the exception listener, it has handled the exception. Your exception listener must not rethrow the exception (or wrap the exception and throw a different exception type). ::: ```csharp showLineNumbers public class MyApp { ... public static void WaitToBePoked(Client client) { client.setExceptionListener( ex=>Console.Error.WriteLine(ex)); client.subscribe( x=>Console.WriteLine("Hey! {0} poked you!", x.UserId), "pokes", string.Format("/Pokee LIKE '{0}-.*'", System.Environment.UserName), 5000); Console.ReadKey(); } } ``` In this example we have added a call to `client.setExceptionListener()`, registering a simple function that writes the text of the exception out to the console. Even though our application waits for a user to press a key, messages to the console will still be produced, both as incoming `poke` messages arrive, and as issues arise inside of AMPS. --- # Ending Subscriptions The AMPS server continues a subscription until the client explicitly ends the subscription (that is, *unsubscribes*) or the connection to the client is closed. With the synchronous message processing interface, AMPS automatically unsubscribes when the `dispose()` method for the `MessageStream` is called. You can also explicitly call the `close()` method on the `MessageStream` to remove the subscription. In the asynchronous message processing interface, when a subscription is successfully made, messages will begin flowing to the message handler, and the `subscribe()` or `executeAsync()` call will return the identifier for the newly created subscription. A `Client` can have any number of subscriptions, and this identifier is how AMPS designates messages intended for this particular subscription. To unsubscribe, we simply call `unsubscribe` with the subscription identifier, as shown below: ```csharp showLineNumbers Client c = ...; Command subscribe_command = new Command("subscribe").setTopic("messages"); CommandId subscriptionId = c.executeAsync(subscribe_command, (message) => Console.WriteLine(message)); ... client.unsubscribe(subscriptionId); ``` In this example, we use the `executeAsync()` method to create a subscription to the `messages` topic. When our application is done listening to this topic, it unsubscribes by executing an `unsubscribe` command that contains the `subscriptionId` returned when the subscription was created. After the subscription is removed, no more messages will flow into our `(message)` lambda function. When an application calls `unsubscribe()`, the client sends an explicit `unsubscribe` command to AMPS. The AMPS server removes that subscription from the set of subscriptions for the client, and stops sending messages for that subscription. On the client side, the client unregisters the subscription so that the `MessageStream` or `MessageHandler` for that subscription will no longer receive messages for that subscription. Notice that calling `unsubscribe` does not destroy messages that the server has already sent to the client. If there are messages on the way to the client for this subscription, the AMPS client must consume those messages. If a `LastChanceMessageHandler` is registered, the handler will receive the messages. Otherwise, they will be discarded since no `MessageHandler` matches the subscription ID on the message. --- # Utility Classes The AMPS C# client includes a set of utilities and helper classes to make working with AMPS easier. ## Composite Message Types The client provides a pair of classes for creating and parsing composite message types: - `CompositeMessageBuilder` allows you to assemble the parts of a composite message and then serialize them in a format suitable for AMPS. - `CompositeMessageParser` extracts the individual parts of a composite message type. For more information regarding composite message types, refer to the [*Message Types*](/docs/amps-user-guide/message-types) chapter in the *AMPS User Guide*. ### Building Composite Messages To build a composite message, create an instance of `CompositeMessageBuilder`, and populate the parts. The `CompositeMessageBuilder` copies the parts provided, in order, to the underlying message. The builder simply writes to an internal buffer with the appropriate formatting, and does not allow you to update or change the individual parts of a message once they've been added to the builder. The snippet below shows how to build a composite message that includes a JSON part, constructed as a string, and a binary part consisting of the bytes from a `List`. ```csharp showLineNumbers StringBuilder sb = new StringBuilder(); sb.append("{\"data\":\"sample\"}"); List theData = new List(); // populate theData ... // Create a byte array from the data: this is // what the program will send. byte[] outBytes = null; using (MemoryStream stream = new MemoryStream()) { BinaryFormatter format = new BinaryFormatter(); format.Serialize(stream,theData); outBytes = stream.ToArray(); } // Create the payload for the composite message. CompositeMessageBuilder builder; // Construct the composite CompositeMessageBuilder builder = new CompositeMessageBuilder(); builder.append(sb.ToString()); builder.append(outBytes, 0, outBytes.Length); // send the message Field outMessage = new Field(); builder.setField(outMessage); topic = "messages"; byte[] topicBytes = System.Text.Encoding.UTF8.GetBytes(topic.ToCharArray()); client.publish(topicBytes, 0, topicBytes.Length, outMessage.buffer, 0, outMessage.length); ``` ### Parsing Composite Messages To parse a composite message, create an instance of `CompositeMessageParser`, then use the `parse()` method to parse the message provided by the AMPS client. The `CompositeMessageParser` gives you access to each part of the message as a sequence of bytes. For example, the following snippet parses and prints messages that contain a JSON part and a binary part that contains an array of doubles. ```csharp showLineNumbers foreach(Message message in client.subscribe("messages")) { int parts = parser.parse(message); string json = parser.getString(0); Field binary = new Field(); parser.getField(1, binary); List theData = new List(); using (MemoryStream stream = new MemoryStream()) { BinaryFormatter format = new BinaryFormatter(); stream.Write(binary.buffer, binary.position, binary.length); stream.Seek(0, SeekOrigin.Begin); theData = (List)format.Deserialize(stream); } System.Console.WriteLine("Received message with " + parts + " parts"); System.Console.WriteLine(json); foreach (double d in theData) { System.Console.Write(d + " "); } System.Console.WriteLine(); } ``` Notice that the receiving application is written with explicit knowledge of the structure and content of the composite message type. ## NVFIX Messages The client provides a pair of classes for creating and parsing NVFIX messages: - `NVFIXBuilder` allows you to assemble an NVFIX message and then serialize it in a format suitable for AMPS. - `NVFIXShredder` extracts the individual fields of an NVFIX message type. ### Building NVFIX Messages To build an NVFIX message, create an instance of `NVFIXBuilder`, then add the fields of the message using `append()`. `NVFIXBuilder` copies the fields provided, in order, to the underlying message. The builder simply writes to an internal buffer with the appropriate formatting, and does not allow you to update or change the individual fields of a message once they've been added to the builder. The snippet below shows how to build an NVFIX message and publish it to the AMPS client. ```csharp showLineNumbers // create a builder with 1024 bytes of initial capacity // using the default 0x01 delimiter NVFIXBuilder builder = new NVFIXBuilder(1024, (byte)1); // add fields to the builder builder.append("test", "data"); builder.append("more", "test data"); // create a byte array for the topic byte[] topic = Encoding.ASCII.GetBytes("messages"); // publish the message to the "messages" topic client.publish(topic, 0,topic.Length, builder.getBytes(),0, builder.getSize()); ``` ### Parsing NVFIX Messages To parse an NVFIX message, create an instance of `NVFIXShredder`, then use the `toMap()` method to parse the message provided by the AMPS client. The `NVFIXShredder` gives you access to the message data in a map. The snippet below shows how to parse and print an NVFIX message. ```csharp showLineNumbers try { // create a shredder -- since this just returns // the Map, we can reuse the same shredder. NVFIXShredder shredder = new NVFIXShredder((byte)1); // iterate through each message and write data to console foreach (Message msg in ms) { System.Console.Write("Got a message"); // shred the message to a dictionary Dictionary fields = shredder.toMap(msg.getData()); // iterate over the keys in the map and display the key and data foreach (KeyValuePair key in fields) { System.Console.Write(" " + key + " " + key.Value); } } } finally // close the message stream to release the subscription { ms.close(); } ``` ## FIX Messages The client provides a pair of classes for creating and parsing FIX messages: - `FIXBuilder` allows you to assemble a FIX message and then serialize it in a format suitable for AMPS. - `FIXShredder` extracts the individual fields of a FIX message. ### Building FIX Messages To build a FIX message, create an instance of `FIXBuilder`, then add the fields of the message using `append()`. `FIXBuilder` copies the fields provided, in order, to the underlying message. The builder simply writes to an internal buffer with the appropriate formatting, and does not allow you to update or change the individual fields of a message once they've been added to the builder. The snippet below shows how to build a FIX message and publish it to the AMPS client. ```csharp showLineNumbers // create a builder with 1024 bytes of initial capacity // using the default 0x01 delimiter FIXBuilder builder = new FIXBuilder(1024, (byte)1); // add fields to the builder builder.append(0, "data"); builder.append(1, "more data"); // create a string for the topic string topic = "messages"; // publish the message to the "messages" topic client.publish(topic.getBytes(), 0, topic.length(), builder.getBytes(), 0, builder.getSize()); ``` ### Parsing FIX Messages To parse a FIX message, create an instance of `FIXShredder`, then use the `toMap()` method to parse the message provided by the AMPS client. The `FIXShredder` gives you access to the message data in a map. The snippet below shows how to parse and print a FIX message. ```csharp showLineNumbers try { // create a shredder -- since this just returns // the Map, we can reuse the same shredder. FIXShredder shredder = new FIXShredder((byte)1); // iterate through each message and write data to console foreach (Message msg in ms) { System.Console.Write("Got a message"); // shred the message to a map Dictionary fields = shredder.toMap(msg.getData()); // iterate over the keys in the map and display the key and data foreach (KeyValuePair key in fields) { System.Console.Write(" " + key + " " + key.Value); } } } finally // close the message stream to release the subscription { ms.close(); } ``` --- # Welcome to the AMPS Java Client This guide provides information you need to get started with the AMPS Java client. It focuses specifically on the client and does not cover AMPS itself in detail. For an overview of AMPS and instructions on setting up your development environment, see the [Introduction to AMPS](/docs/intro-guide/intro) guide. :::tip This guide assumes that you have a development environment for Java and access to an AMPS server using the configuration provided with the Java samples (in the full source distribution of the client). ::: --- # Acknowledging Messages For each message delivered on a subscription, AMPS counts the message against the subscription backlog until the message is explicitly acknowledged. In addition, when a queue specifies `at-least-once` delivery, AMPS retains the message in the queue until the message expires or until the message has been explicitly acknowledged and removed from the queue. From the point of view of the AMPS server, acknowledgment is implemented as a `sow_delete` from the queue with the bookmarks of the messages to remove. The AMPS Java client provides several ways to make it easier for applications to create and send the appropriate `sow_delete`. ## Automatic Acknowledgment The AMPS client allows you to specify that messages should be automatically acknowledged. When this mode is on, AMPS acknowledges the message automatically in the following cases: - **Asynchronous Message Processing Interface** - The message handler returns without throwing an exception. - **Synchronous Message Processing Interface** - The application requests the next message from the `MessageStream`. AMPS batches acknowledgments created with this method, as described in the following section. To enable automatic acknowledgment, use the `setAutoAck()` method. ```java client.setAutoAck(true); // enable AutoAck ``` ## Message Convenience Method The AMPS Java client provides a convenience method, `ack()`, on delivered messages. When the application is finished with the message, the application simply calls `ack()` on the message. (This, in turn, provides the topic and bookmark to the `ack()` method of the client that received the message.) For messages that originated from a queue with `at-least-once` semantics, this adds the bookmark from the message to the batch of messages to acknowledge. For other messages, this method has no effect. ```java message.ack(); // Add this message to the next // acknowledgment batch. ``` --- # Acknowledgment Batching The AMPS Java client automatically batches acknowledgments when either of the convenience methods is used. Batching acknowledgments reduces the number of round-trips to AMPS, reducing network traffic and improving overall performance. AMPS sends the batch of acknowledgments when the number of acknowledgments exceeds a specified size, or when the amount of time since the last batch was sent exceeds a specified timeout. You can set the number of messages to batch and the maximum amount of time between batches, as shown below: ```java client.setAckBatchSize(10); // Send batch after 10 messages client.setAckTimeout(1000); // ... or 1 second ``` The AMPS Java client is aware of the subscription backlog for a subscription. When AMPS returns the acknowledgment for a subscription that contains queues, AMPS includes information on the subscription backlog for the subscription. If the batch size is larger than the subscription backlog, the AMPS Java client adjusts the requested batch size to match the subscription backlog. 60East recommends tuning the batch size to improve application performance. A value of 1/3 of the smallest `max_backlog` value is a good initial starting point for testing. 60East does not recommend setting the batch size larger than 1/2 of the `max_backlog` value without testing the setting to ensure that the application does not run out of messages to process while the acknowledgment is being sent to AMPS. --- # Advanced Topics ## C# Client Compatibility AMPS clients are available for many languages. Many AMPS customers write clients using a variety of languages, often both Java and C#. While Java and C# are fundamentally different languages, they share enough syntax that it can be straightforward to port code between the two, and especially from Java to C#. To aid in conversion from Java to C# (and from C# to Java), the C# client has a number of features that make it a little easier to bring code from Java to C#, as described below: - Java-style getters and setters - `getXXX()` / `setXXX()` - are provided corresponding to properties on the `Message` class. For example, given a variable message of type `Message`, the code: `string userName = message.UserName` and `string userName = message.getUserName()` are equivalent. - C# parameters that take lambda functions also take an interface type. The AMPS Java client defines interfaces such as `ClientMessageHandler`, which your application implements with a single `invoke()` method that is called when an event occurs. In C#, the AMPS client uses lambda functions and delegates to provide equivalent functionality. However, the same `*Handler` interfaces exist in C#, and instead of passing a lambda function, you may also implement these interfaces and pass in derived classes. While doing so would be inconvenient in C#, providing this symmetry allows your Java and C# to be ported interchangeably. - Java-style method name conventions are used throughout AMPS. In .NET, method names often begin with a capitalized first letter (e.g. `Connect()` instead of `connect()`). However, the C# AMPS client retains the capitalization style of the Java client where possible, making porting straightforward. ## Rebuilding the Client In the rare occasion that you need to make customizations to the AMPS Java Client, the full packages include the complete source code for the AMPS client. Rebuilding the client is straightforward using the provided `build.xml`, and the Apache `ant` program. The `build.xml` file is located in the `api/client/java/` directory of the root of your AMPS installation. For example, if AMPS is installed in `/opt/AMPS/`, then the Java client's `build.xml` would be located in the `/opt/AMPS/api/client/java/` directory. Assuming that a JDK version 1.6 or greater is installed and the Apache `ant` package has been installed, rebuilding the client is as simple as typing: ```bash ant ``` from the command line in the same directory where the `build.xml` file is located in the AMPS install. Upon successful completion, the libraries will be located in the `api/client/java/dist/lib` directory. ## Transport Filtering The AMPS Java client offers the ability to filter incoming and outgoing messages in the format they are sent and received on the network. This allows you to inspect or modify outgoing messages before they are sent to the network, and incoming messages as they arrive from the network. To create a transport filter, you implement the interface `TransportFilter`, construct an instance of the filter class, and install the filter with the `setTransportFilter` method on the transport. The AMPS Java client does not validate any changes made by the transport filter. This interface is most useful for application debugging or transport development. The client includes a sample filter, `TransportTraceFilter`, that simply writes incoming and outgoing buffers to an `OutputStream`. Notice that the transport filter function is called with the verbatim contents of data received from AMPS. This means that, for incoming data, the function may not be called precisely on message boundaries, and that the binary length encoding used by the client and server will be presented to the transport filter. ## Advanced Memory Management Techniques One of the most important features of the Java language and runtime is automatic memory management. One drawback of this approach is that garbage collection can introduce latency and provide an inconsistent response time for the application. The AMPS Java client is designed to allow you to build an application that requires no memory allocation to process and consume messages during steady-state processing (that is, once the client is connected and messages are flowing to the application). Use the following techniques to avoid memory allocation in your application. ### Publish from a Byte Buffer The `publish()` and `delta_publish()` methods provide overloads that accept a byte array, starting position in the array, and a length rather than a `String`. These methods directly copy the message bytes provided into the buffer that the transport uses to send the message, without creating an intermediate copy or temporary object. ### Avoid Publisher Contention or Use a Synchronized Wrapper The Java `ReentrantLock` used to protect publishes can result in memory allocation if the lock is contended. To avoid this allocation, either avoid publishing messages using a single instance of the `Client` from more than one thread at the same time, or wrap calls that publish messages in a method that is marked `synchronized` to avoid contention in the `Client`. ### Use Asynchronous Message Processing When a subscription uses asynchronous message processing, the `Message` that is provided to the `MessageHandler` is allocated once per client. The contents of the message are references to an underlying buffer that is reused for each message Notice that synchronous message processing (the `MessageStream` interface) allocates a full copy of *each message received*, and should not be used in environments where minimal memory allocation is a goal. ### Use the Raw Versions of Accessors When retrieving information from an instance of the `Message` class, use the versions of accessors labeled with `Raw`. These methods do not allocate memory: instead, they return the `Field` objects within the `Message`. These objects, in turn, are references to an underlying buffer. For `Message` objects provided to a `MessageHandler`, that underlying buffer is the buffer that the `Client` uses for reading from the socket. ### To Process on Another Thread, Use an Object Pool and Fixed-Size Buffer Many applications designed with low-latency in mind can perform the processing that they need within the `MessageHandler`. In many other cases, though, it is necessary for one or more worker threads to process requests. If your application needs to follow this pattern, three best practices apply: 1. Create an object that holds the exact data necessary, without requiring allocation. 2. Use an object pool to manage instances of that object and reuse those instances as necessary. 3. Use a fixed-size data structure (such as a ring buffer) to pass references to those objects between the receive thread and the processing thread. ### Consider Message Type and Parser Implementations When designing a system that will minimize allocation, it is also important to consider the message type and parser that the system will use. Some popular parsers are not designed with minimal allocation in mind, and those parsers are not a good match for systems that need to control allocation. The 60East BFlat parser implementation is an example of a parser that is designed to allow an application to consume messages with minimal memory allocation. The parser can operate on a buffer, and the consumer can optionally reuse a single `BFlatValue` object that is owned by the parser, no matter how many fields are produced or how many messages are parsed. ## Working with Messages and Byte Buffers The AMPS Java client allows you to publish messages that contain data from byte buffers. When working with byte buffers in AMPS, it's best to follow the simple conventions outlined below. AMPS provides overloaded `publish()` methods that allow you to publish messages from various formats. In this case, to publish a message using byte buffers, the message data must be provided as a `byte[]`. Along with the message data, the message topic, to which the message will be published to, must also be provided as a `byte[]`. The example below shows how to serialize an object into a byte buffer, then publish the message to AMPS using the `publish()` method. ```java showLineNumbers ... // create the topic string String topic = "messages"; // create the object data Employee emp = new Employee(); // create the field for the payload of the message Field data = new Field(); try { // create streams used to serialize the object ByteArrayOutputStream bStream = new ByteArrayOutputStream(); ObjectOutputStream oStream = new ObjectOutputStream(bStream); // serialize the object to the ObjectOutputStream oStream.writeObject(emp); // set the byte of the message to a field data.set(bStream.toByteArray()); } catch (Exception e) { e.printStackTrace(); } //publish to the "messages" topic using byte buffers client.publish(topic.getBytes(), 0, topic.getBytes().length, data.buffer, 0, data.buffer.length); ... ``` In addition to publishing messages, AMPS allows you to access the raw bytes in the data part of a message. The method `getDataRaw()` returns a `Field` that is composed of a byte buffer, position of the data in the buffer, and the length of the data. This data can then be deserialized and converted to an object for further use. The example below shows how to access the raw bytes of a message, and then shows how to deserialize the bytes of that message to a Java object. ```java showLineNumbers ... // access the message raw data Field data = message.getDataRaw(); // construct a ByteArrayInputStream object, // an ObjectInputStream object and a Field // to deserialize the data ByteArrayInputStream bais; ObjectInputStream ois; Field data; // construct the payload object Employee emp; ByteArrayInputStream try { // deserialize the data using ObjectInputStream to an object bais = new ByteArrayInputStream(data.buffer, data.position, data.buffer.length); ois = new ObjectInputStream(bais); emp = (Employee) ois.readObject(); } catch(Exception e) { ... } ``` ## Providing SSL Certificates to the AMPS Java Client The AMPS Java client includes support for Secure Sockets Layer (SSL) connections to AMPS. The client automatically attempts to make an SSL connection when the transport in the connection string is set to `tcps`, as described in the section on [Connection Strings for AMPS](./connection-strings.md) in this guide. There is no other change required to application code to use an SSL connection. However, to successfully make a SSL connection to AMPS, the Java runtime requires: - A key to provide for the client connection, as stored in the keystore. - A truststore to use to validate the server certificate used for the connection. Most often, this means that the server certificate for the connection must be signed by a trusted certificate, and that trusted certificate must be in either the default truststore, or a truststore provided to the JVM at runtime. This can also be accomplished by importing the server certificate into the default truststore or a truststore set by the application at run time. The parameters that you need to provide depend on how you have configured your certificates. If the server certificate is in the default truststore, you only need to provide a key for the client connection. Otherwise, you need to provide both a client key and a truststore that can validate the server certificate. There are three common methods of providing these certificates: 1. Setting global parameters on the command line 2. Setting global system properties 3. Creating an SSL context object for a specific connection These options are discussed in more detail below. In general, the first two methods are easiest for simple testing, and allow you to easily set the same options for all connections from the application. The third method is the most flexible, and lets an application use different options for different connections. Java Virtual Machine implementations may differ somewhat in the implementation details for `javax.net.ssl`. For full details on providing certificates to the Java runtime, see the documentation for your JVM implementation. For example, visit the link [https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html) for documentation on the Oracle JVM implementation of `javax.net`. Java Secure Socket Extension (JSSE) Reference Guide, and this section focuses on the Oracle JVM. ### Setting Global Keystore and Truststore on the Command Line For example, if you have a key installed into a keystore file and the server certificate is present in the default truststore or signed by a certificate in the default truststore, you might provide the client key to an application using the following command-line flags: ```bash -Djavax.net.ssl.keyStore= \ -Djavax.net.ssl.keyStorePassword= ``` If the server certificate is in a different truststore, you might provide the following set of flags to specify both the keystore and the truststore to use: ```bash -Djavax.net.ssl.trustStore= \ -Djavax.net.ssl.trustStorePassword= \ -Djavax.net.ssl.keyStore= \ -Djavax.net.ssl.keyStorePassword= ``` When you provide passwords on the command line, it is possible for other users on the system to see the password. This method is most often used during development and debugging. ### Setting Global Keystore and Truststore within the Application Within an application, you can use the `System.setProperty` method to set flags before creating a `Client` or `HAClient`. For example, if you have a key installed into a keystore file and the server certificate is present in the default truststore, you might set these properties when your application starts. ```bash System.setProperty("javax.net.ssl.keyStore", "path to keystore file"); System.setProperty("javax.net.ssl.keyStorePassword", "keystore password"); ``` This is equivalent to the first set of command line arguments above. As with the command line arguments above, you can use this method to set the truststore arguments as necessary. ```bash System.setProperty("javax.net.ssl.keyStore", "path to keystore file"); System.setProperty("javax.net.ssl.keyStorePassword", "keystore password"); System.setProperty("javax.net.ssl.trustStore", "path to truststore file"); System.setProperty("javax.net.ssl.trustStorePassword", "truststore password"); ``` #### Creating an SSL Context Setting options on the command line or with the `System.setProperty` method affects every connection made from the application. You can also create your own `SSLContext` for the application to use for the `tcps` transport. This method provides the most flexibility, and allows you to use different information for different connections. The snippet below has the same end result as the first command line above for the AMPS connection (that is, it sets a keystore and uses the default truststore). However, because this `SSLContext` will only be used by the AMPS client, you can use a keystore and certificates that are only used for AMPS connections, rather than for the application as a whole. This may be required if the application needs to use SSL to connect to a different external system, and the keys or truststore used for AMPS are different than the keys or truststore for the other system. ```java showLineNumbers // Snippet showing how to provide a keystore to the // TCPSTransport for use of the AMPS Java client. // Create a keystore. Assume that the keystore // is in JKS format. KeyStore ks = KeyStore.getInstance("JKS"); String password = ... ; // Obtain password for keystore FileInputStream fis = null; try { fis = new java.io.FileInputStream(); ks.load(fis, password.toCharArray()); } finally { if (fis !=null ) fis.close(); } // Get the key manager factory, using the default // algorithm. KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm()); // Initialize the factory with the keystore. kmf.init(ks, password.toCharArray()); // Get the SSL context SSLContext context = SSLContext.getInstance("TLS"); // Use the key manager just constructed, with defaults // for the trust manager and randomness source. context.init(kmf.getKeyManagers(), null, null); // Set the SSLContext for the TCPS transport // to the context just set up with the keystore. TCPSTransport.setDefaultSSLContext(context); ``` #### Troubleshooting SSL Connectivity to AMPS To troubleshoot SSL connectivity, run the application with the following flag: ```bash -Djavax.net.debug=all ``` With this flag set, the `javax.net` library will produce detailed information about the connection, including extensive information designed to help identify and resolve certificate problems. In particular, this flag will produce information on the certificates that the application is using, whether the application can successfully load those certificates, and whether the application can validate the certificate provided by the AMPS server. --- # Asynchronous Message Processing The AMPS Java client also supports an interface designed for asynchronous message processing. In this case, you add a message handler to the call to subscribe. The client object returns the command ID of the subscribe command once the server has acknowledged that the command has been processed. As messages arrive, the client calls your message handler directly on the background thread. This can be an advantage for some applications. For example, if your application is highly multithreaded and copies message data to a work queue processed by multiple threads, there may be a performance benefit to enqueuing work directly from the background thread. See [Understanding Threading](understanding-threading-section.md) for a discussion of threading considerations, including considerations for message handlers. As with the simple interface, the AMPS client provides both convenience interfaces and interfaces that use a `Command` object. The following example shows how to use the asynchronous interface. ```java showLineNumbers class MyApp { public static void main(String[] args) { /* We create a Client here, then call connect() and logon() to connect to * AMPS. */ Client client = new Client("subscribe"); try { client.connect("tcp://127.0.0.1:9007/amps/json"); client.logon(); /* Here, we create a new MessagePrinter object. The MessagePrinter * class implements MessageHandler, as described below. The subscription * uses this object to handle all messages returned from the subscription. */ MessagePrinter mp = new MessagePrinter(); Command command = new Command("subscribe").setTopic("messages"); /* Here, we call the overload of Client.execute() that specifies the * command and the message handler to invoke with messages received * in response to the command. */ CommandId subscriptionId = client.executeAsync(command, mp); } catch(AMPSException e){;} finally { client.close(); } } } ``` A sample message handler implementation is shown below: ```java showLineNumbers /* An implementation of MessageHandler provides an invoke() method that receives * a com.crankuptheamps.client.Message object. Notice that the same instance of * this class is called for all messages on a given subscription, and that the * instance is called asynchronously from the background thread created by the * client. Design your message handlers so they have access to any program state * that they need to do their work. */ public class MessagePrinter implements MessageHandler { public void invoke(Message m) { System.out.println(m.getData()); } } ``` :::warning When using asynchronous message processing, the AMPS client resets and reuses the message provided to `MessageHandler` functions between calls. This improves performance in the client, but means if your `MessageHandler` function needs to preserve information contained within the message you must copy the information rather than just saving the message object. Otherwise, the AMPS client cannot guarantee the state of the object or the contents of the object when your program goes to use it. ::: --- # Backlog and Smart Pipelining AMPS queues are designed for high-volume applications that need minimal latency and overhead. One of the features that helps performance is the *subscription backlog* feature, which allows applications to receive multiple messages at a time. The subscription backlog sets the maximum number of unacknowledged messages that AMPS will provide to the subscription. When the subscription backlog is larger than `1`, AMPS delivers additional messages to a subscriber before the subscriber has acknowledged the first message received. This technique allows subscribers to process messages as fast as possible, without ever having to wait for messages to be delivered. The technique of providing a consistent flow of messages to the application is called *smart pipelining*. ## Subscription Backlog The AMPS server determines the backlog for each subscription. An application can set the maximum backlog that it is willing to accept with the `max_backlog` option. Depending on the configuration of the queue (or queues) specified in the subscription, AMPS may assign a smaller backlog to the subscription. If no `max_backlog` option is specified, AMPS uses a `max_backlog` of `1` for that subscription. In general, applications that have a constant flow of messages perform better with a `max_backlog` setting higher than `1`. The reason for this is that, with a backlog greater than `1`, the application can always have a message waiting when the previous message is processed. Setting the optimum `max_backlog` is a matter of understanding the messaging pattern of your application and how quickly your application can process messages. To request a `max_backlog` for a subscription, you explicitly set the option on the subscribe command, as shown below: ```java Command cmd = new Command("subscribe") .setTopic("my_queue") .setOptions("max_backlog=10"); ``` --- # Before You Start Welcome to developing applications with AMPS, the Advanced Message Processing System from 60East Technologies! These guides will help you learn how to develop applications using AMPS. Before reading this guide, it is important to have a good understanding of the following topics: * *Developing Applications in Java* To be successful using this guide, you will need to possess a working knowledge of the Java language. Visit [http://java.oracle.com](http://java.oracle.com) for resources on learning Java. * *AMPS Concepts* This guide focuses on using the AMPS client libraries and how those libraries work with the AMPS server. Before working through this guide, we recommend reading the [Introduction to AMPS](/docs/intro-guide/intro) guide. Detailed explanations of the AMPS server behavior are in the [AMPS Server Documentation](/docs). You will also need a system on which you can compile and run code, and a server where you can host the AMPS server. ## Setting up a Development Instance You will need an installed and running AMPS server to use the product as well. You can write and compile programs that use AMPS without a running server, but you will get the most out of this guide by running the programs against a working server. Instructions for starting an instance of AMPS are available in the [Introduction to AMPS](/docs/intro-guide/intro) guide. :::tip The AMPS server runs on x64 Linux. The [Introduction to AMPS](/docs/intro-guide/intro) and [AMPS FAQ](/faq) contain information on how to run an AMPS server on a development system that does not run Linux. ::: ## Upgrading from a Previous Version of the AMPS Java Client The AMPS Java client is designed to maintain binary compatibility for hotfix versions, unless otherwise specified in the release notes. For versions that introduce or change features, as indicated by a change to the major or minor version number, 60East cannot guarantee binary compatibility. An upgrade that changes either a major or minor version number may require a recompile from source. Changes that do not affect source compatibility are not generally noted in upgrade documents when the major or minor version numbers change. For example, an upgrade from X.Y.1.0 to X.Y.1.1 of the Java client would be a drop-in replacement unless otherwise noted, as would an upgrade from X.Y.1.1 to X.Y.2.7. However, an upgrade from X.Y.2.7 to X.Z.0.0 may require that you recompile your application from source. --- # Client Identification AMPS uses the name of the client as a session identifier and as part of the identifier for messages originating from that client. For this reason, when a transaction log is enabled in the AMPS instance (that is, when the instance is recording a sequence of publishes and attempting to eliminate duplicate publishes), an AMPS instance will only allow one application with a given client name to connect to the instance. When a transaction log is present, AMPS **requires** the client name for a publisher to be: - Unique within a set of replicated AMPS instances - Consistent from invocation to invocation *if* the publisher will be publishing the same *logical* stream of messages If publishers do not meet this contract (for example, if the publisher changes its name and publishes the same messages, or if a different publisher uses the same session name), message loss or duplication can happen. 60East recommends always using consistent, unique client names. For example, the client name could be formed by combining the application name, an identifier for the host system, and the ID of the user running the application. A strategy like this provides a name that will be different for different users or on different systems, but consistent for instances of the application that should be treated as equivalent to the AMPS system. Likewise, if a publisher is sending a completely independent stream of messages (for example, a microservice that sends a different, unrelated sequence of messages each time it connects to AMPS), there is no need for a publisher to retain the same name each time it starts. However, if a publisher is resuming a stream of messages (as in the case when using a file-backed publish store), that publisher **must** use the same client name, since the publisher is resuming the session. --- # Client-Side Conflation In many cases, applications that use SOW topics only need the current value of a message at the time the message is processed, rather than processing each change that led to the current value. On the server side, AMPS provides *conflated topics* to meet this need. Conflated topics are described in more detail in the *AMPS User Guide*, and require no special handling on the client side. In some cases, though, it's important to conflate messages on the client side. This can be particularly useful for applications that do expensive processing on each message, applications that are more efficient when processing batches of messages, or for situations where you cannot provide an appropriate conflation interval for the server to use. A `MessageStream` has the ability to conflate messages received for a subscription to a SOW topic, view, or conflated topic. When conflation is enabled, for each message received, the client checks to see whether it has already received an unprocessed message with the same `SowKey`. If so, the client replaces the unprocessed message with the new message. The application never receives the message that has been replaced. To enable client-side conflation, you call `conflate()` on the `MessageStream`, and then use the `MessageStream` as usual: ```java showLineNumbers // SOW query and subscribe MessageStream results = ampsClient.sowAndSubscribe("orders", "/symbol == 'ROL'"); // Turn on conflation results.conflate(); // Process the results foreach (Message m : results) { // Process message here } ``` Notice that if the `MessageStream` is used for a subscription that does not include `SowKeys` (such as a subscription to a topic that does not have a SOW), no conflation will occur. When using client-side conflation with delta subscriptions, bear in mind that client-side conflation replaces the whole message, and does not attempt to merge deltas. This means that updates can be lost when messages are replaced. For some applications (for example, a ticker application that simply sends delta updates that replace the current price), this causes no problems. For other applications (for example, when several processors may be updating different fields of a message simultaneously), using conflation with deltas could result in lost data, and server-side conflation is a safer alternative. --- # Connection Parameters for AMPS When specifying a URI for connection to an AMPS server, you may specify a number of transport-specific options in the parameters section of the URI connection parameters. Here is an example: ```bash tcp://localhost:9007/amps/json?tcp_nodelay=true&tcp_sndbuf=100000 ``` In this example, we have specified the AMPS instance on `localhost`, port `9007`, connecting to a transport that uses the `amps` protocol and sending JSON messages. We have also set two parameters: `tcp_nodelay`, a Boolean (true/false) parameter, and `tcp_sndbuf`, an integer parameter. Multiple parameters may be combined to finely tune settings available on the transport. Normally, you'll want to stick with the defaults on your platform, but there may be some cases where experimentation and fine-tuning will yield higher or more efficient performance. The AMPS client supports the value of `tcp` in the *scheme* component connection string for TCP/IP connections, and the value of `tcps` as the scheme for SSL encrypted connections. ## IPv6 Connections Starting with version 5.3.3.0, the AMPS client supports creating connections over both IPv4 and IPv6 protocols if supported by the underlying Operating System. By default, the AMPS client will prefer to resolve host names to IPv4 addresses, but this behavior can be adjusted by supplying the `ip_protocol_prefer` transport option, described in the table below. ## TCP and SSL Transport Options The following transport options are available for TCP connections: |Option |Description | |-------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`bind` |

    (IP address) Sets the interface to bind the outgoing socket to.

    Starting with version 5.3.3.0, both IPv4 and IPv6 addresses are fully supported for use with this parameter.

    | |`tcp_connecttimeout` | (integer) Sets the connect timeout in milliseconds. This helps enable failover in cases where an attempt to connect to a server is unresponsive without returning a failure. | |`tcp_rcvbuf` |(integer) Sets the socket receive buffer size. This defaults to the system default size. (On Linux, you can find the system default size in `/proc/sys/net/core/rmem_default`.)| |`tcp_sndbuf` |(integer) Sets the socket send buffer size. This defaults to the system default size. (On Linux, you can find the system default size in `/proc/sys/net/core/wmem_default`.)| |`tcp_nodelay` |(boolean) Enables or disables the `TCP_NODELAY` setting on the socket. By default `TCP_NODELAY` is disabled.| |`tcp_linger` |(integer) Enables and sets the `SO_LINGER` value for the socket By default, `SO_LINGER` is enabled with a value of `10`, which specifies that the socket will linger for 10 seconds.| |`tcp_keepalive` |(boolean) Enables or disables the `SO_KEEPALIVE` value for the socket. The default value for this option is true.| |`ip_protocol_prefer` |(string) Influence the IP protocol to prefer during DNS resolution of the host. If a DNS entry of the preferred protocol can not be found, the other *non-preferred* protocol will then be tried.

    If this parameter is not set, the default will be to prefer IPv4.

    If an explicit IPv4 address or IPv6 IP address is provided as the host, the format of the IP address is used to determine the IP protocol used and this setting has no effect.

    Supported Values:

    `ipv4`: Prefer an IPv4 address when resolving the host

    `ipv6`: Prefer an IPv6 address when resolving the host

    This parameter is available starting with version 5.3.3.0.

    | ## Using HTTP Preflight for Connection Upgrades Some users need to minimize the number of externally accessible ports while still allowing multiple AMPS transports to be used in environments with strict firewall policies for security reasons. To address this, AMPS supports an HTTP Preflight mechanism that enables TCP clients to share the same external port used for WebSockets. Instead of requiring a dedicated external TCP port, clients can establish a connection over an existing HTTP endpoint. This reduces the number of open network ports while maintaining full TCP/TCPS functionality. This feature works by leveraging HTTP Upgrade requests, similar to WebSockets, allowing clients to connect via an initial HTTP request before transitioning to a full TCP/TCPS session. Additionally, the HTTP preflight mechanism enables custom HTTP headers to be included in the initial handshake, making it easier to integrate with reverse proxies. To request HTTP preflight, add the `http_preflight=true` option to the connection string. For example: ``` tcp://proxy-to-amps.example.com:8020/client/amps/json?http_preflight=true ``` To learn more about HTTP Preflight, including how to enable it and configure an NGINX proxy, refer to the [HTTP Preflight](/docs/amps-user-guide/transports/http-preflight) section in the _AMPS User Guide_ and the [HTTP Preflight- Proxy Play: AMP S Unlocked](/blog/http-preflight) blog. ## Compress Network Traffic By default, network traffic between the application and the AMPS server is not compressed. The AMPS Java client optionally supports compressing the network connection to and from AMPS. To enable this, add the `compression` option to the connection string. In this release, the AMPS Java client supports `zlib` compression to the server. Notice that compressing and decompressing data performs additional work on both the client and server side to compress and decompress the data. The goal is reducing the bandwidth required for the traffic. If bandwidth is at a premium and the data compresses well, this can produce gains in overall performance. In other situations, the benefit can be negligible or create extra latency. 60East recommends testing with your message data and volumes to determine the effect of compression for your usage. For example: ``` tcp://amps.example.com:9007/amps/json?compression=zlib ``` ## AMPS Additional Logon Options The connection string can also be used to pass logon parameters to AMPS. AMPS supports the following additional logon option: |Option |Description | |-----------------------|-----------------------------------------------------------------------------------------------| |`pretty` |Provide formatted representations of binary messages rather than the original message contents.| --- # Connection Strings for AMPS The AMPS clients use connection strings to determine the server, port, transport, and protocol to use to connect to AMPS. When the connection point in AMPS accepts multiple message types, the connection string also specifies the precise message type to use for this connection. Connection strings have a number of elements: ![Diagram showing the different elements of a connection string](./assets/client-connection-string-light.svg#gh-light-mode-only) ![Diagram showing the different elements of a connection string](./assets/client-connection-string-dark.svg#gh-dark-mode-only) As shown in the figure above, connection strings have the following elements: * _Transport_ - Defines the network used to send and receive messages from AMPS. In this case, the transport is `tcp`. For connections to transports that use the Secure Sockets Layer (SSL), use `tcps`. For connections to AMPS over a Unix domain socket, use `unix`. * _Host Address_ - Defines the destination on the network where the AMPS instance receives messages. The format of the address is dependent on the transport. For `tcp` and `tcps`, the address consists of a host name and port number. In this case, the host address is `localhost:9007`. For `unix` domain sockets, a value for hostname and port must be provided to form a valid URI, but the content of the hostname and port are ignored, and the file name provided in the **path** parameter is used instead (by convention, many connection strings use `localhost:0` to indicate that this is a local connection that does not use TCP/IP). * _Protocol_ - Sets the format in which AMPS receives commands from the client. Most code uses the default `amps` protocol, which sends header information in JSON format. AMPS supports the ability to develop custom protocols as extension modules, and AMPS also supports legacy protocols for backward compatibility. * _MessageType_ - Specifies the message type that this connection uses. This component of the connection string is required if the protocol accepts multiple message types and the transport is configured to accept multiple message types. If the protocol does not accept multiple message types, this component of the connection string is optional, and defaults to the message type specified in the transport. Legacy protocols such as `fix`, `nvfix` and `xml` only accept a single message type, and therefore do not require or accept a message type in the connection string. As an example, a connection string such as: ```bash showLineNumbers tcp://localhost:9007/amps/json ``` would work for programs connecting from the local host to a `Transport` configured as follows: ```xml showLineNumbers ... any-tcp tcp 9007 amps ... ``` See the [Configuring Transports](/docs/amps-user-guide/transports/configuring-transports) section in the _AMPS User Guide_ for more information on configuring transports. ## Using zlib Compression The AMPS Java Client supports enabling zlib compression by adding the `compression=zlib` URI parameter to the connection string. For example: ``` tcp://localhost:9007/amps/json?compression=zlib ``` No server-side configuration changes are required. The client enables zlib compression for the connection based on the URI parameter. If the connection string already contains other URI parameters, add `compression=zlib` using `&`: ``` tcp://localhost:9007/amps/json?=&compression=zlib ``` --- # Content Filtering One of the most powerful features of AMPS is content filtering. With content filtering, filters based on message content are applied at the server so that your application and the network are not utilized by messages that are not relevant to your application. For example, if your application is only displaying messages from a particular user, you can send a content filter to the server so that only messages from that particular user are sent to the client. To apply a content filter to a subscription, simply pass it into the `Client.subscribe()` call: ```java showLineNumbers CommandId subscriptionId = client.subscribe( mp, // MessageHandler implementation "messages", // Topic "/sender = 'mom'", // Content filter 5000); // Timeout ``` In this example, we have passed in a content filter `/sender = 'mom'`. This will result in the server only sending us messages, from the `messages` topic, that have the sender field equal to `mom` in the message. For example, the AMPS server will send the following message, where `/sender` is `mom`: ```javascript showLineNumbers { "sender" : "mom", "text" : "Happy Birthday!", "reminder" : "Call me Thursday!" } ``` The AMPS server will not send a message with a different `/sender` value: ```javascript showLineNumbers { "sender" : "henry dave", "text" : "Things do not change; we change." } ``` --- # Controlling Blocking The named convenience methods and the `Command` class provide a `timeout` setting that specifies how long the command should wait to receive a `processed` acknowledgment from AMPS. This can be helpful in cases where it is important for the caller to limit the amount of time to block waiting for AMPS to acknowledge the command. If the AMPS client does not receive the processed acknowledgment within the specified time, the client sends an `unsubscribe` command to the server to cancel the command and throws an exception. Acknowledgments from AMPS are processed by the client receive thread on the same socket as data from AMPS. This means that any other data previously returned (such as the results of a large query) must be consumed before the acknowledgment can be processed. An application that submits a set of SOW queries in rapid succession should set a timeout that takes into account the amount of time required to process the results of the previous query. --- # AMPS Programming: Working with Commands The AMPS clients provide named convenience methods for core AMPS functionality. These named methods work by creating messages and sending those messages to AMPS. All communication with AMPS occurs through messages. You can use the `Command` object to customize the messages that AMPS sends. This is useful for more advanced scenarios where you need precise control over AMPS, in cases where you need to use an earlier version of the client to communicate with a more recent version of AMPS, or in cases where a named method is not available. ## Understanding AMPS Messages AMPS messages are represented in the client as `AMPS.Message` objects. The `Message` object is generic, and can represent any type of AMPS message, including both outgoing and incoming messages. This section includes a brief overview of elements common to AMPS command messages. Full details of commands to AMPS are provided in the *AMPS Command Reference* (linked at the bottom of this page). All AMPS command messages contain the following elements: - **Command** - The *command* tells AMPS how to interpret the message. Without a command, AMPS will reject the message. Examples of commands include `publish`, `subscribe`, and `sow`. - **CommandId** - The *command ID*, together with the name of the client, uniquely identifies a command to AMPS. The command ID can be used later on to refer to the command or the results of the command. For example, the command ID for a `subscribe` message becomes the identifier for the subscription. The AMPS client provides a command ID when the command requires one and no command ID is set. Most AMPS commands contain the following fields: - **Topic** - The *topic* that the command applies to, or a regular expression that identifies a set of topics that the command applies to. For most commands, the topic is required. Commands such as `logon`, `start_timer`, and `stop_timer` do not apply to a specific topic, and do not need this field. - **Ack Type** - The *ack type* tells AMPS how to acknowledge the message to the client. Each command has a default acknowledgment type that AMPS uses if no other type is provided. - **Options** - The `options` are a comma-separated list of options that affect how AMPS processes and responds to the message. Beyond these fields, different commands include fields that are relevant to that particular command. For example, SOW queries, subscriptions, and some forms of SOW deletes accept the **Filter** field, which specifies the filter to apply to the subscription or query. As another example, publish commands accept the **Expiration** field, which sets the SOW expiration for the message. For full details on the options available for each command and the acknowledgment messages returned by AMPS, see the *AMPS Command Reference*. ## Creating and Populating the Command To create a command, you simply construct a command object of the appropriate type: ```java Command command = new Command("sow"); ``` Once created, you set the appropriate fields on the command. For example, the following code creates a SOW query, setting the command, topic and filter for the query: ```java showLineNumbers Command command = new Command("sow"); command.setTopic("messages-sow"); command.setFilter("/id > 20"); ``` When sent to AMPS, AMPS performs a SOW query from the topic `messages-sow` using the `filter` of `/id > 20`. The results of sending this message to AMPS are no different than using the form of the `sow` method that sets these fields. ## Using Execute Once you've created a command, use the `execute` method to send the command to AMPS. The `execute` method returns a `MessageStream` that provides response messages. The `executeAsync` method sends the message to AMPS, waits for a `processed` acknowledgment, then returns. Messages are processed on the client background thread. For example, the following snippet sends the command created above: ```java client.execute(command); ``` This returns a `MessageStream` identical to the `MessageStream` returned by the equivalent `client.sow()` method. You can also provide a message handler to receive acknowledgments, statistics, or the results of subscriptions and SOW queries. The AMPS client maintains a background thread that receives and processes incoming messages. The call to `executeAsync` returns on the main thread as soon as AMPS acknowledges the command as having been processed, and messages are received and processed on the background thread. ```java showLineNumbers import com.crankuptheamps.client.Message; import com.crankuptheamps.client.MessageHandler; class SimpleMessageHandler implements MessageHandler { // For sample purposes, just print the acknowledgment type and // reason. public void invoke(Message m) { System.out.println(m.getAckType() + " : " + m.getReason() ); } } ``` Then, to send the command and use the message handler, pass the command and the handler to `executeAsync()`. ```java client.executeAsync(command, new SimpleMessageHandler()); ``` While this message handler simply prints the ack type and reason for sample purposes, message handlers in production applications are typically designed with a specific purpose. For example, your message handler may fill a work queue, or check for success and throw an exception if the command failed. ### Using Execute to Publish Notice that the `publish` command does not typically return results other than acknowledgment messages. To send a `publish` command, use the `executeAsync()` method with a null message handler: ```java client.executeAsync(publishCmd, null); ``` Since the code provides a `null` message handler, this code does not receive acknowledgments. To detect publish failures, set the `FailedWriteHandler` for the client. The `publish` methods of the AMPS Java client are implemented internally as calls to `executeAsync` with a `null` message handler. ## AMPS Command Cookbook The [AMPS Command Reference](/docs/amps-command-reference) includes information on which fields and options to set on commands to get a specific result. The reference includes both reference information and a [Command Cookbook](/docs/amps-command-reference/cookbook) that provides a concise guide for commonly-used commands. --- # Providing Credentials to AMPS When a client logs on to AMPS, the client sends AMPS a username and password. The username is derived from the URI, using the standard syntax for providing a user name in a URI, for example, `tcp://JohnDoe:@server:port/amps/messagetype` to include the user name `JohnDoe` in the request. For a given user name, the password is provided by an `Authenticator`. The AMPS client distribution includes a `DefaultAuthenticator` that simply returns the password, if any, provided in the URI. A `logon()` command that does not specify an `Authenticator` will use an instance of `DefaultAuthenticator`. If your authentication system requires a different authentication token, you can implement an `Authenticator` that provides the appropriate token. ## Providing Credentials in a Connection String When using the `DefaultAuthenticator`, the AMPS clients support the standard format for including a username and password in a URI, as shown below: ```bash tcp://user:password@host:port/protocol/message_type ``` When provided in this form, the default authenticator provides the username and password specified in the URI. If you have implemented another authenticator, that authenticator controls how passwords are provided to the AMPS server. --- # Delta Publish To delta publish, you use the `delta_publish` command as follows: ```java showLineNumbers // assumes that client is connected and logged on String msg = ... ; // obtain changed fields here client.deltaPublish("myTopic", msg); ``` The message that you provide to AMPS must include the fields that the topic uses to generate the SOW key. Otherwise, AMPS will not be able to identify the message to update. For SOW topics that use a User-Generated SOW Key, use the `Command` form of `delta_publish` to set the `SowKey`, as shown below: ```java showLineNumbers // assumes that client is connected and logged on String msg = ... ; // obtain changed fields here String key = ... ; // obtain user-generated SOW key Command cmd("delta_publish"); cmd.setTopic("delta_topic"); cmd.setSowKey(key); cmd.setData(msg); // Execute the delta publish. Use null for // the message handler since any failure acks will // be routed to the FailedWriteHandler. client.executeAsync(cmd,null); ``` The [AMPS User Guide](/docs/amps-user-guide) section on making [Incremental Message Updates](/docs/amps-user-guide/delta-publish) describes how the AMPS server processes the `delta_publish` command. --- # Delta Subscribe To delta subscribe, you simply use the `delta_subscribe` command as follows: ```java showLineNumbers // assumes that client is connected and logged on Command cmd("delta_subscribe"); cmd.setTopic("delta_topic"); cmd.setFilter("/thingIWant = 'true'"); cmd.setOptions("oof"); // Optional try (MessageStream ms = client.execute(cmd)) for (Message m : ms) { // Work with message here } ``` As described in the [AMPS User Guide](/docs/amps-user-guide) section on [Receiving Only Updated Fields](/docs/amps-user-guide/delta-subscribe), messages provided to a delta subscription will contain the fields used to generate the SOW key and any changed fields in the message. Your application is responsible for choosing how to handle the changed fields. --- # Delta Publish and Subscribe Delta messaging in AMPS has two independent aspects: - **Delta Subscribe** - Allows subscribers to receive just the fields that are updated within a message. - **Delta Publish** - Allows publishers to update and add fields within a message by publishing only the updates into the SOW. This chapter describes how to create delta publish and delta subscribe commands using the AMPS Java client. For a discussion of this capability, how it works, and how message types support this capability see the [AMPS User Guide](/docs/amps-user-guide). --- # Detecting Write Failures The `publish` methods in the Java client deliver the message to be published to AMPS and then return immediately, without waiting for AMPS to return an acknowledgment. Likewise, the `sowDelete` methods request deletion of SOW messages, and return before AMPS processes the message and performs the deletion. This approach provides high performance for operations that are unlikely to fail in production. However, this means that the methods return before AMPS has processed the command, without the ability to return an error in the event that the command fails. The AMPS Java client provides a `FailedWriteHandler` that is called when the client receives an acknowledgment that indicates a failure to persist data within AMPS. To use this functionality, you implement the `FailedWriteHandler` interface, construct an instance of your new class, and register that instance with the `setFailedWriteHandler()` method on the client. When a `persisted` acknowledgment returns that indicates a failed write, AMPS calls the registered handler method with information from the acknowledgment message, supplemented with information from the client publish store (if one is available). Your client can log this information, present an error to the user, or take whatever action is appropriate for the failure. If your application needs to know whether publishes succeeded and are durably persisted, the following approach is recommended: - Set a `PublishStore` on the client. This will ensure that messages are retransmitted if the client becomes disconnected before the message is acknowledged *and* request `persisted` acknowledgments for messages. - Install a `FailedWriteHandler`. In the event that AMPS reports an error for a given message, that event will be reported to the `FailedWriteHandler`. - Call `publishFlush()` and verify that all messages are persisted before the application exits. When no `FailedWriteHandler` is registered, acknowledgments that indicate errors in persisting data are treated as unexpected messages and routed to the `LastChanceMessageHandler`. In this case, AMPS provides only the acknowledgment message and does not provide the additional information from the client publish store (even when one is available). --- # Disconnect Handling Every distributed system will experience occasional disconnections between one or more nodes. The reliability of the overall system depends on an application's ability to efficiently detect and recover from these disconnections. Using the AMPS Java client's disconnect handling, you can build powerful applications that are resilient in the face of connection failures and spurious disconnects. --- # Error Handling In every distributed system, the robustness of your application depends on its ability to recover gracefully from unexpected events. The AMPS client provides the building blocks necessary to ensure your application can recover from the kinds of errors and special events that may occur when using AMPS. --- # Examples The AMPS Java Client includes a set of example programs that provide simple demonstrations of client functionality. A sample archive is available that includes a set of samples and a configuration file for AMPS: **[java-examples.zip](./examples/java-examples.zip)**. The samples archive includes samples such as: | Sample Name | Demonstrates | |---------------|---------------| | ConsolePublisher.java | Publish messages to AMPS. | | ConsoleSubscriber.java | Receive messages from AMPS. | | SOWConsolePublisher.java | Publishing messages to a SOW topic. | | SOWConsoleSubscriber.java | Querying messages from a SOW topic. | | SOWandSubscribeConsoleSubscriber.java | Querying a SOW topic and entering a subscription to the topic in a single atomic operation. | | SOWUpdater.java | Publishes message to a SOW topic and then updates them for use with the SowAndSubscribeWithOOF sample. | | SowAndSubscribeWithOOF.java | Querying a SOW topic and entering a subscription to the topic in a single atomic operation while registering for notification that a previously-matching message no longer matches the subscription. | | PublishForReplay.java | Publish messages to the transaction log for use by the SubscribeForReplay example. | | SubscribeForReplay.java | Demonstrate replaying messages from the transaction log using a bookmark subscribe. | | QueuePublisher.java | Publishing messages to a queue topic. | | QueueSubscriber.java | Receiving and acknowledging messages from a queue topic. | | CompositeMessagePublisher.java | Publish a composite message using the CompositeMessageBuilder. | | CompositeMessageSubscriber.java | Receive a composite message and use the CompositeMessageParser to extract the parts of the message. | | FIXBuilderPublisher.java | Use the FIX builder convenience class to build and publish a FIX-format message. | | FIXShredderSubscriber.java | Use the FIX shredder convenience class to display a FIX-format message received from AMPS. | | NVFIXBuilderPublisher.java | Use the NVFIX builder convenience class to build and publish a FIX-format message. | | NVFIXShredderSubscriber.java | Use the NVFIX shredder convenience class to display a FIX-format message received from AMPS. | --- # Exception Handling and Asynchronous Message Processing When using asynchronous message processing, exceptions thrown from the message handler are silently absorbed by the AMPS Java client by default. The AMPS Java client allows you to register an exception listener to detect and respond to these exceptions. When an exception listener is registered, AMPS will call the exception listener with the exception. --- # Exception Types The following table details each of the exception types thrown by AMPS. | Exception | When | Notes | | ------------------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AlreadyConnectedException` | Connecting | Thrown when `connect()` is called on a `Client` that is already connected. | | `AMPSException` | Anytime | Base class for all AMPS exceptions. | | `AuthenticationException` | Anytime | Indicates that an authentication failure occurred on the server. | | `BadFilterException` | Subscribing | This typically indicates a syntax error in a filter expression. | | `BadRegexTopicException` | Subscribing | Indicates that a malformed regular expression was found in the topic name. | | `CommandException` | Anytime | Base class for all exceptions relating to commands sent to AMPS. | | `ConnectionException` | Anytime | Base class for all exceptions relating to the state of the AMPS connection. | | `ConnectionRefusedException` | Connecting | The connection was actively refused by the server. Validate that the server is running, that network connectivity is available, and the settings on the client match those on the server. | | `DisconnectedException` | Anytime | No connection is available when AMPS needed to send data to the server *or* the user's disconnect handler threw an exception. | | `InvalidTopicException` | Subscribe or Query | The topic is not configured for the requested operation. For example, a `sow` command was issued for a topic that is not in the SOW or a bookmark subscribe was issued for a topic that is not recorded in the transaction log. | | `InvalidTransportOptionsException` | Connecting | An invalid option or option value was specified in the URI. | | `InvalidURIException` | Connecting | The URI string provided to `connect()` was formatted improperly. | | `MessageTypeException` | Connecting | The class for a given transport's message type was not found in AMPS. | | `MessageTypeNotFoundException` | Connecting | The message type specified in the URI was not found in AMPS. | | `NameInUseException` | Connecting | The client name (specified when instantiating `Client`) is already in use on the server. | | `NotEntitledException` | Connecting Subscribing | The user does not have permission for the requested command (logging on or subscribing). | | `RetryOperationException` | Anytime | An error occurred that caused processing of the last command to be aborted. Try issuing the command again. | | `StreamException` | Anytime | Indicates that data corruption has occurred on the connection between the client and server. This usually indicates an internal error inside of AMPS—please contact AMPS support. | | `SubscriptionAlreadyExistsException` | Subscribing | A subscription has been requested using the same `CommandId` as another subscription. Create a unique `CommandId` for every subscription. | | `TimedOutException` | Anytime | A timeout occurred waiting for a response to a command. | | `TransportTypeException` | Connecting | Thrown when a transport type, unknown to AMPS is selected in the URI. | | `UnknownException` | Anytime | Thrown when an internal error occurs. Contact AMPS support immediately. | --- # Exceptions Generally speaking, when an error occurs that prohibits an operation from succeeding, AMPS will throw an exception. AMPS exceptions universally derive from `com.crankuptheamps.client.exception.AMPSException`, so by catching `AMPSException`, you will be sure to catch anything AMPS throws. For example: ```java showLineNumbers import com.crankuptheamps.client.Client; import com.crankuptheamps.client.exception.AMPSException; public void readAndEvaluate(Client client) { Scanner scanner = new Scanner(System.in); String payload = scanner.nextLine(); // write a new message to AMPS if ( payload != null) { try { client.publish("UserMessage", " { \"data\" : \"" + payload + "\" }"); } catch (AMPSException e) { System.err.println("An AMPS exception occurred: " + e.toString()); e.printStackTrace(); } } } ``` In this example, if an error occurs, the program writes the error to `stderr` and the `publish()` command fails. However, client is still usable for continued publishing and subscribing. When the error occurs, the exception is written to `stderr`, which calls the exception's `toString()` method. As with most Java exceptions, `toString()` will convert the `Exception` into a string that includes a message. The `printStackTrace()` method will write to `stderr` the stack trace and information on any ”inner” exceptions (exceptions from outside of AMPS that caused AMPS to throw an exception). AMPS exception types vary based on the nature of the error that occurs. In your program, if you would like to handle certain kinds of errors differently than others, you can `catch` the appropriate subclass of `AMPSException` to detect those specific errors and do something different. ```java showLineNumbers public CommandId CreateNewSubscription(Client client) { CommandId id = null; string topicName; while (id == null) { /* Our program is an interactive program that attempts to retrieve a topic * name (or regular expression) from the user. */ topicName = askUserForTopicName(); try { Command command = new Command("subscribe").setTopic(topicName); MessagePrinter mp = new MessagePrinter(); id = client.executeAsync(command, mp); } catch(BadRegexTopicException ex) { /* This line indicates that the program catches the BadRegexTopicException * exception and displays a specific error to the user indicating the topic * name or expression was invalid. By not returning from the function in * this catch block, the while loop runs again and the user is asked for * another topic name. */ System.err.println("Error: bad topic name " + "or regular expression " + topicName + ". The error was: " + ex.toString()); // we’ll ask the user for another topic } catch(AMPSException ex) { /* If an AMPS exception of a type other than BadRegexTopicException is thrown * by AMPS, it is caught here. In that case, the program emits a different * error message to the user. */ System.err.println("Error: error setting " + "up subscription to topic " + topicName + ". The error was: " + ex.toString()); return null; // give up } } return id; } ``` --- # Changing the Filter on a Subscription AMPS allows you to update parameters, such as the content filter, on a subscription. When you replace a filter on the subscription, AMPS immediately begins sending only messages that match the updated filter. Notice that if the subscription was entered with a command that includes a SOW query, using the `replace` option can re-issue the SOW query (as described in the *AMPS User Guide*). To update the filter on a subscription, you create a `subscribe` command. You set the `SubscriptionId` provided on the `Command` to the identifier of the existing subscription and include the `replace` option on the `Command`. When you send the `Command`, AMPS atomically replaces the filter and sends messages that match the updated filter from that point forward. ```java showLineNumbers // Assumes client is connected and logged on to AMPS // Enter subscription Command subscribe_cmd = new Command("sow_and_subscribe") .setTopic("orders-sow") .setSubId("A42") // Used later for replace .setFilter("/details/items/description LIKE 'puppy'"); MyHandler mh = new MyHandler(); client.executeAsync(subscribe_cmd, mh); // ... // Replace filter elsewhere in the program Command replace_cmd = new Command("sow_and_subscribe") .setTopic("orders-sow") .setSubId("A42") // A42 is the ID of the subscription to replace .setFilter("/details/items/description LIKE 'kitten'") .setOptions("replace"); client.executeAsync(replace_cmd, mh); ``` --- # Your First AMPS Program In this chapter, we will learn more about the structure and features of the AMPS Java client library, and build our first Java program using AMPS. ## About the Client Library The AMPS client is packaged as a single JAR file, `amps_client.jar`. You can find `amps_client.jar` in the `dist/lib` directory of the AMPS Java client installation. Every Java application you build will need to reference this JAR file, and the JAR file must be deployed along with your application in order for your application to function properly. ## Connecting to AMPS Let's begin by writing a simple program that connects to an AMPS server and sends a single message to a topic: ```java showLineNumbers import com.crankuptheamps.client.Client; import com.crankuptheamps.client.exception.AMPSException; public class TestPublisher { public static void main(String[] args) { Client client = new Client("TestPublisher-Client"); try { client.connect("tcp://127.0.0.1:9007/amps/json"); client.logon(); client.publish("messages", "{ \"message\" : \"Hello, world!\" } "); } catch (AMPSException aex) { System.err.println("TestListener caught exception."); } finally { client.close(); } } } ``` In the example above, we show the entire program; but future examples will isolate one or more specific portions of the code. The next section describes how to build and run the application and explains the code in further detail. ### Build and Run with a Java Command Line To build this program, you can run the following `javac` command, substituting the path to the `amps_client.jar` with the path where you have installed the AMPS Java Client: ```bash javac -classpath /opt/AMPS/api/client/java/dist/lib/amps_client.jar ./TestPublisher.java ``` `TestPublisher.class` This will create the `TestPublisher.class` file. To run the class file and send your first message to AMPS, you can issue the following command: ```bash java -cp .:/opt/AMPS/api/client/java/dist/lib/amps_client.jar TestPublisher ``` If the message is published successfully, there is no output to the console. We will demonstrate how to create a subscriber to receive messages in \` \<#java-subscriptions-chapter>\`\_. ### Examining the Code Let us now revisit the code we listed earlier. ```java showLineNumbers /* The import statements add names into reference for convenience in typing later * on in the code. These import the names from the AMPS namespaces: * com.crankuptheamps.client.Client — contains the methods for interacting with AMPS * com.crankuptheamps.client.exception.AMPSException — the package contains the * exception classes thrown by AMPS when errors occur. */ import com.crankuptheamps.client.Client; import com.crankuptheamps.client.exception.AMPSException; public class TestPublisher { public static void main(String[] args) { /* This line creates a new Client object. Client encapsulates a single * connection to an AMPS server. Methods on Client allow for connecting, * disconnecting, publishing, and subscribing to an AMPS server. The * argument to the Client constructor, "TestPublisher-Client", is a name * chosen by the client to identify itself to the server. Errors * relating to this connection will be logged with reference to this * name, and AMPS will use this name to help detect duplicate messages. * AMPS enforces uniqueness for client names when a transaction log is * configured, and it is good practice to always use unique client names. */ Client client = new Client("TestPublisher-Client"); // Here, we open a try block that concludes with catch (AMPSException aex). try { /* This statement declares a connection to AMPS with the provided * URI. The URI consists of the transport, the address, and the * protocol to use for the AMPS connection. In this case, the * transport is tcp, the address is 127.0.0.1:9007, and the protocol * is amps. This connection will be used for JSON messages. Check * with the person who manages the AMPS instance to get the connection * string to use for your programs. */ client.connect("tcp://127.0.0.1:9007/amps/json"); /* The AMPS logon() command connects to AMPS and creates a named * connection. If we had provided logon credentials in the URI, the * command would pass those credentials to AMPS. Without credentials, * the client logs on to AMPS anonymously. AMPS versions 5.0 and * later require a logon() command in the default configuration. * * This version of logon uses the DefaultAuthenticator, which provides * credentials from the URI, if any are present. To use a different * authentication scheme, implement an Authenticator. */ client.logon(); client.publish("messages", "{ \"message\" : \"Hello, world!\" } "); } // All caught exceptions in AMPS derive from AMPSException. catch (AMPSException aex) { System.err.println("TestListener caught exception."); } /* We close out the example with a finally block that closes the * Client connection and releases all accompanying resources, making * the connection eligible for garbage collection. */ finally { client.close(); } } } ``` :::tip ### About Authentication When a client logs on to AMPS, the client sends AMPS a username and password. The username is derived from the URI, using the standard syntax for providing a username in a URI. For example, `tcp://JohnDoe:@server:port/amps/messagetype` to include the username `JohnDoe` in the request. For a given username, the password is provided by an `Authenticator`. The AMPS client distribution includes a `DefaultAuthenticator` that simply returns the password, if any, provided in the URI. A `logon()` command that does not specify an `Authenticator` will use an instance of `DefaultAuthenticator`. If your authentication system requires a different authentication token, you can implement an `Authenticator` that provides the appropriate token. ::: You are now able to develop and deploy an application in Java that publishes messages to AMPS. In the following chapters, you will learn how to subscribe to messages, use content filters, work with SOW caches and fine-tune messages that you send. --- # Using a Heartbeat to Detect Disconnection The AMPS client includes a heartbeat feature to help applications detect disconnection from the server within a predictable amount of time. Without using a heartbeat, an application must rely on the operating system to notify the application when a disconnect occurs. For applications that are simply receiving messages, it can be impossible to tell whether a socket is disconnected or whether there are simply no incoming messages for the client. When you set a heartbeat, the AMPS client sends a heartbeat message to the AMPS server at a regular interval, and expects a response from the server within the specified amount of time. If the operating system reports an error on send, or if there is no activity received from the server within the specified amount of time, the AMPS client considers the server to be disconnected. Likewise, the server will ensure that traffic is sent to the client at the specified interval, using heartbeat messages when no other traffic is being sent to the client. If, after sending a heartbeat message, no traffic from the client arrives within a period twice the specified interval, the server will consider the client to be disconnected or nonresponsive. The AMPS client processes heartbeat messages on the client receive thread, which is the thread used for asynchronous message processing. If your application uses asynchronous message processing and occupies the thread for longer than the heartbeat interval, the client may fail to respond to heartbeat messages in a timely manner and may be disconnected by the server. --- # High Availability The AMPS Java Client provides an easy way to create highly-available applications using AMPS, via the `HAClient` class. `HAClient` derives from `Client` and offers the same methods, but also adds protection against network, server, and client outages. Using `HAClient` allows applications to automatically: - Recover from temporary disconnects between client and server. - Failover from one server to another when a server becomes unavailable. Since the `HAClient` automatically manages failover and reconnection, 60East recommends using the `HAClient` for applications that need to: - Automatically reconnect and resume work in the case of disconnection. - Ensure no messages are lost or duplicated after a reconnect or failover. - Persist messages and bookmarks on disk for protection against client failure. You can choose how your application uses `HAClient` features. For example, you might need automatic reconnection, but have no need to resume subscriptions or republish messages. The high availability behavior in `HAClient` is provided by implementations of defined interfaces. You can combine different implementations provided by 60East to meet your needs, and implement those interfaces to provide your own policies. Some of these features require specific configuration settings on your AMPS instance(s). This chapter mentions these features and describes how to use them from the AMPS Java client. You can find full documentation for these settings and server features in the *AMPS User Guide*. ## Overview of HAClient `HAClient` derives from `Client` and offers the same methods for sending commands to AMPS and receiving messages from AMPS. The `HAClient` differs from the `Client` in two ways: - The `HAClient` automatically installs a disconnect handler that reconnects to AMPS and resumes active (asynchronous) subscriptions. The disconnect handler optionally replays `publish` and `sow_delete` messages that have not been acknowledged by AMPS, using a `PublishStore`. The disconnect handler can optionally resume replays from the transaction log at a point that guarantees no messages are skipped and no duplicates are delivered to the application, using a `BookmarkStore`. - The `HAClient` includes the infrastructure needed for client failover, including a list of connection strings and their associated authentication mechanisms (provided by the `ServerChooser`), and options for controlling backoff behavior for reconnects (provided by the `DelayStrategy`). As a result, the `HAClient` provides a `connectAndLogon()` function for establishing a connection to AMPS, rather than treating these as independent steps that an application must manage itself. If your application needs to automatically reconnect to AMPS, 60East recommends using the `HAClient` and the automatically provided disconnect handler rather than using a `Client` or replacing the `HAClient` default disconnect handler. ## Reconnection with HAClient The most important difference between `Client` and `HAClient` is that `HAClient` automatically provides a reconnect handler. This description provides a high-level framework for understanding the components involved in failover with the `HAClient`. The components are described in more detail in the following sections. The `HAClient` reconnect handler performs the following steps when reconnecting: 1. Calls the `ServerChooser` to determine the next URI to connect to and the authenticator to use for that connection. If the connection fails, calls `getError` on the `ServerChooser` to get a description of the failure, sends an exception to the exception listener, and stops the reconnection process. 2. Calls the `DelayStrategy` to determine how long to wait before attempting to reconnect, and waits for that period of time. 3. Connects to the AMPS server. If the connection fails, calls `reportFailure` on the `ServerChooser` and begins the process again. 4. Logs on to the AMPS server. If the connection fails, calls `reportFailure` on the `ServerChooser` and begins the process again. 5. Calls `reportSuccess` on the ServerChooser. 6. Receives the bookmark for the last message that the server has persisted. Discards any older messages from the `PublishStore`. 7. Republishes any messages in the `PublishStore` that have not been persisted by the server. 8. Re-establishes subscriptions using the `SubscriptionManager` for the client. For bookmark subscriptions, the reconnect handler uses the `BookmarkStore` for the client to determine the most recent bookmark, and resubscribes with that bookmark. For subscriptions that do not use a bookmark, the `SubscriptionManager` simply re-enters the subscription, meaning that it is entered at the point at which the `HAClient` reconnects. The `ServerChooser`, `DelayStrategy`, `PublishStore`, `SubscriptionManager`, and `BookmarkStore` are all extension points for the `HAClient`. You can adapt the failover and recovery behavior by setting a different object for the behavior you want to customize on the `HAClient` or by providing your own implementation. For example, the convenience methods in the previous section customize the behavior of the `PublishStore` and `BookmarkStore` by providing either memory-backed or file-backed stores. The reconnection process runs on the thread that discovers the disconnection. This means that, in the event that an application thread discovers the disconnection as a result of a call to the Java AMPS client, that call may not return until a connection is re-established (or until the server chooser indicates failure, in which case the application will receive an exception). The Java client includes a `retryOnDisconnect` setting that controls this retry behavior when the client is disconnected. When set to `true` (the default), any call to the Client that results in a command being sent to AMPS may block until a connection is re-established. When set to `false`, the `HAClient` will retry the connection a single time and throw an exception if the connection cannot be re-established. Regardless of the `retryOnDisconnect` setting, a call to `publish` will result in the message being stored in the `PublishStore` for the client if one is set. ## Choosing Store Durability If your application needs to reliably publish to AMPS, install a `PublishStore` in the `HAClient`. If your application needs to resume replays from the transaction log, install a `BookmarkStore` in the `HAClient`. These stores provide the following capabilities: - A *bookmark store* tracks received messages, and is used to resume subscriptions. - A *publish store* tracks published messages, and is used to ensure that messages are persisted in AMPS. The AMPS Java Client provides a memory-backed version of each store and a file-backed version of each store. An `HAClient` can use either a memory backed store or a file backed store for protection. Each method provides resilience to different failures, as described below: - *Memory-backed stores* provide recovery after disconnection from AMPS by storing messages and bookmarks in your process' address space. This is the highest performance option for working with AMPS in a highly available manner. The trade-off with this method is there is no protection from a crash or failure of your client application. If your application is terminated prematurely or, if the application terminates at the same time as an AMPS instance failure or network outage, then messages may be lost or duplicated. The state of bookmark replays will be lost when the application shuts down. Messages in the publish store when the application shuts down will not be maintained through a restart, so the application will not be able to attempt any necessary redelivery when the application restarts. A memory-backed store should only be used by one instance of a client at a time. - *File-backed stores* provide recovery after client failure and disconnection from AMPS by storing messages and bookmarks on disk. To use this protection method, the `createFileBacked` convenience method requests additional arguments for the two files that will be used for both bookmark storage and message storage. If these files exist and are non-empty (as they would be after a client application is restarted), the `HAClient` loads their contents and ensures synchronization with the AMPS server once connected. The performance of this option depends heavily on the speed of the device on which these files are placed. When the files do not exist (as they would the first time a client starts on a given system), the `HAClient` creates and initializes the files, and in this case the client does not have a point at which to resume the subscription or messages to republish. A store file should only be used by one instance of a client at a time. When using a file-backed bookmark store, 60East recommends periodically removing unneeded entries by calling the `prune()` method. The precise strategy that your application uses to call `prune()` depends on the nature of the application. Most applications call `prune()` when the application exits. There are two basic strategies that applications follow while the application runs: - Install a resize handler and call `prune()` after a specified number of resize operations, or when the store reaches a specific size. - Call `prune()` after a specific number of messages are processed (for example, every 10,000 messages received or every 1,000 updates completed). Regardless of the strategy, it is best to call `prune()` when the application is idle, since the `prune()` call rewrites the log file. The store interface is public, and an application can create and provide a custom store as necessary. While clients provide convenience methods for creating file-backed and memory-backed `HAClient` objects with the appropriate stores, you can also create and set the stores in your application code. The AMPS Java client also includes default stores, which implement the appropriate interface, but do not actually persist messages. Starting in 5.3.2.0, the AMPS client contains a recovery point adapter interface to make it easy to add a custom persistence layer to a bookmark store. The distribution includes a recovery point adapter that can store bookmark recovery information in an AMPS SOW topic. The `HAClient` provides convenience methods for creating clients and setting stores. You can also construct an `HAClient` and set whichever store implementations you choose. In this example, we create several clients. The first client uses memory stores for both bookmarks and publishes. The second client uses files for both bookmarks and publishes. The third client uses a file for bookmarks. The third client does not set a store for publishes, which means that AMPS provides the default store (and no outgoing messages are stored). The final client does not specify any stores, so has no persistence for published messages or bookmark subscriptions, but can take advantage of the automatic failover and reconnection in the `HAClient`. ```java showLineNumbers // Memory publish store, memory bookmark store HAClient memoryClient = HAClient.createMemoryBacked( "lessImportantMessages"); // File-backed publish store, file-backed bookmark store HAClient diskClient = HAClient.createFileBacked( "moreImportantMessages", "/mnt/fastDisk/moreImportantMessages.outgoing", "/mnt/fastDisk/moreImportantMessages.incoming"); // Default publish store, file-backed bookmark store HAClient subscriberClient = new HAClient("subscriber"); subscriberClient .setBookmarkStore(new LoggedBookmarkStore("/mnt/fastDisk/bookmark.store")); // Default publish store, default bookmark store // Failover behavior only HAClient streamReader = new HAClient("streamReader"); ``` :::info While this chapter presents the built-in file and memory-based stores, the AMPS Java Client provides open interfaces that allow development of custom persistent message stores. You can implement the `Store` and `BookmarkStore` interfaces in your code, and then pass instances of those to `setPublishStore()` or `setBookmarkStore()` methods in your `Client`. You can implement the `RecoveryPointAdapter` interface to easily add a custom storage mechanism to one of the 60East-provided bookmark store implementations. ::: ### Using the SOW Recovery Point Adapter The AMPS client also includes the ability to use a SOW topic to store bookmark state for a bookmark store. This can be a useful option in a situation where an application needs a persistent bookmark store, but does not have the ability to store a file on the filesystem, or where an application has a bookmark file, but wants to have the ability to resume the subscription if the file is lost or damaged, or if the application is started on a system that does not have access to the file. To use the SOW topic recovery point adapter, you create a bookmark store of the type you would like to use for the `Client`, passing an adapter when you construct the store. You then set this bookmark store as the store for the `Client` to use. The constructor for the SOW recovery adapter allows you to customize the topic name and field names used to store the recovery point information in AMPS. As with the `RecoveryPointAdapter` interface in general, it is possible to customize the behavior of the SOW recovery point adapter by overriding the provided methods. This section describes how to use the adapter with the default settings. Should you need to change the behavior of the class, you would adjust the guidance in this section accordingly. (For example, if you override methods to produce a message with a different set of keys or a different message format, you would update the topic definition accordingly). ### AMPS Topic Configuration To store recovery point state in AMPS, the AMPS instance that will store the recovery point state must define a `SOW/Topic` to hold the recovery point data. By default, the adapter uses a topic named `/ADMIN/bookmark_store` of `json` message type, with the `/clientName` and `/subId` fields as keys, similar to the following definition: ```xml showLineNumbers /ADMIN/bookmark_store json /clientName /subId ``` You must include this definition, or an equivalent definition, in the configuration file for the AMPS instance that will host the recovery point. If you define a topic with a different configuration (for example, different key names, a different topic name or a different message type), you must ensure that the adapter that you create uses the same parameters as those configured on the server. ### Constructing a Client for the Adapter The AMPS SOW Recovery Point Adapter requires a `Client` or `HAClient` connected to the instance that contains the SOW topic. The Adapter will use this client to recover bookmark state and store bookmarks in AMPS. Notice that this client **must not** be a client that the Adapter is keeping state for. This must be a completely separate client instance, otherwise the client may deadlock while updating the store. The client must be connected and logged in to the instance that contains the SOW topic, using the message type defined for the topic. ### Capacity Planning and Store Sizing When an application uses a file-backed store, it is important to make sure that there is enough space available on the file system to be able to manage the store. For logged bookmark stores, an application needs to keep a bookmark record for each message received, each message discarded, and the persisted acknowledgments delivered by the server approximately once a second. Each bookmark entry consumes roughly 70 bytes of storage *plus* the length of the subscription ID for the subscription receiving the message. The logged bookmark store retains entries until an application explicitly calls `prune()`. The capacity needed for a logged bookmark store will depend on the strategy that the application uses for pruning the file. For a file-backed publish store, the application needs to be able to store published messages until the AMPS server that the publisher is connected to acknowledges those messages as persisted. The volume of messages that needs to be stored depends on the failover policy for the server -- that is, the maximum amount of time that the server will allow a downstream instance to fail to acknowledge a message before the server downgrades that connection to `async` acknowledgment. By default, AMPS does not downgrade connections: this policy must be set explicitly using the AMPS actions. As an example, if the server is configured to downgrade connections that are more than 120 seconds behind, then -- for disaster recovery -- the application must have the capacity to store 120 seconds of published messages at peak publishing load. However, unlike the logged bookmark store, a file-backed publish store removes messages from the store and reuses the space once AMPS has acknowledged the message. ## Connections and the Server Chooser Unlike `Client`, the `HAClient` attempts to keep itself connected to an AMPS instance at all times, by automatically reconnecting or failing over when it detects that the client is disconnected. When you are using the `Client` directly, your disconnect handler usually takes care of reconnection. `HAClient`, on the other hand, provides a disconnect handler that automatically reconnects to the current server or to the next available server. To inform the `HAClient` of the addresses of the AMPS instances in your system, you pass a `ServerChooser` instance to the `HAClient`. `ServerChooser` acts as a smart enumerator over the servers available: `HAClient` calls `ServerChooser` methods to inquire about what server should be connected, and calls methods to indicate whether a given server succeeded or failed. The AMPS Java Client provides a simple implementation of `ServerChooser`, called `DefaultServerChooser`, that provides very simple logic for reconnecting. This server chooser is most suitable for basic testing, or in cases where an application should simply rotate through a list of servers. For most applications, you implement the `ServerChooser` interface yourself for more advanced logic, such as choosing a backup server based on your network topology, or limiting the number of times your application should try to reconnect to a given address. To connect to AMPS, you provide a `ServerChooser` to `HAClient` and then invoke `connectAndLogon()` to create the first connection: ```java showLineNumbers HAClient myClient = HAClient.createMemoryBacked( "myClient"); /* primary.amps.xyz.com is the primary AMPS instance, and * secondary.amps.xyz.com is the secondary */ DefaultServerChooser chooser = new DefaultServerChooser(); chooser.add("tcp://primary.amps.xyz.com:12345/amps/json"); chooser.add("tcp://secondary.amps.xyz.com:12345/amps/json"); myClient.setServerChooser(chooser); myClient.connectAndLogon(); ... myClient.disconnect(); ``` Similar to `Client`, `HAClient` remains connected to the server until `disconnect()` is called. Unlike `Client`, `HAClient` automatically attempts to reconnect to your server if it detects a disconnect and, if that server cannot be connected, fails over to the next server provided by the `ServerChooser`. In this example, the call to `connectAndLogon()` attempts to connect and login to `primary.amps.xyz.com`, and returns if that is successful. If it cannot connect, it tries `secondary.amps.xyz.com`, and continues trying servers from the `ServerChooser` until a connection is established. Likewise, if it detects a disconnection while the client is in use, then `HAClient` attempts to reconnect to the server with which it was most recently connected; if that is not possible, then it moves on to the next server provided by the `ServerChooser`. The default `ServerChooser` simply provides the next URL in the sequence. This strategy works for many applications. If you need a different strategy, you can implement your own logic for failover by creating a class derived from `ServerChooser`. ### Setting a Reconnect Delay and Timeout You can control the amount of time between reconnection attempts and set a total amount of time for the `HAClient` to attempt to reconnect. The AMPS Java Client includes an interface for managing this behavior called the `ReconnectDelayStrategy`. Two implementations of this interface are provided with the client: - `FixedDelayStrategy` provides the same delay each time the `HAClient` tries to reconnect. - `ExponentialDelayStrategy` provides an exponential backoff until a connection attempt succeeds. To use either of these classes, you simply create an instance, set the appropriate parameters, and install that instance as the delay strategy for the `HAClient`. For example, the following code sets up a reconnect delay that starts at 200ms and increases the delay by 1.5 times after each failure. The strategy allows a maximum delay between connection attempts of 5 seconds, and will not retry longer than 60 seconds. ```java showLineNumbers HAClient theClient = HAClient.createMemoryBacked("demo"); ExponentialDelayStrategy theStrategy = new ExponentialDelayStrategy(); theStrategy.setInitialDelay(200); theStrategy.setBackoffExponent(1.5); theStrategy.setMaximumDelay(5000); theStrategy.setMaximumRetryTime(60000); theClient.setDelayStrategy(theStrategy); ``` ### Implementing a Server Chooser As described above, you provide the `HAClient` with connection strings to one or more AMPS servers using a `ServerChooser`. The purpose of a `ServerChooser` is to provide information to the `HAClient`. A `ServerChooser` does not manage the reconnection process, and should not call methods on the `HAClient`. A `ServerChooser` has two required responsibilities to the `HAClient`: - Tells the `HAClient` the connection string for the server to connect to. If there are no servers, or the `ServerChooser` wants the connection to fail, the `ServerChooser` returns an empty string. To provide this information, the `ServerChooser` implements the `getCurrentURI()` method. - Provides an `Authenticator` for the current connection string. This is especially important for installations where different servers require different credentials or authentication tokens must be reset after each connection attempt. To provide the authenticator, the `ServerChooser` implements the `getCurrentAuthenticator()` method. The `HAClient` calls the `getCurrentURI()` and `getCurrentAuthenticator()` methods each time it needs to make a connection. Each time a connection succeeds, the `HAClient` calls the `reportSuccess()` method of the `ServerChooser`. Each time a connection fails, the `HAClient` calls the `reportFailure()` method of the `ServerChooser`. The `HAClient` does not require the `ServerChooser` to take any particular action when it calls these methods. These methods are provided for the `HAClient` to do internal maintenance, logging, or record keeping. For example, an `HAClient` might keep a list of available URIs with a current failure count, and skip over URIs that have failed more than 5 consecutive times until all URIs in the list have failed more than 5 consecutive times. When the `ServerChooser` returns an empty string from `getCurrentURI()`, indicating that no servers are available for connection, the `HAClient` calls `getError()` method on the `ServerChooser` and includes the string returned by `getError()` in the generated exception. ## Heartbeats and Failure Detection Use of the `HAClient` allows your application to quickly recover from detected connection failures. By default, connection failure detection occurs when AMPS receives an operating system error on the connection. This system may result in unpredictable delays in detecting a connection failure on the client, particularly when failures in network routing hardware occur, and the client primarily acts as a subscriber. The heartbeat feature of the AMPS client allows connection failure to be detected quickly. Heartbeats ensure that regular messages are sent between the AMPS client and server on a predictable schedule. The AMPS client and server both assume disconnection has occurred if there is no other activity and these regular heartbeats cease, ensuring disconnection is detected in a timely manner. To use the heartbeat feature, call the `setHeartbeat` method on `Client` or `HAClient`: ```java showLineNumbers HAClient client = HAClient.createMemoryBacked("importantStuff"); ... client.setHeartbeat(3); client.connectAndLogon(); ... ``` Method `setHeartbeat` takes one parameter: the heartbeat interval. The heartbeat interval specifies the periodicity of heartbeat messages sent by the server: the value `3` indicates messages are sent on a three-second interval. If the client receives no messages in a six-second window (two heartbeat intervals), the connection is assumed to be dead, and the `HAClient` attempts reconnection. An additional variant of `setHeartbeat` allows the idle period to be set to a value other than two heartbeat intervals. (The server, however, will always consider the connection to be closed after two heartbeat intervals without any traffic.) Notice that, for `HAClient`, `setHeartbeat` must be called *before* the client is connected. For `Client`, `setHeartbeat` may be called either *before or after* the client is connected. :::warning Heartbeats are serviced on the receive thread created by the AMPS client. Your application must not block the receive thread for longer than the heartbeat interval, or the application is subject to being disconnected. ::: ## Considerations for Publishers Publishing with an `HAClient` is nearly identical to regular publishing; you simply call the `publish()` method with your message’s topic and data. The AMPS client sends the message to AMPS, and then returns from the `publish()` call. For maximum performance, the client does not wait for the AMPS server to acknowledge that the message has been received. When an `HAClient` uses a publish store (other than the `DefaultPublishStore`), the publish store retains a copy of each outgoing message and requests that AMPS acknowledge that the message has been persisted. The AMPS server acknowledges messages back to the publisher. Acknowledgments can be delivered for multiple messages at periodic intervals (for topics recorded in the transaction log) or after each message (for topics that are not recorded in the transaction log). When an acknowledgment for a message is received, the `HAClient` removes that message from the publish store. When a connection to a server is made, the `HAClient` automatically determines which messages from the publish store (if any) the server has not processed, and replays those messages to the server once the connection is established. For reliable publishers, the application must choose how best to handle application shutdown. For example, it is possible for the network to fail immediately after the publisher sends the message, while the message is still in transit. In this case, the publisher has sent the message, but the server has not processed it and acknowledged it. During normal operation, the `HAClient` will automatically connect and retry the message. On shutdown, however, the application must decide whether to wait for messages to be acknowledged, or whether to exit. Publish store implementations provide an `unpersistedCount()` method that reports the number of messages that have not yet been acknowledged by the AMPS server. When the `unpersistedCount()` reaches `0`, there are no unpersisted messages in the local publish store. For the highest level of safety, an application can wait until the `unpersistedCount()` reaches `0`, which indicates that all of the messages have been persisted to the instance that the application is connected to, and the synchronous replication destinations configured for that instance. When a synchronous replication destination goes offline, this approach will cause the publisher to wait to exit until the destination comes back online or until the destination is downgraded to asynchronous replication. For applications that are shut down periodically for short periods of time (for example, applications that are only offline during a weekly maintenance window), another approach is to use the `publishFlush()` method to ensure that messages are delivered to AMPS, and then rely on the connection logic to replay messages as necessary when the application restarts. For example, the following code flushes messages to AMPS, then warns if not all messages have been acknowledged: ```java showLineNumbers HAClient pub = HAClient.createMemoryBacked( "importantStuff"); ... pub.connectAndLogon(); String topic = "loggedTopic"; String data = ...; for (int i = 0; i < MESSAGE_COUNT; i++) { pub.publish(topic, data); } // We think we are done, but the server may not // have acknowledged us yet. Wait for up to ten // seconds for the server to acknowledge publishes. try { pub.publishFlush(10000); } catch (TimedOutException e) { System.out.println("Timed out waiting for final " + " ack from the server..."); // Log error or alert user that server has not // acknowledged messages. Since this is a // memory backed publish store, the messages // will not be preserved on restart. } pub.disconnect(); ``` In this example, the client sends each message immediately when `publish()` is called. If AMPS becomes unavailable between the final `publish()` and the `disconnect()`, or one of the servers that the AMPS instance replicates to is offline, the client may not have received a persisted acknowledgment for all of the published messages. For example, if a message has not yet been persisted by all of the servers in the replication fabric that are connected with synchronous replication, AMPS will not have acknowledged the message. Before shutting down the client, the code does two things: - The code flushes messages to the server to ensure that all messages have been delivered to AMPS. - The code waits for up to 10 seconds for all of the messages in the publish store to be acknowledged as persisted by AMPS. If the messages have not been acknowledged, they will remain in the publish store file and will be published to AMPS, if necessary, the next time the application connects. An application may choose to loop until `unpersistedCount()` returns `0`, or (as we do in this case) simply warn that AMPS has not confirmed that the messages are fully persisted. The behavior you choose in your application should be consistent with the high-availability guarantees your application needs to provide. :::warning AMPS uses the name of the `HAClient` to determine the origin of messages. For the AMPS server to correctly identify duplicate messages, each instance of an application that publishes messages must use a distinct name. That name must be consistent across different runs of the application. ::: If your application crashes or is terminated, some published messages may not have been persisted in the AMPS server. If you use the file-based store—in other words, the store created by adding a file-backed publish store to the client or using `HAClient.createFileBacked()` — the `HAClient` will recover the messages, and once logged on, will correlate the message store to what the AMPS server has received, re-publishing any missing messages. This occurs automatically when `HAClient` connects, without any explicit consideration in your code, other than ensuring that the same file name is passed to `createFileBacked()` if recovery is desired. :::warning AMPS provides persisted acknowledgment messages for topics that do not have a transaction log enabled. However, the level of durability provided for topics with no transaction log is minimal. Learn more about transaction logs in the *AMPS User Guide*. ::: ## Detecting Failover Ahead of Replication AMPS replication provides two different acknowledgment modes for outgoing replication links from an instance: - For a link in `sync` acknowledgment mode, a message must be successfully acknowledged by the downstream instance of AMPS before this instance of AMPS will acknowledge the message. - For a link in `async` acknowledgment mode, this link is not considered for acknowledging the message. In this mode, the downstream side of the replication link may not have received or processed the message at the time that the publisher receives an acknowledgment. As described in the *AMPS User Guide*, a publisher must not failover from one instance of AMPS to another instance when any link between those instances uses `async` acknowledgment *unless* replication is certain to have reached that instance. (For example, if replication is taking a maximum of 1.2 seconds between the instances and the publisher has been disconnected for 30 seconds, all messages from that publisher will have been replicated). To help detect a situation where a publisher may be "jumping ahead" of messages that it has published, but which have not yet been replicated, the AMPS client allows an application to consider it to be an error to make a connection to a server that has not received messages previously published by the application. To enable this behavior, set the `setErrorOnPublishGap()` method to set this property on the `PublishStore` in use for the client. When this property is set, the client will consider it to be an error to connect to a server that has not received messages previously published by the client, and consider the connection to have failed. Notice that an application that uses this approach may need to handle situations where no server has received the message, particularly if the replication configuration uses automated replication downgrade. ## Considerations for Subscribers `HAClient` provides two important features for applications that subscribe to one or more topics: re-subscription, and a bookmark store to track the correct point at which to resume a bookmark subscription. ### Resubscription with Asynchronous Message Processing Any asynchronous subscription placed using an `HAClient` is automatically reinstated after a disconnect or a failover. These subscriptions are placed in an in-memory `SubscriptionManager`, which is created automatically when the `HAClient` is instantiated. Most applications will use this built-in subscription manager, but for applications that create a varying number of subscriptions, you may wish to implement `SubscriptionManager` to store subscriptions in a more durable place. Note that these subscriptions contain no message data, but rather simply contain the parameters of the subscription itself (for instance, the command, topic, message handler, options, and filter). When a re-subscription occurs, the AMPS Java Client re-executes the command as originally submitted, including the original topic, options, and so on. AMPS sends the subscriber any messages for the specified topic (or topic expression) that are published after the subscription is placed. For a `sow_and_subscribe` command, this means that the client re-issues the full command, including the SOW query as well as the subscription. :::tip A `sow` command is a point-in-time query. It isn't added to the subscription manager, and isn't restarted if a disconnection happens in the middle of a query. A `sow_and_subscribe` is a subscription, and is added to the subscription manager. ::: ### Resubscription with Synchronous Message Processing The `HAClient` (starting with the AMPS Java Client version 4.3.1.2) does not track synchronous message processing subscriptions in the `SubscriptionManager`. The reason for this is to preserve the expected behavior of an `Iterator`. That is, once the `MessageStream` indicates that there are no more elements in the stream, the `MessageStream` does not suddenly produce more elements. To re-subscribe when the `HAClient` fails over, you can simply re-issue the subscription. For example, the snippet below re-issues the subscribe command when the message stream ends: ```java showLineNumbers boolean still_need_to_process = true; while (still_need_to_process == true) { MessageStream ms = client.subscribe("topic"); try { for (Message m : ms) { // process message // check condition on still_need_to_process if (still_need_to_process == false) break; } } finally { // End of stream, for a subscribe this means // that the connection is likely closed. if (ms != null) ms.close(); } } ``` ### Bookmark Stores In cases where it is critical not to miss a single message, it is important to be able to resume a subscription at the exact point that a failure occurred. In this case, simply recreating a subscription isn't sufficient. Even though the subscription is recreated, the subscriber may have been disconnected at precisely the wrong time, and will not see the message. To ensure delivery of every message from a topic or set of topics, the AMPS `HAClient` includes a `BookmarkStore` that, combined with the bookmark subscription and transaction log functionality in the AMPS server, ensures that clients receive any messages that might have been missed. The client stores the bookmark associated with each message received, and tracks whether the application has processed that message; if a disconnect occurs, the client uses the `BookmarkStore` to determine the correct resubscription point, and sends that bookmark to AMPS when it re-subscribes. AMPS then replays messages from its transaction log from the point after the specified bookmark, thus ensuring the client is completely up-to-date. `HAClient` helps you to take advantage of this bookmark mechanism through the `BookmarkStore` interface and `bookmarkSubscribe()` method on `Client`. Whenever a disconnection or failover occurs for subscriptions created with `bookmarkSubscribe()`, your application will automatically resubscribe to the message after the last message it processed. `HAClient`s created by `createFileBacked()` additionally store these bookmarks on disk, so that the application can restart with the appropriate message if the client application fails and restarts. To take advantage of the `BookmarkStore` and bookmark subscriptions, do the following: - Ensure the topic(s) to be subscribed to are included in a transaction log. See the *AMPS User Guide* for information on how to specify the contents of a transaction log. - Use `bookmarkSubscribe()` instead of `subscribe()` when creating a `subscription()`, and decide how the application will manage subscription identifiers (SubIds). If you are using a command object, you can simply provide a bookmark on that object. - Use the `BookmarkStore.discard()` method in message handlers to indicate when a message has been fully processed by the application, that is, when the application does not need to receive the message again if the application fails over. The following example creates a bookmark subscription against a transaction-logged topic, and fully processes each message as soon as it is delivered: ```java showLineNumbers final HAClient client = HAClient.createFileBacked( "aClient", "/logs/aClient.publishLog", "/logs/aClient.subscribeLog"); class MyMessageHandler implements MessageHandler { public void invoke(Message message) { ... client.getBookmarkStore().discard( message.getSubIdRaw(), message.getBookmarkSeqNo()); ... } } Command command = new Command("subscribe") .setTopic("myTopic") .setSubId(new CommandId("MySubId")) .setBookmark(Client.Bookmarks.MOST_RECENT); client.executeAsync(command, new myMessageHandler()); ``` In this example, the client is a file-backed client, meaning that arriving bookmarks will be stored in a file (`Client.subscribeLog`). Storing these bookmarks in a file allows the application to restart the subscription from the last message processed, in the event of either server or client failure. :::tip For optimum performance, it is critical to discard every message once its processing is complete. If a message is never discarded, it remains in the bookmark store. During re-subscription, `HAClient` always restarts the bookmark subscription with the oldest undiscarded message, and then filters out any more recent messages that have been discarded. If an old message remains in the store, but is no longer important for the application’s functioning, then the client and the AMPS server will incur unnecessary network, disk, and CPU activity. ::: The `SubscriptionId` field specifies an identifier to be used for this subscription. Passing null, or leaving the field unset, causes `HAClient` to generate a new subscription ID, like most other `Client` functions. However, if you wish to resume a subscription from a previous point after the application has terminated and restarted, the application must pass the same subscription ID as during its previous run. Passing a different subscription ID bypasses any recovery mechanisms, creating an entirely new subscription. When you use an existing subscription ID, the `HAClient` locates the last-used bookmark for that subscription in the local store, and attempts to re-subscribe from that point. Below are the different bookmark types that can be used to enable different recovery strategies for an application: - `Client.Bookmarks.NOW` specifies that the subscription should begin from the moment the server receives the subscription request. This results in the same messages being delivered as if you had invoked `subscribe()` instead, except that the messages will be accompanied by bookmarks. This is also the behavior that results if you supply an invalid bookmark. - `Client.Bookmarks.EPOCH` specifies that the subscription should begin from the beginning of the AMPS transaction log. - `Client.Bookmarks.MOST_RECENT` specifies that the subscription should begin from the last-used message in the associated `BookmarkStore`. Alternatively, if this subscription has not been seen before, it instructs the subscription to begin with `EPOCH`. This is the most common value for this parameter, and is the value used in the preceding example. By using `MOST_RECENT`, the application automatically resumes from wherever the subscription left off, taking into account any messages that have already been processed and discarded. When the `HAClient` re-subscribes after a disconnection and reconnection, it always uses `MOST_RECENT`, ensuring that the continued subscription always begins from the last message used before the disconnect, so that no messages are missed. ## Conclusion With only a few changes, most AMPS applications can take advantage of the `HAClient` and associated classes to become more highly-available and resilient. Using the `PublishStore`, publishers can ensure that every message published has actually been persisted by AMPS. Using `BookmarkStore`, subscribers can make sure that there are no gaps or duplicates in the messages received. `HAClient` makes both kinds of applications more resilient to network and server outages, as well as temporary issues. By using the file based `HAClient`, clients can recover their state after an unexpected termination or crash. Though `HAClient` provides useful defaults for the `Store`, `BookmarkStore`, `SubscriptionManager`, and `ServerChooser`, you can customize any or all of these to the specific needs of your application and architecture. --- # Obtaining and Installing the AMPS Java Client ## Prerequisites Before proceeding with this guide, make sure that the following programs are installed and functioning properly on your development machine: - Java Development Kit version 8 or greater - Java Runtime Environment version 8 or greater - Apache Ant version 1.8 or greater (*optional*) ## Obtaining the Client The AMPS Java client can be installed from the Maven Central Repository using `com.crankuptheamps` as the `groupId` and `amps-client` as the `artifactId` for the package coordinates. The AMPS Java client source code and pre-compiled JARs are also available in the AMPS Java client distribution on the [60East Technologies](https://www.crankuptheamps.com/develop/) website. The pre-compiled JAR files are located in the `dist/lib/` directory, which contains the following JAR files: - `amps_client.jar` contains the `com.crankuptheamps.client` package, which includes the classes necessary to build an AMPS client. This JAR and its contents will be discussed throughout this Developer Guide. - `amps_client-sources.jar` contains the source code files used to create the Java implementation of the AMPS client libraries. This file can be included in an IDE to assist in debugging. It can also be used to rebuild the AMPS client libraries if any custom changes are necessary. See [Rebuilding the Client](advanced-topics#rebuilding-the-client) for instructions on how to recompile the AMPS Java client source. - `amps_client-javadoc.jar`, like the `amps_client-sources.jar`, can be included in an IDE to provide the javadoc annotations for the implemented classes and methods. ## Test Connectivity to AMPS Before writing programs in AMPS, make sure connectivity to your AMPS development instance is working from your AMPS development environment. Launch a terminal window and change the directory to the AMPS directory in your AMPS server installation and use `spark` to test connectivity to your server. For example: ```bash ./bin/spark ping -server localhost:9007/amps/json ``` This uses the `spark` utility included with the AMPS server to test connectivity to an AMPS server instance running on localhost port `9007` using the `amps` protocol and the `json` message type. If the AMPS instance you will use for development is running on a different host or port, you can adjust the hostname, port, and message type to match the values you intend to use in your application. For information on setting up an AMPS development environment, see the [Getting Started with AMPS](../../docs/intro-guide/getting_started) section in the [AMPS Introduction](../../docs/intro-guide/intro). --- # Managing Disconnection The `HAClient` class, included with the AMPS Java client, contains a disconnect handler and other features for building highly-available applications. The `HAClient` includes features for managing a list of failover servers, resuming subscriptions, republishing in-flight messages, and other functionality that is commonly needed for high availability. 60East recommends using the `HAClient` for automatic reconnection wherever possible, as the HAClient disconnect handler has been carefully crafted to handle a wide variety of edge cases and potential failures. If an application needs to reconnect or fail over, use an `HAClient`, and the AMPS client library will automatically handle failover and reconnection. You control which servers the client fails over to using an implementation of the `ServerChooser` interface, and you can control the timing of the failover using an implementation of the `ReconnectDelayStrategy` interface. :::info For most applications, the combination of the `HAClient` disconnect handler and a `ConnectionStateListener` gives you the ability to monitor disconnections and add custom behavior at the appropriate point in the reconnection process. ::: If you need to add custom behavior to the failover (such as logging, resetting an internal cache, refreshing credentials and so on), the `ConnectionStateListener` class allows your application to be notified and take action when disconnection is detected and at each stage of the reconnection process. To extend the behavior of the AMPS client during reconnection, implement a `ConnectionStateListener`. --- # Managing SOW Contents AMPS allows applications to manage the contents of the SOW by explicitly deleting messages that are no longer relevant. For example, if a particular delivery van is retired from service, the application can remove the record for the van by deleting the record for the van. The client provides the following methods for deleting records from the SOW. - `sowDelete` - Accepts a filter, and deletes all messages that match the filter. - `sowDeleteByKeys` - Accepts a set of SOW keys as a comma-delimited string and deletes messages for those keys, regardless of the contents of the messages. A SOW key is provided in the header of a SOW message, and is the internal identifier AMPS uses for that SOW message. - `sowDeleteByData` - Accepts a message, and deletes the record that would be updated by that message. The most efficient way to remove messages from the SOW is to use `sowDeleteByKeys` or `sowDeleteByData`, since those options allow AMPS to exactly target the message or messages to be removed. Many applications use `sowDelete`, since this is the most flexible method for removing items from the SOW when the application does not have information on the exact messages to be removed. Regardless of the command used, AMPS sends an OOF message to all subscribers who have received updates for the messages removed, as described in the previous section. The simple form of the `sowDelete` command returns a `Message` that receives the response. The response is an acknowledgment message that contains information on the delete command. For example, the following snippet simply prints informational text with the number of messages deleted: ```java showLineNumbers Message msg = client.sowDelete("sow_topic", "/id IN (42, 64, 37)", 0); System.out.println("Got an " + msg.getCommand() + " containing " + msg.getAckType() + ": " + " deleted " + msg.getMatches() + " messages."); ``` You can also use `client.execute` to send a SOW delete command. As with the other SOW methods, the client provides an asynchronous versions of the SOW delete commands that require a message handler to be invoked. Acknowledging messages from a queue uses a form of the `sow_delete` command that is only supported for queues. Acknowledgment is discussed in the [Using Queues](queues) chapter in this guide. --- # Manual Acknowledgement 60East generally recommends that applications use an `ack()` method to acknowledge messages during normal processing. This approach functions correctly when used within a message handler, supports batching as explained elsewhere in this chapter, and is generally both easier to code and more efficient. However, in some situations, you may need to manually acknowledge messages in the queue. This is most common when an application needs to operate on all messages with certain characteristics, rather than acknowledging individual messages. For example, an application that is doing updates to an order may want to cancel an order by both publishing a cancellation and immediately expiring all other messages in the queue for that order. With manual acknowledgment, that application can use a filter to remove all previous updates for that order, then publish the cancellation. To manually acknowledge processed messages and remove the messages from the queue, applications use the `sow_delete` command. To remove specific messages from the queue, provide the bookmarks of those messages. To remove messages that match a given filter, provide the filter. Notice that AMPS only supports using a bookmark with `sow_delete` when removing messages from a queue, not when removing records from a SOW. For example, given a `Message` object to acknowledge and a client, the code below acknowledges the message. ```java showLineNumbers void acknowledgeSingle(Client client, Message message) throws AMPSException { Command acknowledge = Command("sow_delete"); acknowledge.setTopic(message.getTopic()) .setBookmark(message.getBookmark()); client.executeAsync(acknowledge, null); } ``` In the example above, the program creates a `sow_delete` command, specifies the topic and the bookmark, and then sends the command to the server. While this method works, creating and sending an acknowledgment for each individual message can be inefficient if your application is processing a large volume of messages. Rather than acknowledging each message individually, your application can build a comma-delimited list of bookmarks from the processed messages and acknowledge all of the messages at the same time. In this case, it's important to be sure that the number of messages you wait for is less than the maximum backlog -- the number of messages your client can have unacknowledged at a given time. Notice that both automatic acknowledgment and the helper method on the `Message` object take the maximum backlog into account. When constructing a command to acknowledge queue messages, AMPS allows an application to specify a filter rather than a set of bookmarks. AMPS interprets this as the client requesting acknowledgment of all messages that match the filter. (This may include messages that the client has not received, subject to the `Leasing` model for the queue.) As a more typical example of manual acknowledgment, the code below expires all messages for a given `id` that have a status other than `cancel`. An application might do this to halt processing of an order that it is about to cancel. ```java showLineNumbers void removePending(Client client, string orderId) throws AMPSException { Command acknowledge = Command("sow_delete"); acknowledge.setTopic(message.getTopic()) .setFilter("/id = '" + orderId + "' and /status != 'cancel'") .setOptions("expire"); client.executeAsync(acknowledge, null); } ``` In the example above, the program specifies a topic and a filter to use to find the messages that should be removed. In this case, the program also provides the `expire` option to indicate that the messages have been removed from the queue rather than successfully processed (of course, whether this is the correct behavior for a canceled order depends on the expected message flow for your application). Notice that, as described in [Understanding Threading](understanding-threading-section.md), this method of acknowledging a message should not be used from a message handler unless the `sow_delete` is sent from a different client than the client that called the message handler. Instead, 60East recommends using the `ack()` function from within a message handler. --- # Understanding Message Objects So far, we have seen that subscribing to a topic involves working with objects of type `com.crankuptheamps.client.Message`. A `Message` represents a single message to or from an AMPS server. Messages are received or sent for every client/server operation in AMPS. ## Header Properties There are two parts of each message in AMPS: a set of headers that provide metadata for the message, and the data that the message contains. Every AMPS message has one or more header fields defined. The precise headers present depend on the type and context of the message. There are many possible fields in any given message, but only a few are used for any given message. For each header field, the `Message` class contains a distinct property that allows for retrieval and setting of that field. For example, the `Message.getCommandId()` function corresponds to the `commandId` header field, the `Message.getBatchSize()` function corresponds to the `BatchSize` header field, and so on. For more information on these header fields, consult the *AMPS User Guide* and *AMPS Command Reference*. To work with header fields, a `Message` contains `getXxx()` / `setXxx()` methods corresponding to the header fields. 60East does not recommend attempting to parse header fields from the raw data of the message. In AMPS, fields sometimes need to be set to a unique identifier value. For example, when creating a new subscription, or sending a manually constructed message, you'll need to assign a new unique identifier to multiple fields such as `CommandId` and `SubscriptionId`. For this purpose, `Message` provides `newXxx()` methods for each field that generates a new unique identifier and sets the field to that new value. ## getData() Method Access to the data section of a message is provided via the `getData()` method. The `data` contains the unparsed data in the message. The `getData()` method returns the data as a Java string, which is suitable for message formats that can be represented as Unicode text, such as JSON, XML, FIX, or NVFIX. For binary data, the AMPS Java client provides a `getDataRaw()` method to allow you to work with the underlying byte array in the message. See the [Working with Messages and Byte Buffers](./advanced-topics.md#working-with-messages-and-byte-buffers) section for details. The AMPS Java client contains a collection of helper classes for working with message types that are specific to AMPS (for example, FIX, NVFIX, and AMPS composite message types). For message types that are widely used, such as JSON or XML, you can use whichever library you typically use in your environment. ## Message Field Reference The [AMPS Command Reference](/docs/amps-command-reference) contains a full description of which fields are available and which fields are returned in response to specific commands. --- # Monitoring Connection State The AMPS client interface provides the ability to set one or more connection state listeners. A connection state listener is a callback that is invoked when the AMPS client detects a change to the connection state. A connection state listener may be called from the client receive thread. An application should not submit commands to AMPS from a connection state listener, or the application risks creating a deadlock for commands that wait for acknowledgement from the server. The AMPS client provides the following state values for a connection state listener: |State |Indicates | |------------------ |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`Connected` |

    The client has established a connection to AMPS. If you are using a `Client`, this is delivered when `connect()` is successful.

    If you are using an `HAClient`, this state indicates that the `connect` part of the connect and logon process has completed. An `HAClient` will attempt to log on immediately after delivering this state.

    Most applications that use `Client` will attempt to log on immediately after the call to `connect()` returns.

    An application should not submit commands to AMPS from the connection state listener while the client is in this state unless the application knows that the state has been delivered from a `Client` and that the `Client` does not call `logon()`.

    | |`LoggedOn` |

    The client has successfully logged on to AMPS. If you are using a `Client`, this is delivered when `logon()` is successful.

    If you are using an `HAClient`, this state indicates that the `logon` part of the connect and logon process has completed.

    This state is delivered after the client is logged on, but before recovery of client state is complete. Recovery will continue after delivering this state: the application should not submit commands to AMPS from the connection state listener while the client is in this state if further recovery will take place.

    | |`HeartbeatInitiated`|

    The client has successfully started heartbeat monitoring with AMPS. This state is delivered if the application has enabled heartbeating on the client.

    This state is delivered before recovery of the client state is complete. Recovery may continue after this state is delivered. The application should not submit commands to AMPS from the connection state listener until the client is completely recovered.

    | |`PublishReplayed` |

    Delivered when a client has completed replay of the publish store when recovering after connecting to AMPS.

    This state is delivered when the client has a PublishStore configured.

    If the client has a subscription manager set, (which is the default for an `HAClient`), the application should not submit commands from the connection state listener until the `Resubscribed` state is received.

    | |`Resubscribed` |

    Delivered when a client has re-entered subscriptions when recovering after connecting to AMPS.

    This state is delivered when the client has a subscription manager set (which is the default for an `HAClient`). This is the final recovery step. An application can submit commands to AMPS from the connection state listener after receiving this state.

    | |`Disconnected` |The client is not connected. For an `HAClient`, this means that the client will attempt to reconnect to AMPS. For a `Client`, this means that the client will invoke the disconnect handler, if one is specified.| |`Shutdown` |The client is shut down. For an `HAClient`, this means that the client will no longer attempt to reconnect to AMPS. This state is delivered when `close()` is called on the client or when a server chooser tells the `HAClient` to stop reconnecting to AMPS.| The enumeration provided for the connection state listener also includes a value of `UNKNOWN`, for use as a default or to represent additional states in a custom `Client` implementation. The 60East implementations of the client do not deliver this state. The following table shows examples of the set of states that will be delivered during connection, in order, depending on what features of the client are set. Notice that, for an instance of the `Client` class, this table assumes that the application calls both `connect()` and `logon()`. For an `HAClient`, this table assumes that the `HAClient` is using the default `DisconnectHandler` for the `HAClient`. |Configuration |States | |----------------------------------------------------------------------|------------------------------------------------------------------------------------------------------| |

    subscription manager

    publish store

    |

    `Connected`

    `LoggedOn`

    `PublishReplayed`

    `Resubscribed`

    | |

    subscription manager

    publish store

    heartbeat set

    |

    `Connected`

    `LoggedOn`

    `HeartbeatInitiated`

    `PublishReplayed`

    `Resubscribed`

    | |subscription manager |

    `Connected`

    `LoggedOn`

    `Resubscribed`

    | |

    subscription manager

    heartbeat set

    |

    `Connected`

    `LoggedOn`

    `HeartbeatInitiated`

    `Resubscribed`

    | |(default `Client` configuration) |

    `Connected`

    `LoggedOn`

    | --- # Performance Tips and Best Practices This chapter presents tips and techniques for writing high-performance applications with AMPS. This section presents principles and approaches that describe how to use the features of AMPS and the AMPS client libraries to achieve high performance and reliability. Specific techniques (for example, the details on how to write a message handler) are described in other parts of the AMPS documentation and referenced here. Other techniques require information specific to the application (for example, determining the minimum set of information required in a message), and are best done as part of your application design. All of the recommendations in this section are general guidelines. There are few, if any, universal rules for performance: at times, a design decision that is absolutely necessary to meet the requirements for an application might reduce performance somewhat. For example, your application might involve sending large binary data that cannot be incrementally updated. That application will use more bandwidth per message than an application that sends 100-byte messages with fields that can be incrementally updated. However, since the application depends on being able to deliver the binary payloads, this difference in bandwidth consumption is a part of the requirements for the application, not a design decision that can be optimized. ## Measure Performance and Set Goals The most important tools for creating high performance applications that use AMPS are clear goals and accurate measurement. Without accurate measurement, it's impossible to know whether a particular change has improved performance or not. Without clear goals, it's difficult to know whether a given result is sufficient, or whether you need to continue improving performance. 60East recommends that your measurements include baseline metrics for the part of your message processing that does not involve AMPS. As an example, imagine your task is to reduce the amount of time that elapses between when an order is sent and when the processed response is received from 100ms in total to 85ms in total. To achieve this reduction, you might first measure the processing that your application performs on the order. If that processing consumes 65ms, the most effective optimization may be to improve the order processing. On the other hand, if processing an order consumes 15ms, then optimizing message delivery or network utilization may be the most effective way to meet your goals. When measuring performance, simulate your production environment as closely as possible. For example, AMPS is highly parallelized, so sending a pattern of subscriptions and publishes from a single test client that would normally come from 20 clients will produce a very different performance profile. Likewise, AMPS can typically perform at rates that fill the available bandwidth. Performance measured on a 1GbE connection may be very different than performance measured over a 10GbE connection. Consider the characteristics of your data, and the number of messages you expect to store and process. A 1GB data set consisting of 1 million records will perform differently than a 1GB data set consisting of 10 million records, or a 1GB data set consisting of 100 records. When collecting information about performance, 60East recommends enabling persistence for the Statistics Database (`stats.db`), so you can easily collect historical data on both AMPS and the operating system. For example, a dip in performance correlated with high CPU and memory usage at the same time each day may be correlated with other activity on the system (such as cron jobs or close of business processing). In a situation like that, where the performance reduction is based on factors external to the AMPS application, the overall system metrics captured in `stats.db` can help you re-create the external state and understand the state of the system as a whole. AMPS collects the statistics in memory by default, and persisting that data into a database does not typically have a measurable effect on performance itself, but makes measuring and tuning performance much easier. For performance testing, 60East recommends using dedicated hardware for AMPS to eliminate the effects of other processes. If dedicated hardware is not available and other processes are consuming resources, 60East recommends disabling AMPS NUMA tuning to ensure that AMPS threads do not unnecessarily compete with other processes during performance tuning. ## Use HAClient and Heartbeating Where Appropriate Not every application that uses AMPS requires high availability and the ability to automatically fail over if connectivity is lost or an instance of AMPS is offline. For applications that do need automatic reconnection, 60East strongly recommends using the `HAClient` and setting heartbeating for the client to effectively detect disconnection. When using the `HAClient` and heartbeating, there are two important guidelines to follow: - Do not replace the disconnect handler on the `HAClient`. The disconnect handler is responsible for reconnection, resubscription, and so on. If you need to detect disconnection, use a connection state listener. - Set the interval for heartbeating to approximately one-half the time that the application can tolerate interruption in message flow. Notice that it's not possible for the `HAClient` to tell the difference between an interruption in message flow caused by a server going offline and interruptions caused by an increase in latency due to network saturation or so on, so the interval should be somewhat larger than the highest expected latency between AMPS and the application. Last, but not least, if the application uses asynchronous message handling, the interval should also be set to a value larger than the maximum amount of time expected for the message handler to process a single message. ## Simplify Message Format and Contents AMPS supports a wide range of message types, and is capable of filtering and processing large and complex messages. For many applications, the simplicity of being able to use messages that contain the full information is the most important consideration. For other applications, however, achieving the minimum possible latency and the maximum possible network utilization is important enough to warrant choosing a simplified message format. To simplify message contents, carefully consider the information that downstream processors require. If a downstream process will not use information in the message, there is no need to send the information. For example, consider an application that provides orders from a UI. In such an application, the object that represents the order often contains information relevant to the local state of the application that is not relevant to a downstream system. Rather than simply serializing the full object, your application may perform better if you serialize only the fields that a downstream system will take action on. To simplify message format, choose the simplest format that can convey the information that your application needs. The general principle is that the simpler the message format is, the more quickly AMPS and client libraries can parse messages of that type. Likewise, the more complicated the structure of each message is, the more work is required to parse the message. For the highest levels of performance, 60East recommends keeping the message structure simple and preferring message formats such as NVFIX, BFlat, or flattened JSON (structured as key/value pairs) as compared with more complicated formats such as XML or BSON. ## Measure Serialization and Deserialization When creating baseline performance numbers, measure serialization and deserialization performance independent of the AMPS server or client libraries. This can help you to: - Understand the baseline performance of creating and processing message data under ideal conditions (that is, where there is no application processing, networking, routing, etc. involved). - Easily compare the application-side performance of different message formats or different message layouts within a single format. When testing this performance, it is helpful to use data similar to the data that the application will actually process during a business day, at the volumes the application would typically process. This will help you understand the performance of serialization and deserialization for this specific application. For example, a library for working with a given message format might be less efficient when processing messages with a large number of string fields in a deeply-nested structure, but your application might exchange only numeric data in a relatively flat structure. Likewise, the library for a given format could be efficient for processing a small number of fields, but have lower performance for a message type with hundreds of fields. As with all performance testing, the more closely the test environment matches the actual data and volumes of a production environment, the more helpful those measurements will be for understanding system performance. ## Use Content Filtering Where Possible AMPS content filtering helps your application perform better by ensuring that your application only receives the messages that it needs. Wherever possible, we recommend using content filtering to precisely specify which messages your application needs. In particular, if at any point your application is receiving a message, parsing the message, and then determining whether to act on the message or not, 60East recommends using content filters to ensure that your application only receives messages that it needs to act on. ## Use Asynchronous Message Processing The synchronous message processing interface is straightforward, and presents a convenient interface for getting started with AMPS. However, the `MessageStream` used by the synchronous interface makes a full copy of each message and provides it from the background reader thread to the thread that consumes the message. This memory overhead and synchronization between the reader thread and consumer thread happens regardless of whether the application needs all of the header fields in the message or even processes the message. The `MessageStream` also does not take into account the speed at which your program is consuming messages, and will read messages into memory as fast as the network and processor allow. If your application cannot consume messages at wire speed, this can lead to increasing memory consumption as the application falls further behind the `MessageStream`. Most applications see improved performance by using a `MessageHandler`. With this approach, the `MessageHandler` does minimal work. If more extensive processing is needed, the `MessageHandler` dispatches the work to another thread: but it does this only when the work is necessary, and it only saves the part of the message needed to accomplish the work. ## Use Hash Indexes Where Possible for SOW Queries When querying a SOW, hash indexes on SOW topics are supported for exact matching on string data as described in the *AMPS User Guide*. A hash index can perform many times faster than a parallel query. If the query pattern for your application can take advantage of hash indexes, 60East recommends creating those hash indexes on your SOW topics. More recent versions of AMPS can use hash indexes for a wider variety of filters. When planning your queries, review the SOW queries section of the *AMPS User Guide* for the version you are using for guidelines on the optimizations available in that version. ## Use a Failed Write Handler and Exception Listener In many cases, particularly during the early stages of development, performance problems can point to defects in the application. Even after the application is tuned, monitoring for failure is important to keep applications running smoothly. 60East recommends always installing a failed write handler if your application is publishing messages. This will help you to quickly identify cases where AMPS is rejecting publishes due to entitlement failures, message type mismatches, or other similar problems. 60East recommends always installing an exception listener if your application is using asynchronous message processing. This will help you to identify and correct any problems with your message handler. An exception listener should typically log the message received and return. If recovery is needed, the listener should set a flag for another thread to process rather than attempting to recover on the thread that calls the exception listener. ## Reduce Bandwidth Requirements In many applications that use AMPS, network bandwidth is the single most important factor in overall performance. Your application can use bandwidth most efficiently by reducing message size. For example, rather than serializing an entire object, you might serialize only the fields that the remote process needs to act on, as mentioned above. Likewise, rather than sending one message that contains a collected set of information that processors will need to extract, consider sending a message in the units that processors will work with. This can reduce bandwidth to processors substantially. For example, rather than sending a single message with all of the activity for a single customer over a given period of time (such as a trading day), consider breaking out the record into the individual transactions for the customer. ### Tune Batch Size for SOW Queries As described in the section on [SOW Batch Size](/docs/amps-user-guide/sow-queries/batching-query-results), tuning the batch size for SOW queries can improve overall performance by improving network utilization. In addition, because the AMPS header is only parsed once per batch, a larger batch size can dramatically improve processing performance for smaller messages. The AMPS clients default to a batch size of `10`. This provides generally good performance for most transactional messages (such as order records or inventory records). For large messages, particularly messages greater than a megabyte in size, a batch size of `1` may reduce memory pressure in the client and improve performance. With smaller messages (for example, message sizes of a few hundred bytes), 60East recommends measuring performance with larger batch sizes such as `50` or `100`. For large messages, reducing the batch size may improve overall performance by requiring less memory consumption on the AMPS server. ### Conflate Fast-Changing Information If your data source publishes information faster than your clients need to consume it, consider using a conflated topic. For example, in a system that presents a user interface and displays fast-moving data, it is common for the data to change at a rate faster than the user interface can format and render the data. In this case, a conflated topic can both reduce bandwidth and simplify processing in the user interface. ### Minimize Bandwidth for Updates If your application uses a SOW and processes frequent updates, consider using delta publish and delta subscribe to reduce the size of the messages transmitted. These features are designed to minimize bandwidth while still providing full-fidelity data streams. ### Conflate Queue Acknowledgments The AMPS clients include the ability to conflate acknowledgments back to AMPS as queue messages are processed. Using these features, with an appropriate `max_backlog`, can reduce the amount of network traffic required for acknowledgments. ### Use a Transaction Log When Monitoring Publish Failures When a topic is not covered by a transaction log, AMPS returns acknowledgment messages for every publish that requests one. This ensures that each message is acknowledged, even when AMPS has no persistent record of the messages in the topic. However, acknowledging each message requires more network traffic for each publish message. When a topic is covered by a transaction log, AMPS conflates persisted acknowledgments. Conflation is possible in this case because AMPS has a full record of the messages and does not have to store additional state to conflate the acknowledgments. With conflated acknowledgments, AMPS will send a success acknowledgment periodically that covers all messages up to that point. If a message fails, AMPS immediately sends the conflated success acknowledgment for all previous messages and the failure acknowledgment for the failed message. ### Combine Conflation and Deltas In many cases, using an approach that combines delta publishes to a SOW with delta subscriptions to a conflated topic can dramatically reduce bandwidth to the application with no loss of information. ## Limit Unnecessary Copies One of the most effective ways to increase performance is to limit the amount of data copied within your application. For example, if your message handler submits work to a set of processors that only use the `Data` and `Bookmark` from a `Message`, create a data structure that holds only those fields and copy that information into instances of that data structure rather than copying the entire `Message`. While this approach requires a few extra lines of code, the performance benefits can be substantial. When publishing messages to AMPS, avoid unnecessary copies of the data. For example, if you have the data in a byte array, use the `publish` methods that use a byte array rather than converting the data to a string unnecessarily. Likewise, if you have the data in the form of a string, avoid converting it to a byte array where possible. ## Manage Publish Stores When using a publish store, the Client holds messages until they are acknowledged as persisted by AMPS, as determined by the replication configuration for the AMPS instance. In the event that an instance with `sync` replication goes offline, the publish store for the Client will grow, since the messages are not being fully persisted. To avoid this problem, 60East recommends that an instance that uses `sync` replication always configure Actions to automatically downgrade the replication link if the remote instance goes offline for a period of time, and upgrade the link when the remote instance comes back online. Further, 60East recommends that, where possible, a publisher is provisioned with enough storage to hold its complete publish stream for the amount of time that a destination may be offline or unavailable without downgrading from `sync` replication to `async` replication. For example, if the server considers a downstream system to be unreachable if it has not acknowledged a replicated message in 60 seconds, and the server checks this threshold every 10 seconds, then a publisher should plan that, at any time, the publisher may need to retain approximately 70 seconds worth of published messages. This is calculated as the 60 seconds threshold that the server has established for a destination to run behind, plus the 10 second interval at which the server checks whether the destination is within the threshold. Also notice that, with a configuration like this, a downstream replication destination could run as much as 59 seconds behind indefinitely. A publisher should be provisioned to be able to run effectively in a "worst case" (or nearly "worst case") scenario for an extended period of time. See the *High Availability and Replication* chapter in the *AMPS User Guide* for more information on replication, sync and async acknowledgment modes, and the Actions used to manage replication. ## Use the Server Logs to Help Troubleshoot When troubleshooting problems with an application that uses AMPS, the server-side logs often provide the most helpful information. For example, `trace` level logging shows the data that is flowing through AMPS. Log messages at `info` level show events as incoming connections, commands from clients, and so on. When questions arise about how the server and application interact, the server logs often contain the information. 60East recommends that an AMPS instance used for development and testing log at `trace` level, and that a server used for production log at `info` level, with the ability to log at `trace` level when necessary for investigating any problems that may arise. When a command does not have the expected result, or an application reports an error, the fastest way to understand the problem is often to review the `trace` level logging for the instance. See the *AMPS User Guide* for details on configuring logging and common patterns for searching for information in AMPS logs. ## Work with 60East as Necessary 60East offers performance advice adapted for your specific usage through your support agreement. Once you've set your performance goals, worked through the general best practices and applied the practices that make sense for your application, 60East can help with detailed performance tuning, including recommendations that are specific to your use case and performance needs. --- # Samples of Using A Queue The AMPS Java client includes the following samples that demonstrate how to publish messages to a queue and how to consume messages from a queue. |Sample Name |Demonstrates | |-----------------------------------|--------------------------------------| |`QueuePublisher.java`|Publishing messages to a queue topic.| |`QueueSubscriber.java`|Consuming messages from a queue topic.| --- # Using Queues AMPS message queues provide a high-performance way of distributing messages across a set of workers. The *AMPS User Guide* describes AMPS [Queues](/docs/amps-user-guide/queues) in detail, including the features of AMPS referred to in this chapter. This chapter does not describe message queues in detail, but instead explains how to use the AMPS Java client with message queues. To publish messages to a message queue, publishers simply publish to any topic that is collected by the queue. There is no difference between publishing to a queue and publishing to any other topic, and a publisher does not need to be aware that the topic will be collected into a queue. Subscribers must be aware that they are subscribing to a queue, and acknowledge messages from the queue when the message is processed. --- # Regular Expression Subscriptions Regular Expression (Regex) subscriptions allow a regular expression to be supplied in the place of a topic name. When you supply a regular expression, it is as if a subscription is made to every topic that matches your expression, including topics that do not yet exist at the time of creating the subscription. To use a regular expression, simply supply the regular expression in place of the topic name in the `subscribe()` call. For example: ```java showLineNumbers // Provided for reference public class MessagePrinter implements MessageHandler { public void invoke(Message m) { System.out.println(m.getTopic() + ": " + m.getData()); } } class MyApp { public static void main(String[] args) { // Assumes client is a connected and logged on // Client or HAClient. ... CommandId id = client.subscribe( new MessagePrinter, // MessageHandler implementation "orders.*", // Topic with a Regex 5000); // Timeout ... } } ``` In this example, messages on topics `orders-north-america`, `orders-europe`, and `new-orders` would match the regular expression. Messages published to any of those topics will be sent to the `MessagePrinter` passed to the subscribe command. As in the example, you can use the `getTopic()` method to determine the actual topic of the message sent to the function. --- # Returning a Message to the Queue A subscriber can also explicitly release a message back to the queue. AMPS returns the message to the queue, and redelivers the message just as though the lease had expired. To do this, the subscriber sends a `sow_delete` command with the bookmark of the message to release and the `cancel` option. When using automatic acknowledgments and the asynchronous API, AMPS will cancel a message if an exception is thrown from the message handler. To return a message to the queue, you can build a `sow_delete` acknowledgment using the `Command` class, or pass an option to the `ack()` method on the message. |Option |Result | |-----------------------|-----------------------------------------------| |`cancel` |Returns the message to the queue. | |`expire` |Immediately expires the message from the queue.| For example, to return a message to a queue, call `ack()` on the message and pass the `cancel` option. ```java message.ack("cancel"); ``` --- ### Samples of Querying and Subscribing to a Topic in the SOW The Java client distribution includes the following samples that demonstrate how to query and subscribe to a topic in the SOW. |Sample Name |Demonstrates | |----------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`SOWandSubscribeConsoleSubscriber.java`|Querying a SOW topic and entering a subscription to the topic in a single atomic operation.| |`SowAndSubscribeWithOOF.java` |Querying a SOW topic and entering a subscription to the topic in a single atomic operation, while registering for notification that a previously-matching message is no longer a match for the subscription.| |`SOWUpdater.java` |Updating records in a SOW topic. | --- # Setting Batch Size The AMPS clients include a batch size parameter that specifies how many messages the AMPS server will return to the client in a single batch when returning the results of a SOW query. The 60East clients set a batch size of 10 by default. This batch size works well for common message sizes and network configurations. Adjusting the batch size may produce better network utilization and produce better performance overall for the application. The larger the batch size, the more messages AMPS will send to the network layer at a time. This can result in fewer packets being sent, and therefore less overhead in the network layer. The effect on performance is generally most noticeable for small messages, where setting a larger batch size will allow several messages to fit into a single packet. For larger messages, a batch size may still improve performance, but the improvement is less noticeable. In general, 60East recommends setting a batch size that is large enough to produce few partially-filled packets. Bear in mind that AMPS holds the messages in memory while batching them, and the client must also hold the messages in memory while receiving the messages. Using batch sizes that require large amounts of memory for these operations can reduce overall application performance, even if network utilization is good. For smaller message sizes, 60East recommends using the default batch size, and experimenting with tuning the batch size if performance improvements are necessary. For relatively large messages (especially messages with sizes over 1MB), 60East recommends explicitly setting a batch size of 1 as an initial value, and increasing the batch size only if performance testing with a larger batch size shows improved network utilization or faster overall performance. --- # SOW and Subscribe Imagine an application that displays real time information about the position and status of a fleet of delivery vans. When the application starts, it should display the current location of each of the vans along with their current status. As vans move around the city and post other status updates, the application should keep its display up to date. Vans upload information to the system by posting messages to the `van_location` topic, configured with a key of `van_id` on the AMPS server. In this application, it is important to not only stay up-to-date on the latest information about each van, but also to ensure all of the active vans are displayed as soon as the application starts. Combining a SOW with a subscription to the topic is exactly what is needed, and that is accomplished by the AMPS `sow_and_subscribe` command. Now we will look at an example: ```java showLineNumbers private void updateVanPosition(Message message) { switch (message.getCommand()) { case Message.Command.SOW: case Message.Command.Publish: /* For each of these messages, addOrUpdateVan() presumably adds the van * to our application's display. As vans send updates to the AMPS server, * those are also received by the client because of the subscription * placed by sowAndSubscribe(). Our application does not need to distinguish * between updates and the original set of vans we found via the SOW * query, so we use addOrUpdateVan() to display the new position of vans * as well. */ addOrUpdateVan(message); break; case Message.Command.OOF: removeVan(message); break; } } public void subscribeToVanLocation(Client client) { try { Command command = new Command("sow_and_subscribe") .setTopic("van_location") .setFilter("/status = 'ACTIVE'") .setBatchSize(100) .setOptions("oof"); /* We issue a sowAndSubscribe() to begin receiving information about all of * the active delivery vans in the system. All of the vans in the system * now are returned as Messages whose getCommand() returns SOW. */ for (Message message : client.execute(command)) { updateVanPosition(message); } } catch (AMPSException aex) { System.err.println("TestListener caught exception."); } } public void addOrUpdateVan(message) { // Use information in the message to add the van or update // the van position. ... } public void removeVan(message) { // Use information in the message to remove information on // the van position. ... } ``` Now we will look at an example that uses the asynchronous form of `sowAndSubscribe`: ```java showLineNumbers public class VanPositionUpdater { public void invoke(Message message) { updateVanPosition(message); } private void updateVanPosition(Message message) { switch (message.getCommand()) { case Message.Command.SOW: case Message.Command.Publish: addOrUpdateVan(message); break; case Message.Command.OOF: removeVan(message); break; } } public void addOrUpdateVan(message) { // Use information in the message to add the van or update // the van position. ... } public void removeVan(message) { // Use information in the message to remove information on // the van position. ... } } public void subscribeToVanLocation(Client client) { try { VanPositionUpdater vp = new VanPositionUpdater(); Command command = new Command("sow_and_subscribe") .setTopic("van_location") .setFilter("/status = 'ACTIVE'") .setBatchSize(100) .setOptions("oof_enabled"); client.execute(command, vp); } catch (AMPSException aex) { System.err.println("TestListener caught exception."); } } ``` --- # State of the World (SOW) AMPS State of the World (SOW) allows you to automatically keep and query the latest information about a topic on the AMPS server, without building a separate database. Using SOW lets you build impressively high-performance applications that provide rich experiences to users. The AMPS Java client lets you query SOW topics and subscribe to changes with ease. ## Performing SOW Queries To begin, we will look at a simple example of issuing a SOW query. ```java showLineNumbers ... public void executeSOWQuery(Client client) { for (Message m : client.sow("orders", "/symbol = 'ROL'")) { if (m.getCommand() == Message.Command.GroupBegin) { System.out.println("--- Begin SOW Results ---"); } if (m.getCommand() == Message.Command.GroupEnd) { System.out.println("--- End SOW Results ---"); } if (m.getCommand() == Message.Command.SOW) { System.out.println(m.getData()); } } } ... ``` In the example above, the `executeSOWQuery()` method invokes `Client.sow()` to initiate a SOW query on the `orders` topic, for all entries that have a symbol of `'ROL'`. As the query executes, the body of the loop processes each matching entry in the topic. Messages containing the data of matching entries have a `Command` of value `sow`; so as those arrive, we write them to the console. AMPS sends a `group_begin` message at the beginning of the results and an `group_end` message at the end of the results. We use those messages to delimit the results of the query. As with subscribe, the SOW command also provides an asynchronous version, as well as versions that accept a `Command`. For example, the listing below shows an asynchronous SOW command that specifies the *batch size*, or the maximum number of records that AMPS will return at a time. ```java showLineNumbers public void executeSOWQuery(Client client) { Command command = new Command(Message.Command.SOW) .setTopic("orders") .setFilter("/symbol = 'ROL'") .setBatchSize(100); client.executeAsync(command, new MessagePrinter()); } ... public class MessagePrinter implements MessageHandler { public void invoke(Message m) { if (m.getCommand() == Message.Command.SOW) { System.out.println(m.getData()); } } } ``` ### Samples of Querying a Topic in the SOW The Java client distribution includes the following samples that demonstrate how to query a topic in the SOW. |Sample Name |Demonstrates | |----------------------------------------------|-----------------------------------| |`SOWConsolePublisher.java` |Publishing messages to a SOW topic. | |`SOWConsoleSubscriber.java`|Querying messages from a SOW topic. | --- # Subscriptions Messages published to a topic on an AMPS server are available to other clients via a subscription. Before messages can be received, a client must subscribe to one or more topics on the AMPS server so that the server will begin sending messages to the client. The server will continue sending messages to the client until the client unsubscribes, or the client disconnects. With content filtering, the AMPS server will limit the messages sent to only those messages that match a client-supplied filter. In this chapter, you will learn how to subscribe, unsubscribe, and supply filters for messages using the AMPS Java client. ## Subscribing The AMPS client makes it simple to subscribe to a topic. You call `Client.subscribe()` with the topic to subscribe to and the parameters for the subscription. The client submits a subscription to AMPS and returns a `MessageStream` that you can iterate over to receive the messages from the subscription. Below is a short example (error handling and connection details are omitted for brevity): ```java showLineNumbers class MyApp { public static void main(String[] args) { // We create a Client, then connect() and logon(). Client client = new Client(...); try { client.connect(...); client.logon(); /* Here, we subscribe to the topic "messages". * We do not provide a * filter, so the * subscription receives all of the messages * published to the topic, regardless of content. * * We protect the MessageStream in a try with * resources block, so that the stream is closed * (and the subscription ends) when control * exits the block. */ try (MessageStream ms = client.subscribe("messages")) { /* Here, we iterate over the messages returned by the * MessageStream. When we no longer need to subscribe, we can * break out of the loop. When the MessageStream is cleaned up, * the client sends an unsubscribe command to AMPS and stops * receiving messages. */ for (Message m : ms) { /* Within the loop, we process the message. In this case, we * simply print the contents of the message */ System.out.println(m.getData()); } } } catch(AMPSException e){System.err.println(e);} finally { client.close(); } } } ``` AMPS creates a background thread that receives messages and copies them into the `MessageStream` that you iterate over. This means that the client application as a whole can continue to receive messages while you are doing processing work. The simple method described above is provided for convenience. The AMPS Java client provides convenience methods for the most common forms of the commands. AMPS also provides an interface that allows you precise control over the command. Using that interface, the example above becomes: ```java showLineNumbers class MyApp { public static void main(String[] args) { // We create a Client that is properly connected to an AMPS server. Client client = new Client("subscribe"); try { client.connect("tcp://127.0.0.1:9007/amps"); client.logon(); // We create a Command object to subscribe to the messages topic. Command command = new Command("subscribe").setTopic("messages"); /* Here we execute the command and subscribe to the topic messages. * This works exactly the same way as the command in Example 1. If, * at any time, we no longer need to subscribe, we can break out of * the loop. We use a try with resources to automatically clean up * the MessageStream when we leave the try block. When the * MessageStream is cleaned up, the client sends an unsubscribe * command to AMPS and stops receiving messages. */ try (MessageStream ms = client.execute(command)) for (Message m : ms) { /* Within the loop, we process the message. In this case, we * simply print the contents of the message */ System.out.println(m.getData()); } } catch(AMPSException e){;} finally { client.close(); } } } ``` The `Command` interface allows you to precisely customize the commands you send to AMPS. For flexibility and ease of maintenance, 60East recommends using the `Command` interface (rather than a named method) for any command that will receive messages from AMPS. For publishing messages, there can be a slight performance advantage to using the named commands where possible. --- # Synchronous Message Processing As mentioned [earlier](subscriptions.md), one way for an application to receive messages is to have the AMPS Java client return a `MessageStream` object that can be used to iterate over the results of the command. The `MessageStream` object makes copies of the incoming messages. When there is no message available, the `MessageStream` will block. A `MessageStream` will only remain active while the client that produced it is connected. If the client disconnects, the `MessageStream` will continue to provide any messages that have not yet been consumed, then throw an exception. The advantages of using a `MessageStream` that it provides a simple processing model, that receiving messages from a `MessageStream` does not block the client receive thread (see [Understanding Threading](understanding-threading-section.md) ) and that a copy of the message is automatically made for the application. In return for these advantages, a `MessageStream` has higher overhead than [Asynchronous Message Processing](async-message-processing.md), it will not be resumed if the client disconnects, and, by default, it will use as much memory as necessary to hold messages coming from the AMPS server. --- # Understanding Threading The first time a command causes an instance of the `Client` or `HAClient` to connect to AMPS (typically, the `logon()` command), the client creates a thread that runs in the background. This background thread is responsible for processing incoming messages from AMPS, which includes both messages that contain data and acknowledgments from the server. When you call a command on the AMPS client, the command typically waits for an acknowledgment from the server and then returns. (The exception to this is `publish`. For performance, the `publish` command does not wait for an acknowledgment from the server before returning.) In the simple case, using synchronous message processing, the client provides an internal handler function that populates the `MessageStream`. The client receive thread calls the internal handler function, which makes a deep copy of the incoming message and adds it to the `MessageStream`. The `MessageStream` is used on the calling thread, so operations on the `MessageStream` do not block the client receive thread. When using asynchronous message processing, AMPS calls the handler function from the client receive thread. Message handlers provided for *asynchronous* message processing must be aware of the following considerations: - The client creates one client receive thread at a time, and the lifetime of the thread lasts for the lifetime of the connection to the AMPS server. A message handler that is only provided to a single client will only be called from a single thread at a time. If your message handler will be used by multiple clients, then multiple threads will call your message handler. In this case, you should take care to protect any state that will be shared between threads. Notice that if the client connection fails (or is closed), and the client reconnects, the client will create a different thread for the new connection. - For maximum performance, do as little work in the message handler as possible. For example, if you use the contents of the message to update an external database, a message handler that adds the relevant data to an update queue, that is processed by a different thread, will typically perform better than a message handler that does this update during the message handling. - While your message handler is running, the thread that calls your message handler is no longer receiving messages. This makes it easier to write a message handler because you know that no other messages are arriving from the same subscription. However, this also means that you cannot use the same client that called the message handler to send commands to AMPS. Acknowledgments from AMPS cannot be processed and your application will deadlock waiting for the acknowledgment. Instead, enqueue the command in a work queue to be processed by a separate thread or use a different client object to submit the commands. - The AMPS client resets and reuses the `Message` provided to this function between calls. This improves performance in the client, but means that if your handler function needs to preserve information contained within the message, you must copy the information (either by making a copy of the entire message or copying the required fields) rather than just saving the message object. Otherwise, the AMPS client cannot guarantee the state of the object or the contents of the object when your program goes to use it. Likewise, a message handler should not modify the `Message` -- this will result in modifying the message provided to other handlers (including handlers internal to the AMPS client). --- # Unexpected Messages The AMPS Java client handles most incoming messages and takes appropriate action. Some messages are unexpected or occur only in very rare circumstances. The AMPS Java client provides a way for clients to process these messages. Rather than providing handlers for all of these unusual events, AMPS provides a single handler function for messages that can't be handled during normal processing. Your application registers this handler by setting the `lastChanceMessageHandler` for the client. This handler is called when the client receives a message that can't be processed by any other handler. This is a rare event, and typically indicates an unexpected condition. For example, if a client publishes a message that AMPS cannot parse, AMPS returns a failure acknowledgment. This is an unexpected event, so AMPS does not include an explicit handler for this event, and failure acknowledgments are received in the method registered as the `lastChanceMessageHandler`. Your application is responsible for taking any corrective action needed. For example, if a message publication fails, your application can decide to republish the message, publish a compensating message, log the error, stop publication altogether, or any other action that is appropriate. --- # Unhandled Exceptions In the AMPS Java client, exceptions can occur that are not thrown to the user. For example, when an exception occurs in the process of reading subscription data from the AMPS server, the exception occurs on a thread inside of AMPS. Consider the following example: ```java showLineNumbers public class MyApp { ... public static void waitToBePoked(Client client) { Command command = new Command( "subscribe" ).setTopic( "pokes" ).setFilter( "/Pokee LIKE'" + System.getProperty("user.name") + "-.*" ).setTimeout(5000); client.execute(command, new MsgPrinter()); Console c = System.console(); Reader r = c.reader(); while(r.read() == null) { Thread.sleep(10); } } class MsgPrinter implements MessageHandler { public void invoke(Message m) { System.out.println(m.getData()); } } } ``` In this example, we set up a simple subscription to wait for messages on the “pokes” topic, whose “Pokee” tag begins with our user name. When messages arrive, we print a message out to the console, but otherwise our application waits for a key to be pressed. Inside of the AMPS client, the client creates a new thread of execution that reads data from the server, and invokes message handlers and disconnect handlers when those events occur. When exceptions occur inside this thread, however, there is no caller for them to be thrown to, and by default they are ignored. In applications where it is important to deal with every issue that occurs in using AMPS, you can set an `ExceptionListener` via `Client.setExceptionListener()` that receives these otherwise unhandled exceptions. Making the modifications shown in the example below, to our previous example, will allow those exceptions to be caught and handled. In this case we are simply printing those caught exceptions out to the console. ```java showLineNumbers public class MyApp { ... client.setExceptionListener(new CustomExceptionListener()); ... } class CustomExceptionListener implements ExceptionListener { public void exceptionThrown(Exception ex) { System.out.println(ex.toString()); } } ``` In this example we have added a call to `setExceptionListener()`, registering a simple function that writes the text of the exception out to the console. Even though our application waits for a user to press a key, messages to the console will still be produced, both as incoming “poke” messages arrive, and as issues arise inside of AMPS. If your application will attempt to recover from an exception thrown on the background processing thread, your application should set a flag and attempt recovery on a *different* thread than the thread that called the exception listener. :::tip At the point that the AMPS client calls the exception listener, it has handled the exception. Your exception listener must not rethrow the exception (or wrap the exception and throw a different exception type). ::: --- # Ending Subscriptions The AMPS server continues a subscription until the client explicitly ends the subscription (that is, *unsubscribes*) or until the connection to the client is closed. With the synchronous interface, AMPS automatically unsubscribes to the topic when the destructor for the `MessageStream` runs. You can also explicitly call the `close()` method on the `MessageStream` object to remove the subscription. In the asynchronous interface, when a subscription is successfully made, messages will begin flowing to the message handler, and the `subscribe()` or `executeAsync()` call will return a string for the subscription id that serves as the identifier for this subscription. A `Client` can have any number of active subscriptions, and this subscription id is how AMPS designates messages intended for this particular subscription. To unsubscribe, we simply call `unsubscribe` with the subscription identifier: ```java showLineNumbers Client c = ...; // try/catch block to manage client lifetime // is left out // ... subscribe using the asynchronous message // processing interface and save the subId CommandId subId = c.subscribe(new MyMessageHandler(), "messages"); // ... other code here ... c.unsubscribe(subId); ``` In this example we use the `client.subscribe()` method to create a subscription to the `messages` topic. The subscribe method returns an identifier for the subscription created in AMPS. When our application is done listening to this topic, it unsubscribes by passing in the `subId` returned by `subscribe()`. AMPS deletes the subscription. After the subscription is removed, no more messages will flow into the `MyMessageHandler()` instance for that subscription. When an application calls `unsubscribe()`, the client sends an explicit `unsubscribe` command to AMPS. The AMPS server removes that subscription from the set of subscriptions for the client, and stops sending messages for that subscription. On the client side, the client unregisters the subscription so that the `MessageStream` or `MessageHandler` for that subscription will no longer receive messages for that subscription. Notice that calling `unsubscribe` does not destroy messages that the server has already sent to the client. If there are messages on the way to the client for this subscription, the AMPS client must consume those messages. If a `LastChanceMessageHandler` is registered, the handler may receive the messages. Otherwise, they will be discarded since no message handler matches the subscription ID on the message. --- # Utility Classes The AMPS Java client includes a set of utilities and helper classes to make working with AMPS easier. ## Composite Message Types The client provides a pair of classes for creating and parsing composite message types: - `CompositeMessageBuilder` allows you to assemble the parts of a composite message and then serialize them in a format suitable for AMPS. - `CompositeMessageParser` extracts the individual parts of a composite message type. For more information regarding composite message types, refer to the *Message Types* chapter in the *AMPS User Guide*. ### Building Composite Messages To build a composite message, create an instance of `CompositeMessageBuilder`, and populate the parts. The `CompositeMessageBuilder` copies the parts provided, in order, to the underlying message. The builder simply writes to an internal buffer with the appropriate formatting, and does not allow you to update or change the individual parts of a message once they've been added to the builder. The snippet below shows how to build a composite message that includes a JSON part, constructed as a string, and a binary part consisting of the bytes from an `ArrayList`. ```java showLineNumbers StringBuilder sb = new StringBuilder(); sb.append("{\"data\":\"sample\"}"); List theData = new ArrayList(); // Populate theData ... // Create a byte array from the data: this is // what the program will send. ByteArrayOutputStream outBytes = new ByteArrayOutputStream(); ObjectOutputStream listWriter = new ObjectOutputStream(outBytes); listWriter.writeObject(theData); // Create the payload for the composite message. CompositeMessageBuilder builder; // Construct the composite CompositeMessageBuilder builder = new CompositeMessageBuilder(); builder.append(sb.toString()); builder.append(outBytes.toByteArray(), 0, outBytes.size()); // Send the message String topic = "messages"; Field outMessage = new Field(); builder.setField(outMessage); client.publish(topic.getBytes(), 0, topic.getBytes().length, outMessage.buffer, 0, outMessage.length); ``` ### Parsing Composite Messages To parse a composite message, create an instance of `CompositeMessageParser`, then use the `parse()` method to parse the message provided by the AMPS client. The `CompositeMessageParser` gives you access to each part of the message as a sequence of bytes. For example, the following snippet parses and prints messages that contain a JSON part and a binary part that contains an array of doubles. ```java showLineNumbers try (MessageStream stream = client.subscribe("messages")) { for (Message message : stream) { int parts = parser.parse(message); String json = parser.getString(0); Field binary = new Field(); parser.getField(1, binary); ByteArrayInputStream inBytes = new ByteArrayInputStream(binary.buffer, binary.position, binary.length); ObjectInputStream listReader = new ObjectInputStream(inBytes); List theData = (List)listReader.readObject(); System.out.println("Received message with " + parts + " parts."); System.out.println(json); for (Double d : theData) { System.out.print(d + " "); } System.out.print("\n"); } } ``` Notice that the receiving application is written with explicit knowledge of the structure and content of the composite message type. ## NVFIX Messages The client provides a pair of classes for creating and parsing NVFIX messages: - `NVFIXBuilder` allows you to assemble an NVFIX message and then serialize it in a format suitable for AMPS. - `NVFIXShredder` extracts the individual fields of an NVFIX message. ### Building NVFIX Messages To build an NVFIX message, create an instance of `NVFIXBuilder`, then add the fields of the message using `append()`. `NVFIXBuilder` copies the fields provided, in order, to the underlying message. The builder simply writes to an internal buffer with the appropriate formatting, and does not allow you to update or change the individual fields of a message once they've been added to the builder. The snippet below shows how to build an NVFIX message and publish it to the AMPS client. ```java showLineNumbers // Build the message payload. // Create a builder with 1024 bytes of initial capacity, // using the default 0x01 delimiter. NVFIXBuilder builder = new NVFIXBuilder(1024, (byte)1); // Add fields builder.append("test-field", "24"); builder.append("another", "Here's another field"); builder.append("data", "1234567890"); // Create a string for the topic. String topic = "test-topic"; client.connect(uri_); System.out.println("connected.."); client.logon(); // Publish the message, using the overload that takes a byte array and // length for the topic and payload. client.publish(topic.getBytes(), 0, topic.length(), builder.getBytes(), 0, builder.getSize()); ``` ### Parsing NVFIX Messages To parse an NVFIX message, create an instance of `NVFIXShredder`, then use the `toNVMap()` method to parse the message provided by the AMPS client. The `NVFIXShredder` gives you access to each field of the message in a map. The snippet below shows how to parse and print an NVFIX message. ```java showLineNumbers Client client = new Client("ConsoleSubscriber"); try { // Connect to the AMPS server and logon. client.connect(uri_); // Subscribe to the test-topic topic. // when a message arrives, print the message. MessageStream ms = client.subscribe("test-topic"); try { // Create a shredder -- since this just returns // the Map, we can reuse the same shredder. NVFIXShredder shredder = new NVFIXShredder((byte)1); for (Message m : ms) { // Skip messages with no data. if (m.getCommand() != Message.Command.SOW && m.getCommand() != Message.Command.Publish) continue; System.out.println("Got a message"); // Shred the message into a Map. Map fields = shredder.toNVMap(m.getData()); // Iterate over the keys in the map and print the key and data. for (CharSequence key : fields.keySet()) { System.out.println(" " +key + "=" + fields.get(key)); } } } finally { // Close the message stream to release the subscription. ms.close(); } } ``` ## FIX Messages The client provides a pair of classes for creating and parsing FIX messages: - `FIXBuilder` allows you to assemble a FIX message and then serialize it in a format suitable for AMPS. - `FIXShredder` extracts the individual fields of a FIX message. ### Building FIX Messages To build a FIX message, create an instance of `FIXBuilder`, then add the fields of the message using `append()`. `FIXBuilder` copies the fields provided, in order, to the underlying message. The builder simply writes to an internal buffer with the appropriate formatting, and does not allow you to update or change the individual fields of a message once they've been added to the builder. The snippet below shows how to build a FIX message and publish it to the AMPS client. ```java showLineNumbers // Build the message payload. // Create a builder with 1024 bytes of initial capacity, // using the default 0x01 delimiter. FIXBuilder builder = new FIXBuilder(1024, (byte)1); // Add fields builder.append(91290, "24"); builder.append(42, "Here's another field"); builder.append(8675309, "1234567890"); // Create a string for the topic. String topic = "test-topic"; client.connect(uri_); System.out.println("connected.."); client.logon(); // Publish the message, using the overload that takes a byte array and // length for the topic and payload. client.publish(topic.getBytes(), 0, topic.length(), builder.getBytes(), 0, builder.getSize()); ``` ### Parsing FIX Messages To parse a FIX message, create an instance of `FIXShredder`, then use the `toMap()` method to parse the message provided by the AMPS client. The `FIXShredder` gives you access to each field of the message in a map. The snippet below shows how to parse and print a FIX message. ```java showLineNumbers Client client = new Client("ConsoleSubscriber"); try { // Connect to the AMPS server and logon. client.connect(uri_); // Subscribe to the test-topic topic. // When a message arrives, print the message. MessageStream ms = client.subscribe("test-topic"); try { // Create a shredder -- since this just returns // the Map, we can reuse the same shredder. FIXShredder shredder = new FIXShredder((byte)1); for (Message m : ms) { // Skip messages with no data. if (m.getCommand() != Message.Command.SOW && m.getCommand() != Message.Command.Publish) continue; System.out.println("Got a message"); // Shred the message into a Map. Map fields = shredder.toMap(m.getData()); // Iterate over the keys in the map and print the key and data. for (Integer key : fields.keySet()) { System.out.println(" " +key + "=" + fields.get(key)); } } } } ``` --- # Welcome to the AMPS JavaScript Client This guide provides information you need to get started with the AMPS JavaScript client. It focuses specifically on the client and does not cover AMPS itself in detail. For an overview of AMPS and instructions on setting up your development environment, see the [Introduction to AMPS](/docs/intro-guide/intro) guide. :::tip This guide assumes that you have a development environment for JavaScript and access to an AMPS server using the configuration provided with the JavaScript samples (in the full source distribution of the client). ::: --- # Acknowledging Messages For each message delivered on a subscription, AMPS counts the message against the subscription backlog until the message is explicitly acknowledged. In addition, when a queue specifies `at-least-once` delivery, AMPS retains the message in the queue until the message expires or until the message has been explicitly acknowledged and removed from the queue. From the point of view of the AMPS server, acknowledgment is implemented as a `sow_delete` from the queue with the bookmarks of the messages to remove. The AMPS JavaScript client provides several ways to make it easier for applications to create and send the appropriate `sow_delete`. ## Automatic Acknowledgment The AMPS client allows you to specify that messages should be automatically acknowledged. When this mode is on, AMPS acknowledges the message automatically if the message handler returns without throwing an exception. AMPS batches acknowledgments created with this method, as described in the following section. To enable automatic acknowledgment, use the `Client.autoAck()` method. ```javascript client.autoAck(true) // enable AutoAck ``` ## Message Convenience Method The AMPS JavaScript client provides a convenience method, `Client.ack()`, on delivered messages. When the application is finished with the message, the application simply calls `Client.ack()` on the message. For messages that originated from a queue with `at-least-once` semantics, this adds the bookmark from the message to the batch of messages to acknowledge. For other messages, this method has no effect. ```javascript // Add the message to the next acknowledgment batch client.ack(message) ``` --- # Acknowledgment Batching The AMPS JavaScript client automatically batches acknowledgments when either of the convenience methods is used. Batching acknowledgments reduces the number of round-trips to AMPS, which reduces network traffic and improves overall performance. AMPS sends the batch of acknowledgments when the number of acknowledgments exceeds a specified size, or when the amount of time since the last batch was sent exceeds a specified timeout. You can set the number of messages to batch and the maximum amount of time between batches, as shown below: ```javascript client.ackBatchSize(10) // Send batch after 10 messages client.ackTimeout(1000) // ... or 1 second ``` The AMPS JavaScript client is aware of the subscription backlog for a subscription. When AMPS returns the acknowledgment for a subscription that contains queues, AMPS includes information on the subscription backlog for the subscription. If the requested batch size is larger than the subscription backlog, the AMPS JavaScript client adjusts the requested batch size to match the subscription backlog. --- # Advanced Topics ## Transport Filtering The AMPS JavaScript client offers the ability to filter incoming and outgoing messages in the format they are sent and received on the network. This allows you to inspect or modify outgoing messages before they are sent to the network, and incoming messages as they arrive from the network. This can be especially useful when using SSL connections, since this gives you a way to monitor outgoing network traffic before it is encrypted, and incoming network traffic after it is decrypted. To create a transport filter, you create a callable that expects a string that contains the raw data, and a direction parameter indicating whether the string is output or not. For example, the following function simply prints the direction and data: ```javascript showLineNumbers const printingFilter = (data, outgoing) => { if (outgoing) { console.log('OUTGOING ---> ', data) } else { console.log('INCOMING ---> ', data.data) } } ``` You then register the filter by calling `Client.transportFilter()` with the function, as shown below. ```javascript showLineNumbers // client is an AMPS client client.transportFilter(printingFilter) ``` ## Using WebSocket over SSL The AMPS JavaScript client includes support for Secure Sockets Layer. To use this support in the JavaScript client, you need only use `wss` for the transport type in the connection string, as described in the section on [Connection Strings for AMPS](./connection-strings.md) in this guide. --- # Backlog and Smart Pipelining AMPS queues are designed for high-volume applications that need minimal latency and overhead. One of the features that helps performance is the *subscription backlog* feature, which allows applications to receive multiple messages at a time. The subscription backlog sets the maximum number of unacknowledged messages that AMPS will provide to the subscription. When the subscription backlog is larger than `1`, AMPS delivers additional messages to a subscriber before the subscriber has acknowledged the first message received. This technique allows subscribers to process messages as fast as possible, without ever having to wait for messages to be delivered. The technique of providing a consistent flow of messages to the application is called *smart pipelining*. ## Subscription Backlog The AMPS server determines the backlog for each subscription. An application can set the maximum backlog that it is willing to accept with the `max_backlog` option. Depending on the configuration of the queue (or queues) specified in the subscription, AMPS may assign a smaller backlog to the subscription. If no `max_backlog` option is specified, AMPS uses a `max_backlog` of `1` for that subscription. In general, applications that have a constant flow of messages perform better with a `max_backlog` setting higher than `1`. The reason for this is that, with a backlog greater than `1`, the application can always have a message waiting when the previous message is processed. Setting the optimum `max_backlog` is a matter of understanding the messaging pattern of your application and how quickly your application can process messages. To request a `max_backlog` for a subscription, you explicitly set the option on the subscribe command, as shown below: ```javascript const command = new Command('subscribe') .topic('my-queue') .options('max_backlog=10') ``` --- # Before You Start Welcome to developing applications with AMPS, the Advanced Message Processing System from 60East Technologies! These guides will help you learn how to develop applications using AMPS. Before reading this guide, it is important to have a good understanding of the following topics: - *Developing in JavaScript* To be successful using this guide, you will need to possess a working knowledge of the JavaScript language. Visit [https://developer.mozilla.org/en-US/docs/Learn/JavaScript](https://developer.mozilla.org/en-US/docs/Learn/JavaScript) for resources on learning JavaScript. - *AMPS Concepts* Before reading this guide, you will need to understand the basic concepts of AMPS, such as *topics*, *subscriptions*, *messages* and *SOW*. Before working through this guide, we recommend reading the [Introduction to AMPS](/docs/intro-guide/intro) guide. Detailed explanations of the AMPS server behavior are in the [AMPS Server Documentation](/docs). - *An Installed Browser or Node.js Runtime* The AMPS JavaScript Client currently supports [most JavaScript environments](#javascript-support). You will also need a system on which you can compile and run code, and a server where you can host the AMPS server. ## JavaScript Support The AMPS JavaScript client supports both **Browser** and **Node.js** as well as their derivatives, such as Electron, and OpenFin. The supported environments are the following: - Node.js **0.12.1** or higher - Microsoft Internet Explorer **11** - Microsoft Edge **12** or higher - Google Chrome **16** or higher - Mozilla Firefox **11** or higher - Apple Safari **6.1** (iOS 6.0) or higher - Apple iOS Safari **7** or higher - Google Android Browser **4.4** (Android KitKat) or higher - Opera **12.1** or higher The JavaScript client supports all API features. ## Setting up a Development Instance You will need an installed and running AMPS server to use the product as well. You can write and compile programs that use AMPS without a running server, but you will get the most out of this guide by running the programs against a working server. Instructions for starting an instance of AMPS are available in the [Introduction to AMPS](/docs/intro-guide/intro) guide. :::tip The AMPS server runs on x64 Linux. The [Introduction to AMPS](/docs/intro-guide/intro) and [AMPS FAQ](/faq) contain information on how to run an AMPS server on a development system that does not run Linux. ::: --- # Client Identification AMPS uses the name of the client as a session identifier and as part of the identifier for messages originating from that client. For this reason, when a transaction log is enabled in the AMPS instance (that is, when the instance is recording a sequence of publishes and attempting to eliminate duplicate publishes), an AMPS instance will only allow one application with a given client name to connect to the instance. When a transaction log is present, AMPS **requires** the client name for a publisher to be: - Unique within a set of replicated AMPS instances - Consistent from invocation to invocation *if* the publisher will be publishing the same *logical* stream of messages If publishers do not meet this contract (for example, if the publisher changes its name and publishes the same messages, or if a different publisher uses the same session name), message loss or duplication can happen. 60East recommends always using consistent, unique client names. For example, the client name could be formed by combining the application name, an identifier for the host system, and the ID of the user running the application. A strategy like this provides a name that will be different for different users or on different systems, but consistent for instances of the application that should be treated as equivalent to the AMPS system. Likewise, if a publisher is sending a completely independent stream of messages (for example, a microservice that sends a different, unrelated sequence of messages each time it connects to AMPS), there is no need for a publisher to retain the same name each time it starts. However, if a publisher is resuming a stream of messages (as in the case when using a file-backed publish store), that publisher **must** use the same client name, since the publisher is resuming the session. --- # Connection Strings For AMPS The AMPS clients use connection strings to determine the server, port, transport, and protocol to use to connect to AMPS. When the connection point in AMPS accepts multiple message types, the connection string also specifies the precise message type to use for this connection. Connection strings have a number of elements: ![Diagram showing the different elements of a connection string](./assets/client-connection-string-js-light.svg#gh-light-mode-only) ![Diagram showing the different elements of a connection string](./assets/client-connection-string-js-dark.svg#gh-dark-mode-only) As shown in the figure above, connection strings have the following elements: - *Transport* - Defines the network used to send and receive messages from AMPS. In this case, the transport is `wss`. :::warning `ws` (WebSocket) and `wss` (WebSocket Secure) are the only supported transports for the JavaScript client. ::: - *Host Address* - Defines the destination on the network where the AMPS instance receives messages. The format of the address is dependent on the transport. For `ws` and `wss`, the address consists of a host name and port number. In this case, the host address is `localhost:9007`. - *Protocol* - Sets the format in which AMPS receives commands from the client. Most code uses the default `amps` protocol, which sends header information in JSON format. AMPS supports the ability to develop custom protocols as extension modules, and AMPS also supports legacy protocols for backward compatibility. - *Message Type* - Specifies the message type that this connection uses. This component of the connection string is required if the protocol accepts multiple message types and the transport is configured to accept multiple message types. If the protocol does not accept multiple message types, this component of the connection string is optional, and defaults to the message type specified in the transport. Legacy protocols such as `fix`, `nvfix` and `xml` only accept a single message type, and therefore do not require or accept a message type in the connection string. The JavaScript client does not support legacy protocols, the `amps` protocol is **required**. As an example, a connection string such as: ```bash ws://localhost:9007/amps/json ``` would work for programs connecting from the local host to a `Transport` configured as follows: ```xml showLineNumbers ... ... websocket-any websocket ... ... websocket-any tcp 9007 websocket-any ... ``` See the [Configuring Transports](/docs/amps-user-guide/transports/configuring-transports) and [Configuring Protocols](/docs/amps-user-guide/transports/configuring-protocols) sections in the _AMPS User Guide_ for more information on configuring transports and protocols. --- # Content Filtering One of the most powerful features of AMPS is content filtering. With content filtering, filters based on message content are applied at the server so that your application and the network are not utilized by messages that are not relevant to your application. For example, if your application is only displaying messages from a particular user, you can send a content filter to the server so that only messages from that particular user are sent to the client. To apply a content filter to a subscription, simply pass it into the `Client.subscribe()` call or use the `filter()` method to add a filter to the `Command`: ```javascript showLineNumbers await client.subscribe( msg => console.log('Mom says: ', msg.data), 'letters', // Topic '/sender="mom"', // Filter {timeout: 5000} // Timeout (optional) ) ``` In this example, we have passed in a content filter `/sender = 'mom'`. This will result in the server only sending us messages, from the `messages` topic, that have the sender field equal to `mom` in the message. For example, the AMPS server will send the following message, where `/sender` is `mom`: ```javascript showLineNumbers { "sender" : "mom", "text" : "Happy Birthday!", "reminder" : "Call me Thursday!" } ``` The AMPS server will not send a message with a different `/sender` value: ```javascript showLineNumbers { "sender" : "henry dave", "text" : "Things do not change; we change." } ``` --- # AMPS Programming: Working with Commands The AMPS clients provide named convenience methods for core AMPS functionality. These named methods work by creating messages and sending those messages to AMPS. All communication with AMPS occurs through messages. You can use the `Command` object to customize the messages that AMPS sends. This is useful for more advanced scenarios where you need precise control over the message, or in cases where you need to use an earlier version of the client to communicate with a more recent version of AMPS, or in cases where a named method is not available. ## Understanding AMPS Messages and Commands AMPS messages are represented in the client as `Message` objects. The `Message` object is generic, and can represent any type of AMPS message. This section includes a brief overview of elements common to `Command` - the class that is used to build AMPS messages. Full details of commands to AMPS are provided in the *AMPS Command Reference* (linked at the bottom of this page). All AMPS `Command` objects contain the following elements: - **Command** - The command tells AMPS how to interpret the message. Without a command, AMPS will reject the message. Examples of commands include `publish`, `subscribe`, and `sow`. - **CommandId** - The command ID, together with the name of the client, uniquely identifies a command to AMPS. The command ID can be used later on to refer to the command or the results of the command. For example, the command ID for a `subscribe` command becomes the identifier for the subscription. The AMPS client provides a command ID when the command requires one and no command ID is set. Most AMPS commands contain the following fields: - **Topic** - The topic that the command applies to, or a regular expression that identifies a set of topics that the command applies to. For most commands, the topic is required. Commands such as `logon`, `start_timer`, and `stop_timer` do not apply to a specific topic, and do not need this field. - **Ack Type** - The ack type tells AMPS how to acknowledge the message to the client. Each command has a default acknowledgment type that AMPS uses if no other type is provided. - **Options** - The `options` are a comma-separated list of options that affect how AMPS processes and responds to the message. Beyond these fields, different commands include fields that are relevant to that particular command. For example, SOW queries, subscriptions, and some forms of SOW deletes accept the **Filter** field, which specifies the filter to apply to the subscription or query. As another example, publish commands accept the **Expiration** field, which sets the SOW expiration for the message. For full details on the options available for each command and the acknowledgment messages returned by AMPS, see the *AMPS Command Reference*. ## Creating and Populating a Command To create a command, you simply construct a command object of the appropriate type: ```javascript var command = new amps.Command('sow') ``` Once created, you set the appropriate fields on the command. For example, the following code creates a publish message, setting the command, topic, data to publish, and an expiration for the message: ```javascript showLineNumbers var command = new amps.command('publish') .topic('messages') .data({id: 1, hello: 'World'}) .expiration(5) ``` Another example is a SOW command: ```javascript showLineNumbers var command = new amps.command('sow') .topic('messages-sow') .filter('/id > 20') ``` When sent to AMPS using the `Client.execute()` method, AMPS performs a SOW query from the topic `messages-sow` using a filter of `/id > 20`. The results of sending this message to AMPS are no different than using the form of the `sow` method that sets these fields. ## Using Execute Once you've created a command, use the `Client.execute()` method to send the command to AMPS. The method returns the `Promise` object, sends the message to AMPS, waits for a `processed` acknowledgment, then resolves the promise with the command ID. Messages are processed on the client asynchronously. For example, the following snippet sends the command created above: ```javascript showLineNumbers client .execute(command, messageHandler) .then(function(commandId) { ... }) .catch(...) ``` This is equivalent to calling the `Client.sow()` method. You can also provide a message handler to receive acknowledgments, statistics, or the results of subscriptions and SOW queries: ```javascript showLineNumbers function messageHandler(message) { // Received the requested ack 'processed' message if (message.c === 'ack') { console.log(message.a + ':', message.reason) } else { // Other messages are handled here // ... } } client.execute( // Request the ack message from AMPS command.ackType('processed'), // Message handler messageHandler ) ``` While this message handler simply prints the ack type and reason for sample purposes, message handlers in production applications are typically designed with a specific purpose. For example, your message handler may fill a work queue, or check for success and throw an exception if the command failed. Notice that the `publish` command does not typically return results other than acknowledgment messages. To send a `publish` command, use the `execute()` method, without providing a message handler: ```javascript client.execute(publishCmd) ``` ## AMPS Command Cookbook The [AMPS Command Reference](/docs/amps-command-reference) includes information on which fields and options to set on commands to get a specific result. The reference includes both reference information and a [Command Cookbook](/docs/amps-command-reference/cookbook) that provides a concise guide for commonly-used commands. --- # Providing Credentials to AMPS When using the default authenticator, the AMPS clients support the standard format for including a username and password in a URI, as shown below: ```bash ws://user:password@host:port/protocol/message_type ``` When provided in this form, the default authenticator provides the username and password specified in the URI. If you have implemented another authenticator, that authenticator controls how passwords are provided to the AMPS server. --- # Delta Publish To delta publish, you use the `Client.deltaPublish()` method as follows: ```javascript showLineNumbers const data = { ... } // obtain changed fields here client.deltaPublish('my-topic', data) ``` The message that you provide to AMPS must include the fields that the topic uses to generate the SOW key. Otherwise, AMPS will not be able to identify the message to update. For SOW topics that use a User-Generated SOW Key, use the `Command` form of `delta_publish` to set the `SowKey`, as shown below: ```javascript showLineNumbers const data = { ... } // obtain changed fields here const key = '...' // obtain user-generated SOW key const command = new Command('delta_publish') command.topic('delta-topic') command.sowKey(key) command.data(data) /** * Execute the delta publish. Do not provide a message * handler since delta_publish is not waiting for the * acknowledgment from the server. */ client.execute(command) ``` The [AMPS User Guide](/docs/amps-user-guide) section on making [Incremental Message Updates](/docs/amps-user-guide/delta-publish) describes how the AMPS server processes the `delta_publish` command. --- # Delta Subscribe To delta subscribe, you simply use the `delta_subscribe` command as follows: ```javascript showLineNumbers const subId = await client.execute( // The delta_subscribe command to execute new Command('delta_subscribe') .topic('delta-topic') .filter('/thingIWant = "true"'), // Message handler message => { /* Delta messages arrive here */ } ) // Once subscribed, it resolves with the subscription id. console.log(subId) ``` The convenience method `Client.deltaSubscribe()` is available. If the `delta_subscribe` is not a valid command for the topic provided, a regular subscription will be created. ```javascript showLineNumbers const subId = await client.deltaSubscribe( message => { /* ... */ }, // Message handler 'delta-topic', // Topic '/thingIWant = "true"' // Filter (optional) ) // Once subscribed, it resolves with the subscription id. console.log(subId) ``` As described in the [AMPS User Guide](/docs/amps-user-guide) section on [Receiving Only Updated Fields](/docs/amps-user-guide/delta-subscribe), messages provided to a delta subscription will contain the fields used to generate the SOW key and any changed fields in the message. Your application is responsible for choosing how to handle the changed fields. --- # Delta Publish and Subscribe Delta messaging in AMPS has two independent aspects: - **Delta Subscribe** - Allows subscribers to receive just the fields that are updated within a message. - **Delta Publish** - Allows publishers to update and add fields within a message by publishing only the updates into the SOW. This chapter describes how to create delta publish and delta subscribe commands using the AMPS JavaScript client. For a discussion of this capability, how it works, and how message types support this capability see the [AMPS User Guide](/docs/amps-user-guide). --- # Disconnect Handling Every distributed system will experience occasional disconnections between one or more nodes. The reliability of the overall system depends on an application's ability to efficiently detect and recover from these disconnections. Using the AMPS JavaScript client's disconnect handling, you can build powerful applications that are resilient in the face of connection failures and spurious disconnects. For additional reliability, you can also use the high availability functionality (discussed in the following sections), which provides both disconnect handling and features to help ensure that messages are reliably delivered. --- # Error Handling In every distributed system, the robustness of your application depends on its ability to recover gracefully from unexpected events. The AMPS client provides the building blocks necessary to ensure your application can recover from the kinds of errors and special events that may occur when using AMPS. --- # Changing the Filter on a Subscription AMPS allows you to update parameters, such as the content filter, on a subscription. When you replace a filter on the subscription, AMPS immediately begins sending only messages that match the updated filter. Notice that if the subscription was entered with a command that includes a SOW query, using the `replace` option can re-issue the SOW query (as described in the *AMPS User Guide*). To update the filter on a subscription, you create a `subscribe` command. You set the `SubscriptionId` provided on the `Command` to the identifier of the existing subscription and include the `replace` option on the `Command`. When you send the `Command`, AMPS atomically replaces the filter and sends messages that match the updated filter from that point forward. ```javascript showLineNumbers await client.connect('wss://localhost:9000/amps/json') // Original subscription const subId = await client.execute( new Command('subscribe').topic('orders').filter('/id > 100'), onMessagePrinter ) // Replace the subscription with a new filter value await client.execute( new Command('subscribe') .topic('orders') .filter('/id < 5') .subId(subId) .options('replace'), onMessagePrinter ) ``` --- # Your First AMPS Program In this chapter, we will learn more about the structure and features of the AMPS JavaScript client and build our first JavaScript program using AMPS. ## About the Client Library The AMPS client library is packaged as a single file, `amps.js` in development environments, or `amps.min.js` for production. Every JavaScript application you build will need to reference this file, and the file must be deployed along with your application in order for your application to function properly. Alternatively, for applications that are distributed through NPM, the client library can be listed as a dependency, and thus installed automatically. ## Including the Client in a Project There are several ways of including the JavaScript client in order to use it. Depending on your project's structure and environment, you can include the client in one of the following ways. ### Node.js / CommonJS Module ```javascript showLineNumbers // NPM installation: Import everything at once const amps = require('amps') // or // NPM installation: Include only selected components of AMPS const { Client, Command } = import('amps') // Same as above with manual installation const amps = require('./lib/amps') const { Client, Command } = import('./lib/amps') ``` ### ES6 / TypeScript Module ```javascript showLineNumbers // NPM installation import { Client, Command } from 'amps' // Manual installation import { Client, Command } from './lib/amps' ``` In case of **TypeScript** and manual installation, you need to reference the typing file (provided within the client package) at the top of the file: ```typescript /// import { Client, Command } from './lib/amps' ``` The AMPS JavaScript NPM package already includes typings, so that the library can be seamlessly used in native TypeScript projects. ### AMD / RequireJS Module ```javascript showLineNumbers // Included library will be available as the amps argument define(['./lib/amps'], amps => { // ... }) ``` ### Global Import in a Browser Environment ```html ``` Once the library in included in the HTML file, it becomes available as a global object, either as `amps` or `window.amps`. ## Connecting to AMPS Let's begin by writing a simple program that connects to an AMPS server and sends a single message to a topic. This code will be covered in detail just following the example. ```javascript showLineNumbers const uri = 'wss://127.0.0.1:9007/amps/json' async function main() { const client = new Client('examplePublisher') try { await client.connect(uri) client.publish('messages', {hi: 'Hello, World!'}) client.disconnect() } catch (err) { console.error(err) } } main() ``` In the example above, we show the entire program; but future examples will isolate one or more specific portions of the code. ### Examining the Code Let us now revisit the code we listed above: ```javascript showLineNumbers /** * The URI to use to connect to AMPS. The URI consists of * the transport, the address, and the protocol to use for * the AMPS connection. In this case, the transport is `wss` * (WebSocket Secure), the address is `127.0.0.1:9007`, and * the protocol is `amps`. This connection will be used for * JSON messages. Check with the person who manages the AMPS * instance to get the connection string to use for your programs. */ const uri = 'wss://127.0.0.1:9007/amps/json' async function main() { /** * This line creates a new Client object. Client encapsulates * a single connection to an AMPS server. Methods on Client allow * for connecting, disconnecting, publishing, and subscribing to * an AMPS server. The argument to the Client constructor, * `examplePublisher`, is a name chosen by the client to identify * itself to the server. Errors relating to this connection will be * logged with reference to this name, and AMPS uses this name to * help detect duplicate messages. AMPS enforces uniqueness for * client names when a transaction log is configured, and it is * good practice to always use unique client names. */ const client = new Client('examplePublisher') /** * Now, we attempt to connect to the AMPS server. Most API methods * use the Promise pattern to encapsulate asynchronous operations, * such as connecting or subscribing. We utilize the async/await * syntax to work with the asynchronous code in a somewhat * synchronous manner. * For example, once the connection is established, the publish * function will be called; if the call to connect() fails because * the provided address is not reachable, an error will be thrown * and caught in the `catch` clause with an Error object that * contains information about the occurred error. */ try { /** * Here, we provide a URI string to the client and establish * connection to AMPS. */ await client.connect(uri) /** * Here, we publish a single message to AMPS on the messages * topic, containing the data `Hello, world!`. This data is * placed into a JSON message and sent to the server. Upon * successful completion of this function, the AMPS client has * sent the message to be sent to the server, and subscribers * to the messages topic will receive this JSON message: * '{"hi": "Hello, world!"}'. */ client.publish('messages', {hi: 'Hello, World!'}) /** * We close the connection to AMPS. It's good practice to * close connections when you are done using them as the * client object will stay connected otherwise. */ client.disconnect() } catch (err) { // Failed connection error console.error(err) } } // running the above function main() ``` ### Syntax Convention In the above example and throughout this guide we use the modern JavaScript syntax which is widely supported and typically is used in projects with a build/deployment system, such as `webpack`, `angular-cli`, and `create-react-app`. However, the JavaScript client library fully supports the obsolete syntax used in older browsers and projects without a build system as well. Below is the same program, written in the classic **ES5** syntax. ```javascript showLineNumbers var uri = 'wss://127.0.0.1:9007/amps/json' var client = new amps.Client('examplePublisher') client.connect(uri) .then(function() { client.publish('messages', {hi: 'Hello, world!'}) client.disconnect() }) .catch(function(error) { console.error(error) }) ``` --- # Using a Heartbeat to Detect Disconnection The AMPS client includes a heartbeat feature to help applications detect disconnection from the server within a predictable amount of time. Without using a heartbeat, an application must rely on the environment to notify it when a disconnect occurs. For applications that are simply receiving messages, it can be impossible to tell whether a socket is disconnected or whether there are simply no incoming messages for the client. When you set a heartbeat, the AMPS client sends a heartbeat message to the AMPS server at a regular interval, and expects a response from the server within the specified amount of time. If the operating system reports an error on send, or if there is no activity received from the server within the specified amount of time, the AMPS client considers the server to be disconnected. Likewise, the server will ensure that traffic is sent to the client at the specified interval, using heartbeat messages when no other traffic is being sent to the client. If, after sending a heartbeat message, no traffic from the client arrives within a period twice the specified interval, the server will consider the client to be disconnected or nonresponsive. The AMPS client processes heartbeat messages asynchronously. If your application publishes messages in a loop (a synchronous operation), or the message handler is receiving significant amount of messages at the moment, the client may fail to respond to heartbeat messages in a timely manner and may be disconnected by the server. ```javascript // Create a new client and set the heartbeat interval to 5 sec const client = new Client('heartbeat-demo').heartbeat(5) ``` The `Client` class, included with the AMPS JavaScript client, contains a disconnect handler and other features for building highly-available applications. The `Client` includes features for managing a list of failover servers, resuming subscriptions, republishing in-flight messages, and other functionality that is commonly needed for high availability. This section covers the use of a custom disconnect handler in the event that the behavior of the `Client` does not suit the needs of your application. 60East recommends using the provided disconnect handler unless there is specific behavior that does not meet your needs (for example, if your application does not want to reconnect to AMPS in the event of a disconnection). :::tip The `Client` class in the JavaScript library combines functionality of both `Client` and `HAClient` classes of other AMPS client libraries. ::: --- # High Availability The AMPS JavaScript Client provides an easy way to create highly-available applications using AMPS, via the `Client` class. In other clients the highly available version of the client is called `HAClient`, however in JavaScript the `Client` class has all the functions of `Client`, providing both regular and highly available modes. Among its regular functions, `Client` provides protection against network, server, and client outages. :::info The `Client` class in the JavaScript library combines functionality of both `Client` and `HAClient` classes of other AMPS client libraries. ::: Using High Availability (HA) functions of the `Client` allows applications to automatically: - Recover from temporary disconnects between client and server. - Failover from one server to another when a server becomes unavailable. Since the `Client` can automatically manage failover and reconnection, 60East recommends using the HA functions for applications that need to: - Ensure no messages are lost or duplicated after a reconnect or failover. - Persist messages and bookmarks in memory for protection against client failure. You can choose how your application uses HA features. For example, you might need automatic reconnection, but have no need to resume subscriptions or republish messages. The high availability behavior in `Client` is provided by implementations of defined interfaces. You can combine different implementations provided by 60East to meet your needs, and implement those interfaces to provide your own policies. Some of these features require specific configuration settings on your AMPS instance(s). This chapter mentions these features and describes how to use them from the AMPS JavaScript client. You can find full documentation for these settings and server features in the *AMPS User Guide*. ## Reconnection with Client This description provides a high-level framework for understanding the components involved in failover with the `Client`. The components are described in more detail in the following sections. The `Client` reconnect handler performs the following steps when reconnecting: 1. Calls the `ServerChooser` to determine the next URI to connect to and the `Authenticator` to use for that connection. If the connection fails, calls `getError()` on the `ServerChooser` to get a description of the failure, sends an error to the error handler, and stops the reconnection process. 2. Calls the `DelayStrategy` to determine how long to wait before attempting to reconnect, and waits for that period of time. 3. Connects to the AMPS server. If the connection fails, calls `reportFailure()` on the `ServerChooser` and begins the process again. 4. Logs on to the AMPS server. If the connection fails, calls `reportFailure()` on the `ServerChooser` and begins the process again. 5. Calls `reportSuccess()` on the `ServerChooser`. 6. Receives the bookmark for the last message that the server has persisted. Discards any older messages from the `PublishStore`. 7. Republishes any messages in the `PublishStore` that have not been persisted by the server. 8. Re-establishes subscriptions using the `SubscriptionManager` for the client. For bookmark subscriptions, the reconnect handler uses the `BookmarkStore` for the client to determine the most recent bookmark, and resubscribes with that bookmark. For subscriptions that do not use a bookmark, the `SubscriptionManager` simply re-enters the subscription, meaning that it is entered at the point at which the `Client` reconnects. The `ServerChooser`, `DelayStrategy`, `PublishStore`, `SubscriptionManager`, and `BookmarkStore` are all extension points for the `Client`. You can adapt the failover and recovery behavior by setting a different object for the behavior you want to customize on the `Client` or by providing your own implementation. ## Choosing Store Durability The `Client` provides recovery after disconnection using *Stores*. As the name implies, *stores* hold information about the state of the client. These stores provide the following capabilities: - A *bookmark store* tracks received messages, and is used to resume subscriptions. - A *publish store* tracks published messages, and is used to ensure that messages are persisted in AMPS. The AMPS JavaScript client provides a memory-backed version of each store. The store interface is public, and an application can create and provide a custom store as necessary. A `Client` can use a memory backed store for protection, as described below: - *Memory-backed stores* provide disconnection recovery from AMPS by storing messages and bookmarks in your process' address space. This is the highest performance option for working with AMPS in a highly available manner. The trade-off with this method is there is no protection from a crash or failure of your client application. If your application is terminated prematurely or, if the application terminates at the same time as an AMPS instance failure or network outage, then messages may be lost or duplicated. A memory-backed store should only be used by one instance of a client at a time. The store interface is public, and an application can create and provide a custom store as necessary. While clients provide convenience methods for creating memory-backed `Client` objects with the appropriate stores, you can also create and set the stores in your application code. The `Client` provides convenience methods for creating clients and setting stores. You can also construct a `Client` and set the store implementations you choose. In this example, we create several clients. The first client has the full set of HA features, such as: supports automatic connection failover and reconnection, uses memory stores for both bookmarks and publishes, and automatically re-subscribes in case of a failover. The second client does not set a store for publishes, which means that AMPS will not store outgoing messages, but is using the bookmark store to track incoming messages. The final client does not specify any stores, and so has no persistence for published messages or bookmark subscriptions, but takes advantage of the automatic failover and reconnection in the `Client`. ```javascript showLineNumbers // Failover, Bookmark and Publish stores, re-subscription const memoryClient = Client.createMemoryBacked('memory-backed') // No publish store, no failover, memory-backed bookmark store const subscriberClient = new Client('subscriber') subscriberClient.bookmarkStore(new MemoryBookmarkStore()) // Failover behavior only. const failoverClient = new Client('failover') const serverChooser = new DefaultServerChooser() serverChooser.add('ws://localhost:9000/amps/json') serverChooser.add('ws://localhost:9100/amps/json') serverChooser.add('ws://localhost:9200/amps/json') failoverClient.serverChooser(serverChooser) failoverClient.delayStrategy(new FixedDelayStrategy()) ``` ## Connections and the Server Chooser In the high availability mode, `Client` attempts to keep itself connected to an AMPS instance at all times, by automatically reconnecting or failing over when it detects that the client is disconnected. When you are using the `Client` directly, your disconnect handler usually takes care of reconnection. `Client` with HA classes on the other hand, can provide a disconnect handler that automatically reconnects to the current server or to the next available server. To inform the `Client` of the addresses of the AMPS instances in your system, you pass a `ServerChooser` instance to the `Client`. `ServerChooser` acts as a smart enumerator over the servers available: `Client` calls `ServerChooser` methods to inquire about what server should be connected, and calls methods to indicate whether a given server succeeded or failed. The AMPS JavaScript Client provides a simple implementation of `ServerChooser`, called `DefaultServerChooser`, that provides very simple logic for reconnecting. This server chooser is most suitable for basic testing, or in cases where an application should simply rotate through a list of servers. For most applications, you implement the `ServerChooser` interface yourself for more advanced logic, such as choosing a backup server based on your network topology, or limiting the number of times your application should try to reconnect to a given address. To connect to AMPS, you provide a `ServerChooser` to `Client` and then call `Client.connect()` to create the first connection: ```javascript showLineNumbers const memoryClient = new Client('server-chooser-demo') // primary.amps.xyz.com is the primary AMPS instance, and // secondary.amps.xyz.com is the secondary const chooser = new DefaultServerChooser() chooser.add('ws://primary.amps.xyz.com:12345/amps/fix') chooser.add('ws://secondary.amps.xyz.com:12345/amps/fix') memoryClient.serverChooser(chooser) try { await memoryClient.connect() // ... later await client.disconnect() } catch (err) { // handle connection error } ``` `Client` remains connected to the server until `disconnect()` is called. `Client` automatically attempts to reconnect to your server if it detects a disconnect and, if that server cannot be connected, fails over to the next server provided by the `ServerChooser`. In this example, the call to `connect()` attempts to connect and login to `primary.amps.xyz.com`, and resolves if that is successful. If it cannot connect, it tries `secondary.amps.xyz.com`, and continues trying servers from the `ServerChooser` until a connection is established. Likewise, if it detects a disconnection while the client is in use, then `Client` attempts to reconnect to the server with which it was most recently connected; if that is not possible, then it moves on to the next server provided by the `ServerChooser`. The default `ServerChooser` simply provides the next URL in the sequence. This strategy works for many applications. If you need a different strategy, you can implement your own logic for failover by creating a class derived from `ServerChooser`. ### Setting a Reconnect Delay and Timeout You can control the amount of time between reconnection attempts and set a total amount of time for the `Client` to attempt to reconnect. The AMPS JavaScript client includes a method for setting a delay strategy on a client, `Client.delayStrategy()`. This method accepts an instance of any type that provides the methods `getConnectWaitDuration()` and `reset()`, as described in the API documentation. While you can easily implement your own delay strategy, the client also provides two delay strategies: - `FixedDelayStrategy` provides the same delay each time the `Client` tries to reconnect. - `ExponentialDelayStrategy` provides an exponential backoff until a connection attempt succeeds. To use either of these classes, you simply create an instance, set the appropriate parameters, and install that instance as the delay strategy for the `Client`. For example, the following code sets up a reconnect delay that starts at 200ms and increases the delay by 1.5 times after each failure. The strategy allows a maximum delay between connection attempts of 5 seconds, and will not retry longer than 60 seconds. ```javascript showLineNumbers const client = new Client('delay-strategy-demo') client.delayStrategy( new ExponentialDelayStrategy({ initialDelay: 200, maximumDelay: 5 * 1000, backoffExponent: 1.5, maximumRetryTime: 60 * 1000 }) ) ``` ### Implementing a Server Chooser As described above, you provide the `Client` with connection strings to one or more AMPS servers using a `ServerChooser`. The purpose of a `ServerChooser` is to provide information to the `Client`. A `ServerChooser` does not manage the reconnection process, and should not call methods on the `Client`. A `ServerChooser` has two required responsibilities to the `Client`: - Tells the `Client` the connection string for the server to connect to. If there are no servers, or the `ServerChooser` wants the connection to fail, the `ServerChooser` returns `null`. To provide this information, the `ServerChooser` implements the `getCurrentUri()` method. - Provides an `Authenticator` for the current connection string. This is especially important for installations where different servers require different credentials or authentication tokens must be reset after each connection attempt. To provide the authenticator, the `ServerChooser` implements the `getCurrentAuthenticator()` method. The `Client` calls the `getCurrentUri()` and `getCurrentAuthenticator()` methods each time it needs to make a connection. Each time a connection succeeds, the `Client` calls the `reportSuccess()` method of the `ServerChooser`. Each time a connection fails, the `Client` calls the `reportFailure()` method of the `ServerChooser`. The `Client` does not require the `ServerChooser` to take any particular action when it calls these methods. These methods are provided for the `Client` to do internal maintenance, logging, or record keeping. For example, a `Client` might keep a list of available URIs with a current failure count, and skip over URIs that have failed more than 5 consecutive times until all URIs in the list have failed more than 5 consecutive times. When the `ServerChooser` returns a `null` from `getCurrentUri()`, indicating that no servers are available for connection, the `Client` calls the `getError()` method on the `ServerChooser`, if one is provided, and includes the string returned by `getError()` in the generated exception. ## Heartbeats and Failure Detection Use of the `Client` allows your application to quickly recover from detected connection failures. By default, connection failure detection occurs when AMPS receives an operating system error on the connection. This system may result in unpredictable delays in detecting a connection failure on the client, particularly when failures in network routing hardware occur, and the client primarily acts as a subscriber. The heartbeat feature of the AMPS client allows connection failure to be detected quickly. Heartbeats ensure that regular messages are sent between the AMPS client and server on a predictable schedule. The AMPS client and server both assume disconnection has occurred if these regular heartbeats cease, ensuring disconnection is detected in a timely manner. To use the heartbeat feature, call the `heartbeat()` method on `Client`: ```javascript showLineNumbers const memoryClient = new Client('importantStuff') // ... memoryClient.heartbeat(3) await memoryClient.connect() ``` Method `heartbeat()` takes one parameter: the heartbeat interval. The heartbeat interval specifies the periodicity of heartbeat messages sent by the server: the value `3` indicates messages are sent on a three-second interval. If the client receives no messages in a six-second window (two heartbeat intervals), the connection is assumed to be dead, and the `Client` attempts reconnection. The optional second parameter of the `heartbeat()` method allows the idle period to be set to a value other than two heartbeat intervals. :::warning Heartbeats are handled asynchronously by the AMPS client. Your application must not flood the execution queue for longer than the heartbeat interval, or the application is subject to being disconnected. ::: ## Considerations for Publishers Publishing with the `Client` in the HA mode is nearly identical to regular publishing; you simply call the `publish()` method with your message’s topic and data. The AMPS client sends the message to AMPS, and then returns from the `publish()` call. For maximum performance, the client does not wait for the AMPS server to acknowledge that the message has been received. When a `Client` sets a publish store, the publish store retains a copy of each outgoing message and requests that AMPS acknowledge that the message has been persisted. The AMPS server acknowledges messages back to the publisher. Acknowledgments can be delivered for multiple messages at periodic intervals (for topics recorded in the transaction log) or after each message (for topics that are not recorded in the transaction log). When an acknowledgment for a message is received, the `Client` removes that message from the publish store. When a connection to a server is made, the `Client` automatically determines which messages from the publish store (if any) the server has not processed, and replays those messages to the server once the connection is established. For reliable publishers, the application must choose how best to handle application shutdown. For example, it is possible for the network to fail immediately after the publisher sends the message, while the message is still in transit. In this case, the publisher has sent the message, but the server has not processed it and acknowledged it. During normal operation, the `Client` will automatically connect and retry the message. On shutdown, however, the application must decide whether to wait for messages to be acknowledged, or whether to exit. Publish store implementations provide an `unpersistedCount()` method that reports the number of messages that have not yet been acknowledged by the AMPS server. When the `unpersistedCount()` reaches `0`, there are no unpersisted messages in the local publish store. For the highest level of safety, an application can wait until the `unpersistedCount()` reaches `0`, which indicates that all of the messages have been persisted to the instance that the application is connected to, and the synchronous replication destinations configured for that instance. When a synchronous replication destination goes offline, this approach will cause the publisher to wait to exit until the destination comes back online or until the destination is downgraded to asynchronous replication. For applications that are shut down periodically for short periods of time (for example, applications that are only offline during a weekly maintenance window), another approach is to use the `Client.flush()` method to ensure that messages are delivered to AMPS, and then rely on the connection logic to replay messages as necessary when the application restarts. For example, the following code flushes messages to AMPS, then warns if not all messages have been acknowledged: ```javascript showLineNumbers const client = Client.createMemoryBacked('ha-publisher') // ... await client.connect() // Publish messages client.publish('topic', {id: 1}) client.publish('topic', {id: 2}) // ... /** * We think we are done, but the server may not * have received or acknowledged all messages yet. * Wait until the server has received all messages. * The program could also specify a timeout in this * command to avoid blocking forever if the network * is down or all servers are offline. */ await client.flush() /** * Print warning to the console if messages have * been published but not yet acknowledged as persisted */ if (client.publishStore().unpersistedCount() > 0) { console.log('All messages have been published') console.log('But not all have been persisted') } client.disconnect() ``` In this example, the client sends each message immediately when `publish()` is called. If AMPS becomes unavailable between the final `publish()` and the `disconnect()`, or one of the servers that the AMPS instance replicates to is offline, the client may not have received a persisted acknowledgment for all of the published messages. For example, if a message has not yet been persisted by all of the servers in the replication fabric that are connected with synchronous replication, AMPS will not have acknowledged the message. Before shutting down the client, the code does two things: - First, the code flushes messages to the server to ensure that all messages have been delivered to AMPS. - Next, the code checks to see if all of the messages in the publish store have been acknowledged as persisted by AMPS. If the messages have not been acknowledged, they will remain in the publish store and will be published to AMPS, if necessary, the next time the client connects. An application may choose to wait until `unpersistedCount()` returns `0`, or (as we do in this case) simply warn that AMPS has not confirmed that the messages are fully persisted. The behavior you choose in your application should be consistent with the high-availability guarantees your application needs to provide. :::warning AMPS uses the name of the `Client` to determine the origin of messages. For the AMPS server to correctly identify duplicate messages, each instance of an application that publishes messages must use a distinct name. That name must be consistent across different runs of the application. ::: :::warning AMPS provides persisted acknowledgment messages for topics that do not have a transaction log enabled. However, the level of durability provided for topics with no transaction log is minimal. Learn more about transaction logs in the *AMPS User Guide*. ::: ## Considerations for Subscribers `Client` provides two important features for applications that subscribe to one or more topics: re-subscription, and a bookmark store to track the correct point at which to resume a bookmark subscription. ### Resubscription with the Subscription Manager Any asynchronous subscription placed using a `Client` is automatically reinstated after a disconnect or a failover. These subscriptions are placed in an in-memory `SubscriptionManager`, which is created automatically when the `Client.createMemoryBacked()` static method is called. Alternatively, it can be created and assigned before the client is connected: ```javascript showLineNumbers const client = new Client('subscription-manager-demo') client.subscriptionManager(new DefaultSubscriptionManager()) ``` Most applications will use this built-in subscription manager, but for applications that create a varying number of subscriptions, you may wish to implement `SubscriptionManager` to store subscriptions in a more durable place. Note that these subscriptions contain no message data, but rather simply contain the parameters of the subscription itself (for instance, the command, topic, message handler, options, and filter). When a re-subscription occurs, the AMPS JavaScript Client re-executes the command as originally submitted, including the original topic, options, and so on. AMPS sends the subscriber any messages for the specified topic (or topic expression) that are published after the subscription is placed. For a `sow_and_subscribe` command, this means that the client reissues the full command, including the SOW query as well as the subscription. :::tip A `sow` command is a point-in-time query. It isn't added to the subscription manager, and isn't restarted if a disconnection happens in the middle of a query. A `sow_and_subscribe` is a subscription, and is added to the subscription manager. ::: ### Bookmark Stores In cases where it is critical not to miss a single message, it is important to be able to resume a subscription at the exact point that a failure occurred. In this case, simply recreating a subscription isn't sufficient. Even though the subscription is recreated, the subscriber may have been disconnected at precisely the wrong time, and will not see the message. To ensure delivery of every message from a topic or set of topics, the AMPS `Client` can plug in a `BookmarkStore` that, combined with the bookmark subscription and transaction log functionality in the AMPS server, ensures that clients receive any messages that might have been missed. The client stores the bookmark associated with each message received, and tracks whether the application has processed that message; if a disconnect occurs, the client uses the `BookmarkStore` to determine the correct resubscription point, and sends that bookmark to AMPS when it re-subscribes. AMPS then replays messages from its transaction log from the point after the specified bookmark, thus ensuring the client is completely up-to-date. `Client` helps you to take advantage of this bookmark mechanism through the `Client.bookmarkStore()` method and `MemoryBookmarkStore` class. When a bookmark store is assigned to the client, whenever a disconnection or failover occurs, your application automatically re-subscribes to the message after the last message it processed. To take advantage of bookmark subscriptions, do the following: - Ensure the topic(s) to be subscribed to are included in a transaction log. See the *AMPS User Guide* for information on how to specify the contents of a transaction log. - Before connecting, create and assign a bookmark store object to the client. - Use the `Client.bookmarkStore().discard()` method in message handlers to indicate when a message has been fully processed by the application. The following example creates a bookmark subscription against a transaction-logged topic, and fully processes each message as soon as it is delivered: ```javascript showLineNumbers const client = new Client('aClient') client.subscriptionManager(new DefaultSubscriptionManager()) client.bookmarkStore(new MemoryBookmarkStore()) // ... await client.execute( // The subscription command new Command('subscribe') .topic('myTopic') .bookmark(Client.Bookmarks.MOST_RECENT) .subId('MySubId'), // Message handler discards every message after processing message => { console.log(message.data) client.bookmarkStore().discard(message) } ) ``` Storing these bookmarks in the bookmark store allows the application to restart the subscription from the last message processed, in the event of either server failure or disconnect. :::info For optimum performance, it is critical to discard every message received from a bookmark replay once its processing is complete. If a message is never discarded, it remains in the bookmark store. During re-subscription, ``Client`` always restarts the bookmark subscription with the oldest undiscarded message, and then filters out any more recent messages that have been discarded. If an old message remains in the store, but is no longer important for the application’s functioning, then the client and the AMPS server will incur unnecessary network and CPU activity. ::: The command method, `subId()`, specifies an identifier to be used for this subscription. If the `subId` is not provided, `Client` will generate one and resolve the `Client.execute()` Promise with it, like most other `Client` functions. If you wish to resume a subscription from a previous point after the application has disconnected, the application must pass the same subscription ID as before. Passing a different subscription ID bypasses any recovery mechanisms, creating an entirely new subscription. When you use an existing subscription ID, the `Client` locates the last-used bookmark for that subscription in the bookmark store, and attempts to re-subscribe from that point. - `Client.Bookmarks.NOW` specifies that the subscription should begin from the moment the server receives the subscription request. This results in the same messages being delivered as if you had invoked `subscribe()` instead, except that the messages will be accompanied by bookmarks. This is also the behavior that results if you supply an invalid bookmark. - `Client.Bookmarks.EPOCH` specifies that the subscription should begin from the beginning of the AMPS transaction log (that is, the first entry in the oldest journal file for the transaction log). - `Client.Bookmarks.MOST_RECENT` specifies that the subscription should begin from the last-used message in the associated `BookmarkStore`. Alternatively, if this subscription has not been seen before, it instructs the subscription to begin with `EPOCH`. This is the most common value for this parameter, and is the value used in the preceding example. By using `MOST_RECENT`, the application automatically resumes from wherever the subscription left off, taking into account any messages that have already been processed and discarded. When the `Client` re-subscribes after a disconnection and reconnection, it always uses `MOST_RECENT`, ensuring that the continued subscription always begins from the last message used before the disconnect, so that no messages are missed. ## Conclusion With only a few changes, most AMPS applications can take advantage of the high availability features of the `Client` to become more highly-available and resilient. Using the `PublishStore`, publishers can ensure that every message published has actually been persisted by AMPS. Using `BookmarkStore`, subscribers can make sure that there are no gaps or duplicates in the messages received. `Client` makes both kinds of applications more resilient to network and server outages and temporary issues. Though `Client` provides useful defaults for the `PublishStore`, `BookmarkStore`, `SubscriptionManager`, `ServerChooser`, and `DelayStrategy`, you can customize any or all of these to the specific needs of your application and architecture. --- # Obtaining and Installing the AMPS Client ## Manual Installation The AMPS JavaScript client is available as a download from the [60East Technologies](https://www.crankuptheamps.com/develop/) website. Download the client from the site, then extract it. The client source files are in the directory where you unpacked the files. By default, this is `amps-javascript-client-`, where `` is the current version of the JavaScript client (such as `amps-javascript-client-5.2.1.1`). Once unpacked, `amps.js` can be included in the project in order to use the client. Optionally, depending on your development environment, the `es6-promise.js` might also be required. ## Installation via Node Package Manager If your project is using [Node Package Manager](https://www.npmjs.com/) to manage and install dependencies, the client can be downloaded and installed via **NPM**: ```bash npm install --save amps ``` The client library will be automatically installed and included in your project. :::tip NPM also automatically resolves external client dependencies, allows installing specific versions of the JavaScript client, and supports automatic updates of the library. ::: --- # Managing Disconnection The `Client` class contains a disconnect handler and other features for building highly-available applications. The `Client` includes features for managing a list of failover servers, resuming subscriptions, republishing in-flight messages, and other functionality that is commonly needed for high availability. 60East recommends using the `Client` for automatic reconnection wherever possible, as the disconnect handler has been carefully crafted to handle a wide variety of edge cases and potential failures. If an application needs to reconnect or fail over, use a `Client` with a `ServerChooser` set, and the AMPS client library will automatically handle failover and reconnection. You control which servers the client fails over to by implementing the `ServerChooser` and you can control the timing of the failover by using one of the provided `ReconnectDelayStrategy` classes or implementing your own. :::info For most applications, the combination of the provided `Client` disconnect handler and a `ConnectionStateListener` gives you the ability to monitor disconnections and add custom behavior at the appropriate point in the reconnection process. ::: If you need to add custom behavior to the failover (such as logging, resetting an internal cache, refreshing credentials and so on), the `ConnectionStateListener` class allows your application to be notified and take action when disconnection is detected and at each stage of the reconnection process. To extend the behavior of the AMPS client during reconnection, implement a `ConnectionStateListener`. --- # Managing SOW Contents AMPS allows applications to manage the contents of the SOW by explicitly deleting messages that are no longer relevant. For example, if a particular delivery van is retired from service, the application can remove the record for the van by deleting the record for the van. The client provides the following methods for deleting records from the SOW: - `sowDelete()` - Accepts a topic and filter, and deletes all messages that match the filter from the topic specified. - `sowDeleteByKeys()` - Accepts a set of SOW keys as a comma-delimited string and deletes messages for those keys, regardless of the contents of the messages. SOW keys are provided in the header of a SOW message, and are the internal identifier AMPS uses for that SOW message. - `sowDeleteByData()` - Accepts a topic and message, and deletes the SOW record that would be updated by that message. Most applications use `sowDelete()`, since this is the most useful and flexible method for removing items from the SOW. In some cases, particularly when working with extremely large SOW databases, `sowDeleteByKeys()` can provide better performance. In either case, AMPS sends an OOF message to all subscribers who have received updates for the messages removed, as described in the previous section. `sowDelete()` returns a Promise that resolves with a `Message` object. This `Message` is an acknowledgment that contains information on the delete command. For example, the following snippet simply prints informational text with the number of messages deleted: ```javascript showLineNumbers const ack = await client.sowDelete('sow-topic', '/id IN (42, 37)') console.log( 'Got an', ack.header.command(), 'message containing', ack.header.ackType(), '-- deleted', ack.header.matches(), 'SOW entries' ) // Got an ack message containing stats -- deleted 2 SOW entries ``` Acknowledging messages from a queue uses a form of the `sow_delete` command that is only supported for queues. Acknowledgment is discussed in the [Using Queues](queues) chapter in this guide. --- # Manual Acknowledgment To manually acknowledge processed messages and remove the messages from the queue, applications use the `sow_delete` command with the bookmarks of the messages to remove. Notice that AMPS only supports using a bookmark with `sow_delete` when removing messages from a queue, not when removing records from a SOW. For example, given a `Message` object to acknowledge and a client, the code below acknowledges the message. ```javascript showLineNumbers const acknowledgeSingle = async (client, message) => { return client.execute( new Command('sow_delete') .topic(message.header.topic()) .bookmark(message.header.bookmark()) ) } ``` In the example above, the program creates a `sow_delete` command, specifies the topic and the bookmark, and then sends the command to the server. While this method works, creating and sending an acknowledgment for each individual message can be inefficient if your application is processing a large volume of messages. Rather than acknowledging each message individually, your application can build a comma-delimited list of bookmarks from the processed messages and acknowledge all of the messages at the same time. In this case, it's important to be sure that the number of messages you wait for is less than the maximum backlog -- the number of messages your client can have unacknowledged at a given time. Notice that both automatic acknowledgment and the convenience method `Client.ack()` take the maximum backlog into account. --- # Understanding Messages Objects So far, we have seen that subscribing to a topic involves working with objects of the `Message` type. A `Message` represents a single message from an AMPS server, while a `Command` is sent to a server. Commands are sent and messages are received for every client/server operation in AMPS. ## Header Properties There are two parts of each message in AMPS: a set of headers that provide metadata for the message, and the data that the message contains. Every AMPS message has one or more header fields defined. The precise headers present depend on the type and context of the message. There are many possible fields in any given message, but only a few are used for any given message. For each header field, the `Message` object contains a distinct method that allows for retrieval of that field. For example, the `Message.header.commandId()` corresponds to the `CommandId` header field, the `Message.header.batchSize()` corresponds to the `BatchSize` header field, and so on. For more information on these header fields, consult the *AMPS User Guide* and *AMPS Command Reference*. `Message` class represents messages received from an AMPS server. When creating a message to be sent, the `Command` class is used. To work with header fields, a `Command` contains a set of `()` methods, which work as both setters and getters, allowing to chain command properties when creating a new command. ## Message Data Received message data is contained in the `Message.data` property. The `data` property will contain the parsed data of the message. The AMPS JavaScript client contains a collection of helper classes for working with message types that are specific to AMPS (for example, JSON, FIX, NVFIX, and AMPS composite message types). You can replace default parsers to implement required specific behavior, as well as add new helpers. ## Message Field Reference The [AMPS Command Reference](/docs/amps-command-reference) contains a full description of which fields are available and which fields are returned in response to specific commands. --- # Message Types Unlike other AMPS client libraries, the JavaScript library parses message data contents before delivering the message to a handler. This happens for all the supported message types. By default, the client supports the following message types: - JSON - FIX / NVFIX - via `FixTypeHelper` class - Binary Messages of these types are automatically parsed into native JavaScript data structures and are ready for consumption. The client library utilizes the concept of a type helper for message data processing. It’s an interface with a set of methods for parsing and serializing message data of a particular type. The provided static `TypeHelper` class is used for fine grained control over message types. It is used to register new composite and custom message types, or replace type helpers for existing types. ## Composite Message Types The `TypeHelper.compositeHelper` static method handles creating and parsing composite message types: ```javascript showLineNumbers // Register the json-xml-json-binary composite message type. TypeHelper.helper( // The name of the new composite message type. 'compositejxjb', // Create the composite type helper. TypeHelper.compositeHelper( 'json', // Part 1: JSON 'xml', // Part 2: XML 'json', // Part 3: JSON 'binary' // Part 4: Binary ) ) ``` The composite type helper created with `TypeHelper.compositeHelper` will automatically build data to send from the array of the message parts and parse multi-part messages from the received raw data. All types used in the new composite type helper should be registered before creating it. ### Building Composite Messages Once the composite type helper is registered, it becomes very easy to build composite messages of that type. In the following example, we register a new composite message type, `compositejjjb` that consists of three JSON parts and one binary part. Then, we create the composite message and publish it. The type helper will take care of converting message parts into the raw message data that will be sent: ```javascript showLineNumbers // Before creating a client, register the composite message type. TypeHelper.helper( 'compositejjjb', TypeHelper.compositeHelper('json', 'json', 'json', 'binary') ) const client = new Client('composite-sender') await client.connect('ws://localhost:9000/amps/compositejjjb') // Create the array with message parts. const parts = [ {id: 5, value: 'part1'}, // JSON part 1 {value: 'part2', data: 22.22}, // JSON part 2 {value: 'part3', data: 33.33}, // JSON part 3 'XXXXXXXXXXXXXXXXXXXXXXXXXXX' // Binary data ] // Publish the composite message. client.publish('composite-topic', parts) ``` ### Parsing Composite Messages Once the composite type helper is registered, the composite messages of that type will be parsed automatically. The `Message.data` field will contain the array of message parts, parsed according to their types: ```javascript showLineNumbers await client.subscribe( message => { const parts = message.data console.log('Received message with', parts.length, 'parts') parts.forEach(part => console.log(part)) }, 'composite-topic' ) ``` Notice that the receiving application is written with explicit knowledge of the structure and content of the composite message type. ## Custom Message Types If a message type used in your application is not supported by default, it is possible to create a new type helper for it. Another situation in which you might need a custom type helper is when the default implementation does not fit your needs. For example, the default implementation of FIX / NVFIX message types assumes that the keys in each message are unique, thus overriding values if the same key occurs twice in the same message. If your messages contain duplicated keys and that is expected behavior, you need to override the default type helper with a custom implementation. Each type helper is an object that must contain the following methods: - `serialize(data: any): string[]` - this method is used to serialize data in order to send it to the server. All data chunks should be converted into an array of strings. - `deserialize(data: any): any` - this method deserializes data from the Server into a format that will be consumed by the message handler. It can be any format as long as the consumer of the data is aware of it. Below we provide some examples on how to utilize `TypeHelper` functionality. ## Create New Type Helpers Create and register a custom type helper for XML messages: ```javascript showLineNumbers class XMLTypeHelper { serialize(data) { return [new XMLSerializer().serializeToString(data)] } deserialize(data) { if (data.constructor === String) { return new DOMParser().parseFromString(data) } // Binary buffer, need to decode utf8. return new DOMParser().parseFromString( decodeURIComponent(escape(Uint8ToString(data))) ) } } // Register the XML helper for parsing XML messages. TypeHelper.helper('xml', new XMLTypeHelper()) ``` ## Override Default Type Helpers In case the custom parsing behavior is expected, it is possible to override the default type helper: ```javascript showLineNumbers /** * Create a JSON type helper that does not parse JSON data into native * JS objects keeping it in the form of a string. */ const jsonHelper = { serialize: data => [data], deserialize: data => JSON.stringify(data) } // Override the default type helper. TypeHelper.helper('json', jsonHelper) ``` More examples are provided in the JavaScript client library API Documentation. ## FIX / NVFIX delimiters In some cases the delimiter value used in FIX / NVFIX messages differs from the default `(\x01)`. In this situation you don’t have to implement a custom type helper; instead, the custom delimiter can be set: ```javascript TypeHelper.helper('nvfix').delimiter('%01') ``` --- # Performance Tips and Best Practices This chapter presents tips and techniques for writing high-performance applications with AMPS. This section presents principles and approaches that describe how to use the features of AMPS and the AMPS client libraries to achieve high performance and reliability. Specific techniques (for example, the details on how to write a message handler) are described in other parts of the AMPS documentation and referenced here. Other techniques require information specific to the application (for example, determining the minimum set of information required in a message), and are best done as part of your application design. All of the recommendations in this section are general guidelines. There are few, if any, universal rules for performance: at times, a design decision that is absolutely necessary to meet the requirements for an application might reduce performance somewhat. For example, your application might involve sending large binary data that cannot be incrementally updated. That application will use more bandwidth per message than an application that sends 100-byte messages with fields that can be incrementally updated. However, since the application depends on being able to deliver the binary payloads, this difference in bandwidth consumption is a part of the requirements for the application, not a design decision that can be optimized. ## Measure Performance and Set Goals The most important tools for creating high performance applications that use AMPS are clear goals and accurate measurement. Without accurate measurement, it's impossible to know whether a particular change has improved performance or not. Without clear goals, it's difficult to know whether a given result is sufficient, or whether you need to continue improving performance. 60East recommends that your measurements include baseline metrics for the part of your message processing that does not involve AMPS. As an example, imagine your task is to reduce the amount of time that elapses between when an order is sent and when the processed response is received from 100ms in total to 85ms in total. To achieve this reduction, you might first measure the processing that your application performs on the order. If that processing consumes 65ms, the most effective optimization may be to improve the order processing. On the other hand, if processing an order consumes 15ms, then optimizing message delivery or network utilization may be the most effective way to meet your goals. When measuring performance, simulate your production environment as closely as possible. For example, AMPS is highly parallelized, so sending a pattern of subscriptions and publishes from a single test client that would normally come from 20 clients will produce a very different performance profile. Likewise, AMPS can typically perform at rates that fill the available bandwidth. Performance measured on a 1GbE connection may be very different than performance measured over a 10GbE connection. Consider the characteristics of your data, and the number of messages you expect to store and process. A 1GB data set consisting of 1 million records will perform differently than a 1GB data set consisting of 10 million records, or a 1GB data set consisting of 100 records. When collecting information about performance, 60East recommends enabling persistence for the Statistics Database (`stats.db`), so you can easily collect historical data on both AMPS and the operating system. For example, a dip in performance correlated with high CPU and memory usage at the same time each day may be correlated with other activity on the system (such as cron jobs or close of business processing). In a situation like that, where the performance reduction is based on factors external to the AMPS application, the overall system metrics captured in `stats.db` can help you re-create the external state and understand the state of the system as a whole. AMPS collects the statistics in memory by default, and persisting that data into a database does not typically have a measurable effect on performance itself, but makes measuring and tuning performance much easier. For performance testing, 60East recommends using dedicated hardware for AMPS to eliminate the effects of other processes. If dedicated hardware is not available and other processes are consuming resources, 60East recommends disabling AMPS NUMA tuning to ensure that AMPS threads do not unnecessarily compete with other processes during performance tuning. ## Use HAClient and Heartbeating Where Appropriate Not every application that uses AMPS requires high availability and the ability to automatically fail over if connectivity is lost or an instance of AMPS is offline. For applications that do need automatic reconnection, 60East strongly recommends using the `HAClient` and setting heartbeating for the client to effectively detect disconnection. When using the `HAClient` and heartbeating, there are two important guidelines to follow: - Do not replace the disconnect handler on the `HAClient`. The disconnect handler is responsible for reconnection, resubscription, and so on. If you need to detect disconnection, use a connection state listener. - Set the interval for heartbeating to approximately one-half the time that the application can tolerate interruption in message flow. Notice that it's not possible for the `HAClient` to tell the difference between an interruption in message flow caused by a server going offline and interruptions caused by an increase in latency due to network saturation or so on, so the interval should be somewhat larger than the highest expected latency between AMPS and the application. Last, but not least, if the application uses asynchronous message handling, the interval should also be set to a value larger than the maximum amount of time expected for the message handler to process a single message. ## Simplify Message Format and Contents AMPS supports a wide range of message types, and is capable of filtering and processing large and complex messages. For many applications, the simplicity of being able to use messages that contain the full information is the most important consideration. For other applications, however, achieving the minimum possible latency and the maximum possible network utilization is important enough to warrant choosing a simplified message format. To simplify message contents, carefully consider the information that downstream processors require. If a downstream process will not use information in the message, there is no need to send the information. For example, consider an application that provides orders from a UI. In such an application, the object that represents the order often contains information relevant to the local state of the application that is not relevant to a downstream system. Rather than simply serializing the full object, your application may perform better if you serialize only the fields that a downstream system will take action on. To simplify message format, choose the simplest format that can convey the information that your application needs. The general principle is that the simpler the message format is, the more quickly AMPS and client libraries can parse messages of that type. Likewise, the more complicated the structure of each message is, the more work is required to parse the message. For the highest levels of performance, 60East recommends keeping the message structure simple and preferring message formats such as NVFIX, BFlat, or flattened JSON (structured as key/value pairs) as compared with more complicated formats such as XML or BSON. ## Measure Serialization and Deserialization When creating baseline performance numbers, measure serialization and deserialization performance independent of the AMPS server or client libraries. This can help you to: - Understand the baseline performance of creating and processing message data under ideal conditions (that is, where there is no application processing, networking, routing, etc. involved). - Easily compare the application-side performance of different message formats or different message layouts within a single format. When testing this performance, it is helpful to use data similar to the data that the application will actually process during a business day, at the volumes the application would typically process. This will help you understand the performance of serialization and deserialization for this specific application. For example, a library for working with a given message format might be less efficient when processing messages with a large number of string fields in a deeply-nested structure, but your application might exchange only numeric data in a relatively flat structure. Likewise, the library for a given format could be efficient for processing a small number of fields, but have lower performance for a message type with hundreds of fields. As with all performance testing, the more closely the test environment matches the actual data and volumes of a production environment, the more helpful those measurements will be for understanding system performance. ## Use Content Filtering Where Possible AMPS content filtering helps your application perform better by ensuring that your application only receives the messages that it needs. Wherever possible, we recommend using content filtering to precisely specify which messages your application needs. In particular, if at any point your application is receiving a message, parsing the message, and then determining whether to act on the message or not, 60East recommends using content filters to ensure that your application only receives messages that it needs to act on. ## Use Asynchronous Message Processing The synchronous message processing interface is straightforward, and presents a convenient interface for getting started with AMPS. However, the `MessageStream` used by the synchronous interface makes a full copy of each message and provides it from the background reader thread to the thread that consumes the message. This memory overhead and synchronization between the reader thread and consumer thread happens regardless of whether the application needs all of the header fields in the message or even processes the message. The `MessageStream` also does not take into account the speed at which your program is consuming messages, and will read messages into memory as fast as the network and processor allow. If your application cannot consume messages at wire speed, this can lead to increasing memory consumption as the application falls further behind the `MessageStream`. Most applications see improved performance by using a `MessageHandler`. With this approach, the `MessageHandler` does minimal work. If more extensive processing is needed, the `MessageHandler` dispatches the work to another thread: but it does this only when the work is necessary, and it only saves the part of the message needed to accomplish the work. ## Use Hash Indexes Where Possible for SOW Queries When querying a SOW, hash indexes on SOW topics are supported for exact matching on string data as described in the *AMPS User Guide*. A hash index can perform many times faster than a parallel query. If the query pattern for your application can take advantage of hash indexes, 60East recommends creating those hash indexes on your SOW topics. More recent versions of AMPS can use hash indexes for a wider variety of filters. When planning your queries, review the SOW queries section of the *AMPS User Guide* for the version you are using for guidelines on the optimizations available in that version. ## Use a Failed Write Handler and Exception Listener In many cases, particularly during the early stages of development, performance problems can point to defects in the application. Even after the application is tuned, monitoring for failure is important to keep applications running smoothly. 60East recommends always installing a failed write handler if your application is publishing messages. This will help you to quickly identify cases where AMPS is rejecting publishes due to entitlement failures, message type mismatches, or other similar problems. 60East recommends always installing an exception listener if your application is using asynchronous message processing. This will help you to identify and correct any problems with your message handler. An exception listener should typically log the message received and return. If recovery is needed, the listener should set a flag for another thread to process rather than attempting to recover on the thread that calls the exception listener. ## Reduce Bandwidth Requirements In many applications that use AMPS, network bandwidth is the single most important factor in overall performance. Your application can use bandwidth most efficiently by reducing message size. For example, rather than serializing an entire object, you might serialize only the fields that the remote process needs to act on, as mentioned above. Likewise, rather than sending one message that contains a collected set of information that processors will need to extract, consider sending a message in the units that processors will work with. This can reduce bandwidth to processors substantially. For example, rather than sending a single message with all of the activity for a single customer over a given period of time (such as a trading day), consider breaking out the record into the individual transactions for the customer. ### Tune Batch Size for SOW Queries As described in the section on [SOW Batch Size](/docs/amps-user-guide/sow-queries/batching-query-results), tuning the batch size for SOW queries can improve overall performance by improving network utilization. In addition, because the AMPS header is only parsed once per batch, a larger batch size can dramatically improve processing performance for smaller messages. The AMPS clients default to a batch size of `10`. This provides generally good performance for most transactional messages (such as order records or inventory records). For large messages, particularly messages greater than a megabyte in size, a batch size of `1` may reduce memory pressure in the client and improve performance. With smaller messages (for example, message sizes of a few hundred bytes), 60East recommends measuring performance with larger batch sizes such as `50` or `100`. For large messages, reducing the batch size may improve overall performance by requiring less memory consumption on the AMPS server. ### Conflate Fast-Changing Information If your data source publishes information faster than your clients need to consume it, consider using a conflated topic. For example, in a system that presents a user interface and displays fast-moving data, it is common for the data to change at a rate faster than the user interface can format and render the data. In this case, a conflated topic can both reduce bandwidth and simplify processing in the user interface. ### Minimize Bandwidth for Updates If your application uses a SOW and processes frequent updates, consider using delta publish and delta subscribe to reduce the size of the messages transmitted. These features are designed to minimize bandwidth while still providing full-fidelity data streams. ### Conflate Queue Acknowledgments The AMPS clients include the ability to conflate acknowledgments back to AMPS as queue messages are processed. Using these features, with an appropriate `max_backlog`, can reduce the amount of network traffic required for acknowledgments. ### Use a Transaction Log When Monitoring Publish Failures When a topic is not covered by a transaction log, AMPS returns acknowledgment messages for every publish that requests one. This ensures that each message is acknowledged, even when AMPS has no persistent record of the messages in the topic. However, acknowledging each message requires more network traffic for each publish message. When a topic is covered by a transaction log, AMPS conflates persisted acknowledgments. Conflation is possible in this case because AMPS has a full record of the messages and does not have to store additional state to conflate the acknowledgments. With conflated acknowledgments, AMPS will send a success acknowledgment periodically that covers all messages up to that point. If a message fails, AMPS immediately sends the conflated success acknowledgment for all previous messages and the failure acknowledgment for the failed message. ### Combine Conflation and Deltas In many cases, using an approach that combines delta publishes to a SOW with delta subscriptions to a conflated topic can dramatically reduce bandwidth to the application with no loss of information. ## Limit Unnecessary Copies One of the most effective ways to increase performance is to limit the amount of data copied within your application. For example, if your message handler submits work to a set of processors that only use the `Data` and `Bookmark` from a `Message`, create a data structure that holds only those fields and copy that information into instances of that data structure rather than copying the entire `Message`. While this approach requires a few extra lines of code, the performance benefits can be substantial. When publishing messages to AMPS, avoid unnecessary copies of the data. For example, if you have the data in a byte array, use the `publish` methods that use a byte array rather than converting the data to a string unnecessarily. Likewise, if you have the data in the form of a string, avoid converting it to a byte array where possible. ## Manage Publish Stores When using a publish store, the Client holds messages until they are acknowledged as persisted by AMPS, as determined by the replication configuration for the AMPS instance. In the event that an instance with `sync` replication goes offline, the publish store for the Client will grow, since the messages are not being fully persisted. To avoid this problem, 60East recommends that an instance that uses `sync` replication always configure Actions to automatically downgrade the replication link if the remote instance goes offline for a period of time, and upgrade the link when the remote instance comes back online. Further, 60East recommends that, where possible, a publisher is provisioned with enough storage to hold its complete publish stream for the amount of time that a destination may be offline or unavailable without downgrading from `sync` replication to `async` replication. For example, if the server considers a downstream system to be unreachable if it has not acknowledged a replicated message in 60 seconds, and the server checks this threshold every 10 seconds, then a publisher should plan that, at any time, the publisher may need to retain approximately 70 seconds worth of published messages. This is calculated as the 60 seconds threshold that the server has established for a destination to run behind, plus the 10 second interval at which the server checks whether the destination is within the threshold. Also notice that, with a configuration like this, a downstream replication destination could run as much as 59 seconds behind indefinitely. A publisher should be provisioned to be able to run effectively in a "worst case" (or nearly "worst case") scenario for an extended period of time. See the *High Availability and Replication* chapter in the *AMPS User Guide* for more information on replication, sync and async acknowledgment modes, and the Actions used to manage replication. ## Use the Server Logs to Help Troubleshoot When troubleshooting problems with an application that uses AMPS, the server-side logs often provide the most helpful information. For example, `trace` level logging shows the data that is flowing through AMPS. Log messages at `info` level show events as incoming connections, commands from clients, and so on. When questions arise about how the server and application interact, the server logs often contain the information. 60East recommends that an AMPS instance used for development and testing log at `trace` level, and that a server used for production log at `info` level, with the ability to log at `trace` level when necessary for investigating any problems that may arise. When a command does not have the expected result, or an application reports an error, the fastest way to understand the problem is often to review the `trace` level logging for the instance. See the *AMPS User Guide* for details on configuring logging and common patterns for searching for information in AMPS logs. ## Work with 60East as Necessary 60East offers performance advice adapted for your specific usage through your support agreement. Once you've set your performance goals, worked through the general best practices and applied the practices that make sense for your application, 60East can help with detailed performance tuning, including recommendations that are specific to your use case and performance needs. --- # Promises and Message Handlers The first time a command causes an instance of the Client to connect to AMPS, the client creates a `WebSocket` connection that runs asynchronously. This asynchronous connection is responsible for processing incoming messages from AMPS, which includes both messages that contain data and acknowledgments from the server. When you execute a command on the AMPS client, the `execute()` method creates and returns a `Promise` object that typically waits for an acknowledgment from the server and then resolves with the id of the command. (The exception to this is `Client.publish()`. For performance, the publish command does not wait for an acknowledgment from the server before returning.) Message handlers provided for message processing must be aware of the following considerations: - For maximum performance, do as little work in the message handler as possible. For example, if you use the contents of the message to perform an extensive calculation, a message handler that passes data into a `WebWorker` instance will typically perform better than a message handler that does this calculation in place. - While your message handler is running, the connection that calls your message handler is no longer receiving messages. This makes it easier to write a message handler because you know that no other messages are arriving from the same subscription. However, this also means that you cannot use the same client that called the message handler to send commands to AMPS. Acknowledgments from AMPS cannot be processed and your application will block while waiting for the acknowledgment. Instead, enqueue the command in a work queue to be processed by a separate worker or use a different client object to submit the commands. --- # Using Queues AMPS message queues provide a high-performance way of distributing messages across a set of workers. The *AMPS User Guide* describes AMPS queues in detail, including the features of AMPS referred to in this chapter. This chapter does not describe AMPS queues in detail, but instead explains how to use the AMPS JavaScript client with message queues. To publish messages to a message queue, publishers simply publish to any topic that is collected by the queue. There is no difference between publishing to a queue and publishing to any other topic, and a publisher does not need to be aware that the topic will be collected into a queue. Subscribers must be aware that they are subscribing to a queue, and acknowledge messages from the queue when the message is processed. ## Backlog and Smart Pipelining AMPS queues are designed for high-volume applications that need minimal latency and overhead. One of the features that helps performance is the *subscription backlog* feature, which allows applications to receive multiple messages at a time. The subscription backlog sets the maximum number of unacknowledged messages that AMPS will provide to the subscription. When the subscription backlog is larger than `1`, AMPS delivers additional messages to a subscriber before the subscriber has acknowledged the first message received. This technique allows subscribers to process messages as fast as possible, without ever having to wait for messages to be delivered. The technique of providing a consistent flow of messages to the application is called *smart pipelining*. ### Subscription Backlog The AMPS server determines the backlog for each subscription. An application can set the maximum backlog that it is willing to accept with the `max_backlog` option. Depending on the configuration of the queue (or queues) specified in the subscription, AMPS may assign a smaller backlog to the subscription. If no `max_backlog` option is specified, AMPS uses a `max_backlog` of `1` for that subscription. In general, applications that have a constant flow of messages perform better with a `max_backlog` setting higher than `1`. The reason for this is that, with a backlog greater than `1`, the application can always have a message waiting when the previous message is processed. Setting the optimum `max_backlog` is a matter of understanding the messaging pattern of your application and how quickly your application can process messages. To request a `max_backlog` for a subscription, you explicitly set the option on the subscribe command, as shown below: ```javascript const command = new Command('subscribe') .topic('my-queue') .options('max_backlog=10') ``` ### Acknowledging Messages For each message delivered on a subscription, AMPS counts the message against the subscription backlog until the message is explicitly acknowledged. In addition, when a queue specifies `at-least-once` delivery, AMPS retains the message in the queue until the message expires or until the message has been explicitly acknowledged and removed from the queue. From the point of view of the AMPS server, acknowledgment is implemented as a `sow_delete` from the queue with the bookmarks of the messages to remove. The AMPS JavaScript client provides several ways to make it easier for applications to create and send the appropriate `sow_delete`. ### Automatic Acknowledgment The AMPS client allows you to specify that messages should be automatically acknowledged. When this mode is on, AMPS acknowledges the message automatically if the message handler returns without throwing an exception. AMPS batches acknowledgments created with this method, as described in the following section. To enable automatic acknowledgment, use the `Client.autoAck()` method. ```javascript client.autoAck(true) // enable AutoAck ``` ### Message Convenience Method The AMPS JavaScript client provides a convenience method, `Client.ack()`, on delivered messages. When the application is finished with the message, the application simply calls `Client.ack()` on the message. For messages that originated from a queue with `at-least-once` semantics, this adds the bookmark from the message to the batch of messages to acknowledge. For other messages, this method has no effect. ```javascript // Add the message to the next acknowledgment batch client.ack(message) ``` ### Acknowledgment Batching The AMPS JavaScript client automatically batches acknowledgments when either of the convenience methods is used. Batching acknowledgments reduces the number of round-trips to AMPS, which reduces network traffic and improves overall performance. AMPS sends the batch of acknowledgments when the number of acknowledgments exceeds a specified size, or when the amount of time since the last batch was sent exceeds a specified timeout. You can set the number of messages to batch and the maximum amount of time between batches, as shown below: ```javascript client.ackBatchSize(10) // Send batch after 10 messages client.ackTimeout(1000) // ... or 1 second ``` The AMPS JavaScript client is aware of the subscription backlog for a subscription. When AMPS returns the acknowledgment for a subscription that contains queues, AMPS includes information on the subscription backlog for the subscription. If the requested batch size is larger than the subscription backlog, the AMPS JavaScript client adjusts the requested batch size to match the subscription backlog. ### Returning a Message to the Queue A subscriber can also explicitly release a message back to the queue. AMPS returns the message to the queue, and redelivers the message just as though the lease had expired. To do this, the subscriber sends a `sow_delete` command with the bookmark of the message to release and the cancel option: ```javascript client.execute( new Command('sow_delete') .topic(message.header.topic()) .bookmark(message.header.bookmark()) .options('cancel') ) ``` When using automatic acknowledgments, AMPS will cancel a message if an exception is thrown from the message handler. ### Manual Acknowledgment To manually acknowledge processed messages and remove the messages from the queue, applications use the `sow_delete` command with the bookmarks of the messages to remove. Notice that AMPS only supports using a bookmark with `sow_delete` when removing messages from a queue, not when removing records from a SOW. For example, given a `Message` object to acknowledge and a client, the code below acknowledges the message. (simple-acknowledge)= ```javascript const acknowledgeSingle = async (client, message) => { return client.execute( new Command('sow_delete') .topic(message.header.topic()) .bookmark(message.header.bookmark()) ) } ``` **Example (CHAPTER_NUMBER).(SIMPLE_QUEUE_ACK):** *Simple queue acknowledgment* In the example above, the program creates a `sow_delete` command, specifies the topic and the bookmark, and then sends the command to the server. While this method works, creating and sending an acknowledgment for each individual message can be inefficient if your application is processing a large volume of messages. Rather than acknowledging each message individually, your application can build a comma-delimited list of bookmarks from the processed messages and acknowledge all of the messages at the same time. In this case, it's important to be sure that the number of messages you wait for is less than the maximum backlog -- the number of messages your client can have unacknowledged at a given time. Notice that both automatic acknowledgment and the convenience method `Client.ack()` take the maxiumum backlog into account. --- # Using Queues AMPS message queues provide a high-performance way of distributing messages across a set of workers. The *AMPS User Guide* describes AMPS [Queues](/docs/amps-user-guide/queues) in detail, including the features of AMPS referred to in this chapter. This chapter does not describe AMPS queues in detail, but instead explains how to use the AMPS JavaScript client with message queues. To publish messages to a message queue, publishers simply publish to any topic that is collected by the queue. There is no difference between publishing to a queue and publishing to any other topic, and a publisher does not need to be aware that the topic will be collected into a queue. Subscribers must be aware that they are subscribing to a queue, and acknowledge messages from the queue when the message is processed. --- # Regular Expression Subscriptions Regular Expression (Regex) subscriptions allow a regular expression to be supplied in the place of a topic name. When you supply a regular expression, it is as if a subscription is made to every topic that matches your expression, including topics that do not yet exist at the time of creating the subscription. To use a regular expression, simply supply the regular expression in place of the topic name in the `subscribe()` call. For example: ```javascript await client.subscribe(message => { ... }, 'orders.*') ``` In this example, messages on topics `orders-north-america`, `orders-europe`, and `new-orders` would match the regular expression, and those messages would all be sent to the message handler function. As in the example, you can use the `message.header.topic()` method to determine the actual topic of the message sent to the function. --- # Replacing Disconnect Handling In some cases, an application does not want the AMPS `Client` to reconnect, but instead wants to take a different action if disconnection occurs. For example, a stateless publisher that sends ephemeral data (such as telemetry or prices) may want to exit with an error if the connection is lost rather than risk falling behind and providing outdated messages. Often, in this case, a monitoring process will start another publisher if a publisher fails, and it is better for a message to be lost than to arrive late. To cover cases where the application has unusual needs, the AMPS client library allows an application to provide custom disconnect handling. Your application gets to specify exactly what happens when a disconnect occurs by supplying a function to `client.setDisconnectHandler()`, which is invoked whenever a disconnect occurs. This may be helpful for situations where a particular connection needs to do something completely different than reconnecting or failing over to another AMPS server. :::warning Setting the disconnect handler completely replaces the disconnection failover behavior for a `Client`. ::: The example below shows the basics: ```javascript showLineNumbers /** * Call this function to establish a connection to AMPS. */ const connectToAMPS = async () => { try { await client.connect('ws://localhost:9000/amps/json') // Successfully connected } catch (err) { // Can't establish connection } } // create a client object const client = new Client('disconnect-handler-demo') /* * disconnectHandler() method is called to supply a function for use * when AMPS detects a disconnect. At any time, this function may be * called by AMPS to indicate that the client has disconnected from * the server, and to allow your application to choose what to do * about it. */ client.disconnectHandler( /** * Our disconnect handler’s implementation begins here. * * Any custom disconnect handler would be application-specific * so, for demonstration purposes, we simply log the error. * * Notice that this disconnect handler replaces all other * disconnect handling behavior. When the client is disconnected, * it will simply log an error to the console. The client * will not reconnect or take any other action. */ (client, error) => { console.log(error) } ) // We begin by connecting and subscribing connectToAMPS() ``` --- # Returning a Message to the Queue A subscriber can also explicitly release a message back to the queue. AMPS returns the message to the queue, and redelivers the message just as though the lease had expired. To do this, the subscriber sends a `sow_delete` command with the bookmark of the message to release and the `cancel` option. ```javascript showLineNumbers client.execute( new Command('sow_delete') .topic(message.header.topic()) .bookmark(message.header.bookmark()) .options('cancel') ) ``` When using automatic acknowledgments, AMPS will cancel a message if an exception is thrown from the message handler. --- # Setting Batch Size The AMPS clients include a batch size parameter that specifies how many messages the AMPS server will return to the client in a single batch when returning the results of a SOW query. The 60East clients set a batch size of 10 by default. This batch size works well for common message sizes and network configurations. Adjusting the batch size may produce better network utilization and produce better performance overall for the application. The larger the batch size, the more messages AMPS will send to the network layer at a time. This can result in fewer packets being sent, and therefore less overhead in the network layer. The effect on performance is generally most noticeable for small messages, where setting a larger batch size will allow several messages to fit into a single packet. For larger messages, a batch size may still improve performance, but the improvement is less noticeable. In general, 60East recommends setting a batch size that is large enough to produce few partially-filled packets. Bear in mind that AMPS holds the messages in memory while batching them, and the client must also hold the messages in memory while receiving the messages. Using batch sizes that require large amounts of memory for these operations can reduce overall application performance, even if network utilization is good. For smaller message sizes, 60East recommends using the default batch size, and experimenting with tuning the batch size if performance improvements are necessary. For relatively large messages (especially messages with sizes over 1MB), 60East recommends explicitly setting a batch size of 1 as an initial value, and increasing the batch size only if performance testing with a larger batch size shows improved network utilization or faster overall performance. --- # SOW and Subscribe Imagine an application that displays real time information about the position and status of a fleet of delivery vans. When the application starts, it should display the current location of each of the vans along with their current status. As vans move around the city and post other status updates, the application should keep its display up to date. Vans upload information to the system by posting messages to the `van_location` topic, configured with a key of `van_id` on the AMPS server. In this application, it is important to not only stay up-to-date on the latest information about each van, but to ensure all of the active vans are displayed as soon as the application starts. Combining a SOW with a subscription to the topic is exactly what is needed, and that is accomplished by the `Client.sowAndSubscribe()` method, or by executing a `sow_and_subscribe` command. ### sowAndSubscribe() First, let's look at an example that uses the convenience method: ```javascript showLineNumbers const reportVanPosition = async client => { /** * sowAndSubscribe() method to begin receiving information about * all of the active delivery vans in the system. All of the vans * in the system now are returned as Message objects whose * `message.header.command()` method returns `sow`. New messages * coming in are returned as Message objects whose * `message.header.command()` method returns `p` (publish). */ return client.sowAndSubscribe( // Message handler message => { const cmdName = message.header.command() if (cmdName === 'sow' || cmdName === 'p') { /** * For each of these messages we call addOrUpdateVan(), * that presumably adds the van to our application’s * display. As vans send updates to the AMPS server, * those are also received by the client because of the * subscription placed by sowAndSubscribe(). Our * application does not need to distinguish between * updates and the original set of vans we found via the * SOW query, so we use addOrUpdateVan() to display * the new position of vans as well. */ addOrUpdateVan(message) } else if (cmdName === 'oof') { removeVan(message) } }, 'van_location', // SOW Topic '/status = "ACTIVE"', // Filter // Additional parameters { batchSize: 100, options: 'oof' } ) } ``` ### Execute a Command Now we will look at an example that uses the `Command` interface with the `Client.execute()` method: ```javascript showLineNumbers // Message Handler const onVanPositionUpdate = message => { const cmdName = message.header.command() if (cmdName === 'sow' || cmdName === 'p') { addOrUpdateVan(message) } else if (cmdName === 'oof') { removeVan(message) } } const reportVanPosition = async client => { // Command object to execute const cmd = new Command('sow_and_subscribe') cmd.topic('van_location') cmd.filter('/status = "ACTIVE"') cmd.batchSize(100) cmd.options('oof') // Execute the command with the above message handler return client.execute(cmd, onVanPositionUpdate) } ``` Notice that the two forms have the same result. ### OOF Messages In the above examples we specified the `oof` option to the command. Setting this option causes AMPS to send *Out-of-Focus* (OOF) messages for the topic. OOF messages are sent when an entry that was sent to us in the past no longer matches our query. This happens when an entry is removed from the SOW cache via a `sow_delete` operation, when the entry expires (as specified by the expiration time on the message or by the configuration of that topic on the AMPS server), or when the entry no longer matches the content filter specified. In our case, if a van's status changes to something other than ACTIVE, it no longer matches the content filter, and becomes out of focus. When this occurs, a message is sent with `Command` set to `oof`. We use OOF messages to remove vans from the display as they become inactive, expire, or are deleted. --- # State of the World (SOW) AMPS State of the World (SOW) allows you to automatically keep and query the latest information about a topic on the AMPS server, without building a separate database. Using SOW lets you build impressively high-performance applications that provide rich experiences to users. The AMPS JavaScript client lets you query SOW topics and subscribe to changes with ease. ## Performing SOW Queries To begin, we will look at a simple example of issuing a SOW query. ```javascript showLineNumbers const onMessage = message => { switch (message.header.command()) { case 'group_begin': console.log('--- Begin SOW Results ---') break case 'sow': console.log(message.data) break case 'group_end': console.log('--- End SOW Results ---') break } } const queryId = await client.sow( onMessage, // Message handler 'orders', // SOW Topic '/symbol="ROL"' // Filter ) ``` In the example above, we invoke `Client.sow()` to initiate a SOW query on the `orders` topic, for all entries that have a symbol of `'ROL'`. As usual, the `Client.sow()` method returns a `Promise` object that resolves with the query ID. As the query executes, the message handler function is invoked for each matching entry in the topic. Messages containing the data of matching entries have a `Command` of value `sow`, so as those arrive, we write them to the console. --- # Subscriptions Messages published to a topic on an AMPS server are available to other clients via a subscription. Before messages can be received, a client must subscribe to one or more topics on the AMPS server so that the server will begin sending messages to the client. The server will continue sending messages to the client until the client unsubscribes, or the client disconnects. With content filtering, the AMPS server will limit the messages sent to only those messages that match a client-supplied filter. In this chapter, you will learn how to subscribe, unsubscribe, and supply filters for messages using the AMPS JavaScript client. ## Subscribing to a Topic The AMPS client makes it simple to subscribe to a topic. You call `Client.subscribe()` with the message handler, the topic to subscribe to and the parameters for the subscription. The client submits the subscription to AMPS and returns a `Promise` object that resolves with the subscription identifier. Received messages are asynchronously delivered to the message handler function. Below is a short example: ```javascript showLineNumbers /** * Let's create a message handler function. It will be invoked * when AMPS delivers messages to the subscription. * Within this handler, we process messages. In this case, * we simply print the contents of the message. */ const onMessage = message => console.log(message.data) // let's create a Client that is connected to an AMPS server. const client = new Client('test') await client.connect('wss://127.0.0.1:9007/amps/json') /** * Here we subscribe to the topic "messages". We do not provide * a filter, so AMPS does not content-filter the topic. * The command resolves with the id of the subscription. */ const subId = await client.subscribe(onMessage, 'messages') console.log('subId:', subId) ``` AMPS creates an asynchronous subscription that receives messages and calls the message handler only if there's a new message. This means that the client application as a whole can continue to receive messages while you are doing processing work, by stacking them in the execution queue. The simple method described above is provided for convenience. The AMPS JavaScript client provides convenience methods for the most common form of AMPS commands. The client also provides an interface that allows you to have precise control over the command. Using that interface, the example above becomes: ```javascript showLineNumbers /** * Let's create a message handler function. It will be invoked * when AMPS delivers messages to the subscription. * Within this handler, we process messages. In this case, * we simply print the contents of the message. */ const onMessage = message => console.log(message.data) // let's create a Client that is connected to an AMPS server. const client = new Client('test') await client.connect('wss://127.0.0.1:9007/amps/json') /** * Here we create a Command object for the subscribe * command, specifying the topic "messages". We do not * provide a filter, so AMPS does not content-filter * the topic. */ const cmd = new Command('subscribe').topic('messages') /** * We execute the Command using the execute() method. * The command execution resolves with the id of the * subscription. */ const subId = await client.execute(cmd, onMessage) console.log('subId: ', subId) ``` The `Command` interface allows you to precisely customize the commands you send to AMPS. For flexibility and ease of maintenance, 60East recommends using the `Command` interface (rather than a named method) for any command that will receive messages from AMPS. For publishing messages, there can be a slight performance advantage to using the named commands where possible. --- # Unhandled Errors When using the asynchronous interface, errors can occur that are not thrown to the user. For example, when a message with invalid JSON data was received, the error occurs in the process of parsing its data inside of the AMPS JavaScript Client. Consider the following example: ```javascript showLineNumbers const onMessage = message => console.log(message.data) await client.subscribe(onMessage, 'pokes', '/Pokee LIKE ' + userId) ``` In this example, we set up a subscription to wait for messages on the pokes topic, whose Pokee tag begins with our user ID. When messages arrive, we print a message out to the console. Inside of the AMPS client, the client received a message that contains invalid JSON data that cannot be parsed. When the error occurs, there is no handler for it to be reported to, and by default it is ignored. In applications where it is important to deal with every issue that occurs in using AMPS, you can set an error handler via `Client.errorHandler()` that receives these otherwise-unhandled errors and exceptions. Making the modifications shown in the example below, to our previous example, will allow those errors to be caught and handled. In this case we are simply printing those caught errors out to the console. ```javascript showLineNumbers const onMessage = message => console.log(message.data) // This time, let's add the error handler client.errorHandler(err => console.error('Error occurred:', err)) await client.subscribe(onMessage, 'pokes', '/Pokee LIKE ' + userId) ``` In this example we have added a call to `client.errorHandler()`, registering a simple function that writes the text of the error out to the console. If errors are thrown in the message handler, those errors are written to the console. --- # Ending Subscriptions With asynchronous message processing, when a subscription is successfully made, messages will begin flowing to the message handler function and the `Client.subscribe()` call returns a Promise object that resolves with a unique string that serves as the identifier for this subscription. A `Client` can have any number of active subscriptions, and this string is used to refer to the particular subscription we have made here. For example, to unsubscribe, we simply pass in this identifier: ```javascript showLineNumbers const client = new Client('exampleClient') await client.connect('wss://localhost:9000/amps/json') const subId = await client.subscribe(onMessagePrinter, 'messages') // when the program is done with the subscription, unsubscribe await client.unsubscribe(subId) console.log('Unsubscribed') ``` In this example, as in the previous section, we use the `Client.subscribe()` method to create a subscription to the `messages` topic. When our application is done listening to this topic, it unsubscribes by passing in the `subId` passed from the successfully resolved Promise of `subscribe()`. After the subscription is removed, no more messages will flow into our `onMessagePrinter` function. AMPS also accepts the keyword `all` to unsubscribe all subscriptions for the client. --- # Welcome to the AMPS Python Client This guide provides information you need to get started with the AMPS Python client. It focuses specifically on the client and does not cover AMPS itself in detail. For an overview of AMPS and instructions on setting up your development environment, see the [Introduction to AMPS](/docs/intro-guide/intro) guide. :::tip This guide assumes that you have a development environment for Python and access to an AMPS server using the configuration provided with the Python samples (in the full source distribution of the client). ::: --- # Acknowledging Messages For each message delivered on a subscription, AMPS counts the message against the subscription backlog until the message is explicitly acknowledged. In addition, when a queue specifies `at-least-once` delivery, AMPS retains the message in the queue until the message expires or until the message has been explicitly acknowledged and removed from the queue. From the point of view of the AMPS server, acknowledgment is implemented as a `sow_delete` from the queue with the bookmarks of the messages to remove. The AMPS Python client provides several ways to make it easier for applications to create and send the appropriate `sow_delete`. ## Automatic Acknowledgment The AMPS client allows you to specify that messages should be automatically acknowledged. When this mode is on, AMPS acknowledges the message automatically in the following cases: - **Asynchronous Message Processing Interface** - The message handler returns without throwing an exception. - **Synchronous Message Processing Interface** - The application requests the next message from the `MessageStream`. AMPS batches acknowledgments created with this method, as described in the following section. To enable automatic acknowledgment, use the `set_auto_ack()` method. ```python client.set_auto_ack(True) # enable AutoAck ``` ## Message Convenience Method The AMPS Python client provides a convenience method, `ack()`, on delivered messages. When the application is finished with the message, the application simply calls `ack()` on the message. (This, in turn, provides the topic and bookmark to the `ack()` method of the client that received the message.) For messages that originated from a queue with `at-least-once` semantics, this adds the bookmark from the message to the batch of messages to acknowledge. For other messages, this method has no effect. ```python message.ack() # Add this message to the next # acknowledgment batch. ``` ## Acknowledgment Batching The AMPS Python client automatically batches acknowledgments when either of the convenience methods is used. Batching acknowledgments reduces the number of round-trips to AMPS, which reduces network traffic and improves overall performance. AMPS sends the batch of acknowledgments when the number of acknowledgments exceeds a specified size, or when the amount of time since the last batch was sent exceeds a specified timeout. You can set the number of messages to batch and the maximum amount of time between batches, as shown below: ```python client.set_ack_batch_size(10) # Send batch after 10 messages client.set_ack_timeout(1000) # ... or 1 second ``` The AMPS Python client is aware of the subscription backlog for a subscription. When AMPS returns the acknowledgment for a subscription that contains queues, AMPS includes information on the subscription backlog for the subscription. If the requested batch size is larger than the subscription backlog, the AMPS Python client adjusts the requested batch size to match the subscription backlog. 60East recommends tuning the batch size to improve application performance. A value of 1/3 of the smallest `max_backlog` value is a good initial starting point for testing. 60East does not recommend setting the batch size larger than 1/2 of the `max_backlog` value without testing to ensure that the application does not run out of messages to process while the acknowledgment is being sent to AMPS. --- # Advanced Topics ## Implementing Message Handlers in C or C++ The AMPS Python client provides a wrapper that works with the python `ctypes` module to allow you to create message handlers in C or C++ and expose them to Python. This can improve performance in the message handler. When you use this technique, messages are delivered directly from the C++ client to your message handler: there is no Python code involved in handling the messages. To use this capability, you: 1. Create a message handler with C linkage, and compile that message handler into a shared library. 2. In your Python program, use the `ctypes` module to load the library. 3. Construct an instance of `CMessageHandler`, a wrapper object that holds a pointer to the message handler function and the user data to be provided to the handler during each call. 4. Pass the `CMessageHandler` to any method that expects a message handler. The AMPS Python client registers the pointer and user data you provide as a C++ message handler. Once the handler is registered, no Python code is called when providing messages to the handler. ## Implementing the Handler To use this capability, you create a message handler that exposes a function with the following signature having C linkage: ```cpp showLineNumbers extern "C" void my_message_handler( AMPS::Message &message, void *userdata ); ``` Notice that this signature is the same signature used by message handlers in the AMPS C++ client. You implement the function and compile it into a shared library or DLL, using the instructions provided with your Python implementation. For details on the C++ client, you can install the client itself, or consult the [C++ API documentation](https://devnull.crankuptheamps.com/documentation/api/cpp/5.3.4.5/index.html). ## Loading and Using the Handler Once you've compiled the library, you use the `ctypes` module to load the library. You then create an instance of the message handler wrapper, and pass that wrapper to the AMPS client methods, as shown below: ```python showLineNumbers import ctypes import AMPS ... # assumes that client is already created and connected # load the shared object dll = ctypes.CDLL("./libmymessagehandler.so") # create a handler that points to the underlying C function # and bind the user data to that handler. handler = AMPS.CMessageHandler(dll.my_message_hander, "user data") # handler can be used anywhere you would use a message handler client.subscribe(handler, "myTopic") client.set_last_chance_message_handler(handler) # and so it goes ``` The `AMPS.CMessageHandler` type accepts a pointer to a message handler with the signature shown above and a Python object that can be marshalled into a native C type through the `ctypes` interface. Once marshalled, the object will be cast to a `void *` and provided in the `userdata` parameter of the message handler. Marshalling the `userdata` parameter follows the `ctypes` module conventions. If you need to explicitly control the way an object is marshalled, you can construct one of the `ctypes` objects and pass that new object into the method. ## Using the C++ Client While the AMPS Python client provides enough performance for a wide variety of applications, in some cases, using the underlying C++ implementation can provide extra performance. The AMPS Python client works with the `ctypes` module to allow you to pass the underlying C++ client to an arbitrary function, effectively allowing you to integrate C++ code directly into your Python program. Consider using the C++ client directly when latency is at a premium or when your application works directly with C++. For example, you might you use the client directly when: - You are assembling messages from a C++ library without a Python binding - You need to customize client behavior that is implemented in C++ (for example, implementing a custom SubscriptionManager or BookmarkStore) - Your application needs to execute a set of commands with AMPS with minimal latency. For example, you might need to publish an array of values as individual messages with as little latency as possible. In this case, using the underlying C++ client directly can reduce latency. To use the underlying C++ client, you: 1. Create a function with C linkage, and compile that function into a shared library. One of the parameters of the function should be a reference to an `AMPS::Client`. 2. In your Python program, use the `ctypes` module to load the library. 3. Call the function on the library, passing the appropriate parameters for the C function. ### Implementing the C++ Function The only requirement on the C++ function is that it have C linkage and that one of the parameters is a reference to an `AMPS::Client`. By convention, 60East recommends that the first parameter is the `AMPS::Client`. However, this is not a requirement of the interface. For example, a function that simply takes an `AMPS::Client` has the following signature: ```cpp showLineNumbers extern "C" void configure_client(AMPS::Client& client); ``` While a function that takes a client, a topic, and a pointer to data to be published might have the following signature: ```cpp showLineNumbers extern "C" void publish_data( AMPS::Client& client, const char * topic, const char * data ); ``` The `ctypes` module provides a standard `AMPS::Client` to these functions. Although the `Client` has been created by Python code, there is nothing Python-specific about the object within the C++ function. You can use the `Client` just as you would any other `Client` object. You can also use the `ctypes` binding with `AMPS::HAClient`, as shown below: ```cpp showLineNumbers extern "C" void install_server_chooser(AMPS::HAClient& client); ``` Since the `ctypes` module passes the data through the C ABI, the module is not able to perform extensive type checking on C++ types. Your Python code must be careful to pass only objects of the appropriate type, or you may cause a segmentation violation. For example, if a method expecting an `HAClient` receives a `Client` and calls `connectAndLogon` (which is not a method provided by Client), your program will likely crash. :::warning The `ctypes` module has a few important caveats. The `ctypes` module does not provide strong type-safety guarantees for C++ classes. It is your responsibility to ensure that you call methods with objects of the appropriate type. The `ctypes` module calls your function through an extern "C" interface. C++ exceptions cannot be propagated out of a function with C binding. You must catch all exceptions that may be thrown, or your application will likely crash. ::: ### Loading and Using the Function Once you've compiled the library, you use the `ctypes` module to load the library. You can then call the function directly from Python, using the name of the C function and passing the appropriate arguments. Let's look at a simple example. For this example, assume that you have compiled a module named `module.so` with the following function: ```cpp showLineNumbers extern "C" void publish_message(AMPS::Client& client, const char* topic, const char* data) { try { if (&client && topic && data) { client.publish(topic,data); } } catch (AMPS::Exception& e) { /* Handle error reporting and recovery logic */ } } ``` You can load the module and call the function as shown below: ```python showLineNumbers import ctypes module = ctypes.CDLL("module.so") client = AMPS.Client("client") client.connect("tcp://localhost:9007/amps/json") client.logon() module.publish_data(client, "my_topic", "some_data") ``` The `ctypes` module handles type conversions between Python and C types. In this case, the module passes the underlying Python client as the first argument of the C function. The two Python strings are passed as NULL-terminated `char *` arrays. The `ctypes` module also handles more complicated signatures and correctly passes arrays. For example, you could implement a method that publishes an array of Python values as follows: ```cpp showLineNumbers extern "C" void vector_publish(AMPS::Client& client, const char* topic, const char** data, size_t vector_length) { try { if (topic && &client) { for (;vector_length;--vector_length,++data) { client.publish(topic,*data); } } } catch (AMPS::Exception& e) { /* Handle reporting and recovery logic */ } } ``` You could then use this function from Python as follows: ```python showLineNumbers import ctypes module = ctypes.CDLL("module.so") client = AMPS.Client("client") client.connect("tcp://localhost:9007/amps/json") client.logon() TOPIC = "topic" DATA = [{"data":x, "string_data":"string_data"} for x in range(5)] # initialize vector of data to publish by # dumping the dictionaries to JSON strings vector = [json.dumps(data) for data in DATA[1:]] # Set up the parameters to be passed to a C # function as explained in the ctype documentation param = (ctypes.c_char_p * len(vector))() param[:] = vector # Call the function module.vector_publish(client,TOPIC, param,len(param)) ``` The sample above creates an array of dictionaries and creates an array of JSON objects from those dictionaries. In this case, it is important for us to control how the array of JSON objects is passed to the C function. We need to pass an array of C-style strings, that is, `const char**`. To control how the array is marshalled, the sample creates an object that knows how to translate between a Python array and `const char**`, then assigns the array to that object (see the `ctype` documentation for full details). Once we have that object, we simply call the `vector_publish` function. None of the Python infrastructure is visible to the `vector_publish` function: that function is able to use the provided data as native C++ data. ## Transport Filtering The AMPS Python client offers the ability to filter incoming and outgoing messages in the format they are sent and received on the network. This allows you to inspect or modify outgoing messages before they are sent to the network, and incoming messages as they arrive from the network. This can be especially useful when using SSL connections, since this gives you a way to monitor outgoing network traffic before it is encrypted, and incoming network traffic after it is decrypted. To create a transport filter, you create a callable that expects a string that contains the raw data, and a direction parameter indicating whether the string is output or not. For example, the following function simply prints the direction and data: ```python showLineNumbers def printing_filter(data, direction): if direction: print(f"INCOMING ---> {data}") else: print(f"OUTGOING ---> {data}") ``` You then register the filter by calling `set_transport_filter` with the callable, as shown below. ```python showLineNumbers # client is an AMPS client client.set_transport_filter(printing_filter) ``` Notice that the transport filter function is called with the verbatim contents of data received from AMPS. This means that, for incoming data, the function may not be called precisely on message boundaries, and that the binary length encoding used by the client and server will be presented to the transport filter. ## Working with Binary Data A `Message` object contains two methods for retrieving the message payload: - `get_data()` returns the payload as a string - `get_data_raw()` returns the payload as bytes If you are working with binary data that is not guaranteed to be valid UTF-8, use the `get_data_raw` method to avoid errors when attempting to encode the data to a string. ## Using SSL The AMPS Python client includes support for Secure Sockets Layer. To use this support in the Python client using the default OpenSSL implementation for the Python installation, you need only use `tcps` for the transport type in the connection string, as described in the section on [Connection Strings for AMPS](./connection-strings.md) in this guide. If your Python client does not have a default OpenSSL implementation, you must load an SSL implementation as described below. This is typically the case for Windows Python builds, and may be the case if your site uses a custom build of Python on Linux. ### Loading a Different SSL Implementation The Python client also allows you to load and use an OpenSSL implementation other than the default implementation for the Python installation. The AMPS Python client provides the method `ssl_init`, which takes the name of the library to load or a full path to the file that contains the library. For example, to load the SSL implementation at `/opt/mycorp/trusted/vetted_ssl.so`, you could use the following line of code: ```python showLineNumbers AMPS.ssl_init("/opt/mycorp/trusted/vetted_ssl.so") ``` You must load the SSL library before making the connection. --- # Asynchronous Message Processing ## Asynchronous Message Processing Interface The AMPS Python client also supports an interface that allows you to process messages asynchronously. In this case, you add a message handler to the method call. The client object returns the command ID of the subscribe command once the server has acknowledged that the command has been processed. As messages arrive, the client calls your message handler directly on the background thread. This can be an advantage for some applications. For example, if your application is highly multithreaded and copies message data to a work queue processed by multiple threads, there may be a performance benefit to enqueuing work directly from the background thread. See [Understanding Threading](understanding-threading.md) for a discussion of threading considerations, including considerations for message handlers. As with the simple, synchronous interface, the AMPS client provides both convenience methods and methods that use a `Command` object. The following example shows how to use the asynchronous message processing interface (error handling and connection details are omitted for brevity): ```python showLineNumbers from AMPS import Client from AMPS import Command ... # Here, we create a Client object and connect to an AMPS server. client = Client("exampleSubscriber") client.connect("tcp://127.0.0.1:9007/amps/json") client.logon() def on_message_printer(message): print(message.get_data()) # Here, we create a subscription with the following parameters: subscriptionid = client.execute_async( Command("subscribe").set_topic("messages"), on_message_printer ) # on_message_printer is a function that acts as our message handler. When a # message is received, this function is invoked, and in this case, the get_data() # method from message is printed to the screen. Message is of type AMPS.Message. ``` ## Using an Instance Method as a Message Handler One of the more common ways of providing a message handler is as an instance method on an object that maintains message state. It's simple to provide a handler with this capability, as shown below: ```python showLineNumbers # Define a class that saves state and # handles messages. class StatefulHandler: # Initialize self with state to save def __init__(self, name): self._name = name self._count = 0 # Use state from this instance while handling # the message. def __call__(self, message): self._count += 1 print(f"{self._name} (count: {self._count}) got {message.get_data()}") ``` You can then provide an instance of the handler directly wherever a message handler is required, as shown below: ```python client.subscribe(StatefulHandler("An instance"), "topic") ``` :::warning When using asynchronous message processing, the AMPS client resets and reuses the message provided to `MessageHandler` functions between calls. This improves performance in the client, but means if your `MessageHandler` function needs to preserve information contained within the message you must copy the information rather than just saving the message object. Otherwise, the AMPS client cannot guarantee the state of the object or the contents of the object when your program goes to use it. ::: --- # Backlog and Smart Pipelining AMPS queues are designed for high-volume applications that need minimal latency and overhead. One of the features that helps performance is the *subscription backlog* feature, which allows applications to receive multiple messages at a time. The subscription backlog sets the maximum number of unacknowledged messages that AMPS will provide to the subscription. When the subscription backlog is larger than `1`, AMPS delivers additional messages to a subscriber before the subscriber has acknowledged the first message received. This technique allows subscribers to process messages as fast as possible, without ever having to wait for messages to be delivered. The technique of providing a consistent flow of messages to the application is called *smart pipelining*. ## Subscription Backlog The AMPS server determines the backlog for each subscription. An application can set the maximum backlog that it is willing to accept with the `max_backlog` option. Depending on the configuration of the queue (or queues) specified in the subscription, AMPS may assign a smaller backlog to the subscription. If no `max_backlog` option is specified, AMPS uses a `max_backlog` of `1` for that subscription. In general, applications that have a constant flow of messages perform better with a `max_backlog` setting higher than `1`. The reason for this is that, with a backlog greater than `1`, the application can always have a message waiting when the previous message is processed. Setting the optimum `max_backlog` is a matter of understanding the messaging pattern of your application and how quickly your application can process messages. To request a `max_backlog` for a subscription, you explicitly set the option on the subscribe command, as shown below: ```python command = Command("subscribe") \ .set_topic("my_queue") \ .set_options("max_backlog=10") ``` --- # Connectivity and Security In this section, we will learn more about the structure and features of the AMPS Python client. ## Connecting to AMPS In the [Quickstart](/clients/amps-client-python/quickstart) page we saw the below publish example which shows a client being created, connecting to AMPS and publishing a message. In the next section we will deep-dive into this example to ensure a solid understanding of each line of code. ```python showLineNumbers import AMPS import sys uri = "tcp://localhost:9007/amps/json" client = AMPS.Client("publish-example") try: client.connect(uri) client.logon() client.publish("messages", '{"hi" : "Hello, world!"}') except AMPS.AMPSException as e: sys.stderr.write(str(e)) client.publish_flush() client.close() ``` ### Examining the Code Below is a comprehensive breakdown of the previous publish example. ```python showLineNumbers # These import the AMPS and sys packages. All programs written using the AMPS # Python client will need to include the import AMPS statement at a minimum. import AMPS import sys # The URI to use to connect to AMPS. The URI consists of the transport, the # address, and the protocol to use for the AMPS connection. In this case, the # transport is tcp, the address is 127.0.0.1:9007, and the protocol is amps. This # connection will be used for JSON messages. Check with the person who manages the # AMPS instance to get the connection string to use for your programs. uri = "tcp://127.0.0.1:9007/amps/json" # This line creates a new Client object. Client encapsulates a single connection # to an AMPS server. Methods on Client allow for connecting, disconnecting, # publishing, and subscribing to an AMPS server. The argument to the Client # constructor, "publish-example", is a name chosen by the client to # identify itself to the server. Errors relating to this connection will be logged # with reference to this name, and AMPS uses this name to help detect duplicate # messages. AMPS enforces uniqueness for client names when a transaction log is # configured, and it is good practice to always use unique client names. client = AMPS.Client("publish-example") # Here, we open a try block that concludes with except AMPS.AMPSException as e. # All exceptions in AMPS derive from AMPSException. If an operation throws another # exception, AMPS will wrap that exception into the AMPSException you receive. For # example, if the call to connect() fails because the provided address is not # reachable, the AMPSException will contain an inner exception from the operating # system, likely a SocketException. try: # With this statement, we provide the URI to the client and declare the AMPS # connection. client.connect(uri) # The AMPS logon() command connects to AMPS and creates a named connection. # # This version of logon() uses the DefaultAuthenticator. If we # had provided logon credentials in the URI, that Authenticator would pass those # credentials to AMPS. Without credentials, the client logs on to AMPS # anonymously. AMPS versions 5.0 and later require a logon() command in the # default configuration. # # If you need to provide credentials in a different way, implement an Authenticator # and pass that Authenticator here in the 'authenticator' parameter. client.logon() # Here, we publish a single message to AMPS on the messages topic, containing the # data Hello, world!. This data is placed into a JSON message and sent to the # server. Upon successful completion of this function, the AMPS client has # enqueued the message to be sent to the server, and subscribers to the messages # topic will receive this JSON message: { "hi" : "Hello, world!" }. client.publish("messages", '{ "hi" : "Hello, world!"}') except AMPS.AMPSException as e: sys.stderr.write(str(e)) # Here, we call publish_flush() on the client. This command waits until all messages # from the client have been sent. If messages may still be in the process of # transmission when your application is ready to close the client, publish_flush() # helps ensure that the messages have been sent. In this case, we allow # publish_flush() to block indefinitely. A production application might specify a # timeout, to avoid hanging in the event that the application loses connectivity # before the messages have been sent. client.publish_flush() # We close the connection to AMPS. While that doesn't matter here, since the # application exits immediately after calling close(), it's good practice to close # connections when you are done using them. client.close() ``` :::tip ### About Authentication When a client logs on to AMPS, the client sends AMPS a username and password. The username is derived from the URI, using the standard syntax for providing a user name in a URI, for example, `tcp://JohnDoe:@server:port/amps/messagetype` to include the user name `JohnDoe` in the request. For a given user name, the password is provided by an `Authenticator`. The AMPS client distribution includes a `DefaultAuthenticator` that simply returns the password, if any, provided in the URI. A `logon()` command that does not specify an `Authenticator` will use an instance of `DefaultAuthenticator`. If your authentication system requires a different authentication token, you can implement an `Authenticator` that provides the appropriate token. ::: --- # Client Identification AMPS uses the name of the client as a session identifier and as part of the identifier for messages originating from that client. For this reason, when a transaction log is enabled in the AMPS instance (that is, when the instance is recording a sequence of publishes and attempting to eliminate duplicate publishes), an AMPS instance will only allow one application with a given client name to connect to the instance. When a transaction log is present, AMPS **requires** the client name for a publisher to be: - Unique within a set of replicated AMPS instances - Consistent from invocation to invocation *if* the publisher will be publishing the same *logical* stream of messages If publishers do not meet this contract (for example, if the publisher changes its name and publishes the same messages, or if a different publisher uses the same session name), message loss or duplication can happen. 60East recommends always using consistent, unique client names. For example, the client name could be formed by combining the application name, an identifier for the host system, and the ID of the user running the application. A strategy like this provides a name that will be different for different users or on different systems, but consistent for instances of the application that should be treated as equivalent to the AMPS system. Likewise, if a publisher is sending a completely independent stream of messages (for example, a microservice that sends a different, unrelated sequence of messages each time it connects to AMPS), there is no need for a publisher to retain the same name each time it starts. However, if a publisher is resuming a stream of messages (as in the case when using a file-backed publish store), that publisher **must** use the same client name, since the publisher is resuming the session. --- # Client Side Conflation In many cases, applications that use SOW topics only need the current value of a message at the time the message is processed, rather than processing each change that led to the current value. On the server side, AMPS provides *conflated topics* to meet this need. Conflated topics are described in more detail in the *AMPS User Guide*, and require no special handling on the client side. In some cases, though, it's important to conflate messages on the client side. This can be particularly useful for applications that do expensive processing on each message, applications that are more efficient when processing batches of messages, or for situations where you cannot provide an appropriate conflation interval for the server to use. A `MessageStream` has the ability to conflate messages received for a subscription to a SOW topic, view, or conflated topic. When conflation is enabled, for each message received, the client checks to see whether it has already received an unprocessed message with the same `SowKey`. If so, the client replaces the unprocessed message with the new message. The application never receives the message that has been replaced. To enable client-side conflation, you call `conflate()` on the `MessageStream`, and then use the `MessageStream` as usual: ```python showLineNumbers # SOW query and subscribe results = client.sow_and_subscribe("orders", "/symbol == 'ROL'") # Turn on conflation results.conflate() # Process the results for message in results: # process message here ``` When a `MessageStream` is used for a subscription that does not include `SowKeys` (such as a subscription to a topic that does not have a SOW), the `MessageStream` will allow you to turn on conflation, but no conflation will occur. When using client-side conflation with delta subscriptions, bear in mind that client-side conflation replaces the whole message, and does not attempt to merge deltas. This means that updates can be lost when messages are replaced. For some applications (for example, a ticker application that simply sends delta updates that replace the current price), this causes no problems. For other applications (for example, when several processors may be updating different fields of a message simultaneously), using conflation with deltas could result in lost data, and server-side conflation is a safer alternative. --- # Connection Parameters for AMPS When specifying a URI for connection to an AMPS server, you may specify a number of transport-specific options in the parameters section of the URI connection parameters. Here is an example: ```bash tcp://localhost:9007/amps/json?tcp_nodelay=true&tcp_sndbuf=100000 ``` In this example, we have specified the AMPS instance on `localhost`, port `9007`, connecting to a transport that uses the `amps` protocol and sending JSON messages. We have also set two parameters: `tcp_nodelay`, a Boolean (true/false) parameter, and `tcp_sndbuf`, an integer parameter. Multiple parameters may be combined to finely tune settings available on the transport. Normally, you'll want to stick with the defaults on your platform, but there may be some cases where experimentation and fine-tuning will yield higher or more efficient performance. The AMPS client supports the value of `tcp` in the *scheme* component connection string for TCP/IP connections, and the value of `tcps` as the scheme for SSL encrypted connections. For connections that use Unix domain sockets, the client supports the value of `unix` in the scheme, and requires an additional option, as described in the Unix Transports Parameters section below. ## IPv6 Connections Starting with version 5.3.3.0, the AMPS client supports creating connections over both IPv4 and IPv6 protocols if supported by the underlying Operating System. By default, the AMPS client will prefer to resolve host names to IPv4 addresses, but this behavior can be adjusted by supplying the `ip_protocol_prefer` transport option, described in the table below. ## TCP and SSL Transport Options The following transport options are available for TCP connections: |Option |Description | |-------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`bind` |

    (IP address) Sets the interface to bind the outgoing socket to.

    Starting with version 5.3.3.0, both IPv4 and IPv6 addresses are fully supported for use with this parameter.

    | |`tcp_connecttimeout` | (integer) Sets the connect timeout in milliseconds. This helps enable failover in cases where an attempt to connect to a server is unresponsive without returning a failure. | |`tcp_rcvbuf` |(integer) Sets the socket receive buffer size. This defaults to the system default size. (On Linux, you can find the system default size in `/proc/sys/net/core/rmem_default`.)| |`tcp_sndbuf` |(integer) Sets the socket send buffer size. This defaults to the system default size. (On Linux, you can find the system default size in `/proc/sys/net/core/wmem_default`.)| |`tcp_nodelay` |(boolean) Enables or disables the `TCP_NODELAY` setting on the socket. By default `TCP_NODELAY` is disabled.| |`tcp_linger` |(integer) Enables and sets the `SO_LINGER` value for the socket By default, `SO_LINGER` is enabled with a value of `10`, which specifies that the socket will linger for 10 seconds.| |`tcp_keepalive` |(boolean) Enables or disables the `SO_KEEPALIVE` value for the socket. The default value for this option is true.| |`ip_protocol_prefer` |

    (string) Influence the IP protocol to prefer during DNS resolution of the host. If a DNS entry of the preferred protocol can not be found, the other non-preferred protocol will then be tried.

    If this parameter is not set, the default will be to prefer IPv4.

    If an explicit IPv4 address or IPv6 IP address is provided as the host, the format of the IP address is used to determine the IP protocol used and this setting has no effect.

    Supported Values:

    `ipv4`: Prefer an IPv4 address when resolving the host

    `ipv6`: Prefer an IPv6 address when resolving the host

    This parameter is available starting with version 5.3.3.0.

    | ## Unix Transport Parameters The `unix` transport type communicates over Unix domain sockets. This transport **requires** the following additional option: |Option |Description | |-----------------------|-------------------------------------------------| |`path` |The path to the Unix domain socket to connect to.| Unix domain sockets always connect to the local system. When the scheme specified is `unix`, the host address is *ignored* in the connection string. For example, the connection string: ```bash unix://localhost:0/amps/json?path=/sockets/the-amps-socket ``` and the connection string: ```bash unix://unix:unix/amps/json?path=/sockets/the-amps-socket ``` are equivalent. The other components of the connection string, including the *protocol*, *message type*, *username*, and *authentication token* are processed just as they would be for TCP/IP sockets. ## AMPS Additional Logon Options The connection string can also be used to pass logon parameters to AMPS. AMPS supports the following additional logon option: |Option |Description | |-----------------------|-----------------------------------------------------------------------------------------------| |`pretty` |Provide formatted representations of binary messages rather than the original message contents.| --- # Connection Strings for AMPS The AMPS clients use connection strings to determine the server, port, transport, and protocol to use to connect to AMPS. When the connection point in AMPS accepts multiple message types, the connection string also specifies the precise message type to use for this connection. Connection strings have a number of elements: ![Diagram showing the different elements of a connection string](./assets/client-connection-string-light.svg#gh-light-mode-only) ![Diagram showing the different elements of a connection string](./assets/client-connection-string-dark.svg#gh-dark-mode-only) As shown in the figure above, connection strings have the following elements: * _Transport_ - Defines the network used to send and receive messages from AMPS. In this case, the transport is `tcp`. For connections to transports that use the Secure Sockets Layer (SSL), use `tcps`. For connections to AMPS over a Unix domain socket, use `unix`. * _Host Address_ - Defines the destination on the network where the AMPS instance receives messages. The format of the address is dependent on the transport. For `tcp` and `tcps`, the address consists of a host name and port number. In this case, the host address is `localhost:9007`. For `unix` domain sockets, a value for hostname and port must be provided to form a valid URI, but the content of the hostname and port are ignored, and the file name provided in the **path** parameter is used instead (by convention, many connection strings use `localhost:0` to indicate that this is a local connection that does not use TCP/IP). * _Protocol_ - Sets the format in which AMPS receives commands from the client. Most code uses the default `amps` protocol, which sends header information in JSON format. AMPS supports the ability to develop custom protocols as extension modules, and AMPS also supports legacy protocols for backward compatibility. * _MessageType_ - Specifies the message type that this connection uses. This component of the connection string is required if the protocol accepts multiple message types and the transport is configured to accept multiple message types. If the protocol does not accept multiple message types, this component of the connection string is optional, and defaults to the message type specified in the transport. Legacy protocols such as `fix`, `nvfix` and `xml` only accept a single message type, and therefore do not require or accept a message type in the connection string. As an example, a connection string such as: ```bash tcp://localhost:9007/amps/json ``` would work for programs connecting from the local host to a `Transport` configured as follows: ```xml showLineNumbers ... any-tcp tcp 9007 amps ... ``` See the [Configuring Transports](/docs/amps-user-guide/transports/configuring-transports) section in the _AMPS User Guide_ for more information on configuring transports. ## Using zlib Compression The AMPS Python Client supports enabling zlib compression by adding the `compression=zlib` URI parameter to the connection string. For example: ``` tcp://localhost:9007/amps/json?compression=zlib ``` No server-side configuration changes are required. The client enables zlib compression for the connection based on the URI parameter. If the connection string already contains other URI parameters, add `compression=zlib` using `&`: ``` tcp://localhost:9007/amps/json?=&compression=zlib ``` ## Using HTTP Preflight for Connection Upgrades Some users need to minimize the number of externally accessible ports while still allowing multiple AMPS transports to be used in environments with strict firewall policies for security reasons. To address this, AMPS supports an HTTP Preflight mechanism that enables TCP clients to share the same external port used for WebSockets. Instead of requiring a dedicated external TCP port, clients can establish a connection over an existing HTTP endpoint. This reduces the number of open network ports while maintaining full TCP/TCPS functionality. This feature works by leveraging HTTP Upgrade requests, similar to WebSockets, allowing clients to connect via an initial HTTP request before transitioning to a full TCP/TCPS session. Additionally, the HTTP preflight mechanism enables custom HTTP headers to be included in the initial handshake, making it easier to integrate with reverse proxies. Sample code snippet: ```python showLineNumbers import AMPS import time # Create an AMPS client instance client = AMPS.Client('test') # Connect using HTTP preflight client.connect('tcp://localhost:80/client/amps/json?http_preflight=true') # Log on to AMPS client.logon() print('Connected!') try: while True: client.publish('test', '{"hello":"world"}') time.sleep(1) except KeyboardInterrupt: print("\nStopped by user. Closing connection...") client.close() # Gracefully close the connection print("\nConnection closed.") ``` To learn more about HTTP Preflight, including how to enable it and configure an NGINX proxy, refer to the [HTTP Preflight](/docs/amps-user-guide/transports/http-preflight) section in the _AMPS User Guide_ and the [HTTP Preflight- Proxy Play: AMPS Unlocked](/blog/http-preflight) blog. --- # Content Filtering One of the most powerful features of AMPS is content filtering. With content filtering, filters based on message content are applied at the server so that your application and the network are not utilized by messages that are not relevant to your application. For example, if your application is only displaying messages from a particular user, you can send a content filter to the server so that only messages from that particular user are sent to the client. To apply a content filter to a subscription, simply pass it into the `client.subscribe()` call or use the `set_filter` method to add a filter to the command: ```python showLineNumbers for message in client.subscribe( "letters", "/sender='mom'", timeout=5000): print(f"Mom says: {message.get_data()}") ``` In this example, we have passed in a content filter `/sender = 'mom'`. This will result in the server only sending us messages from the `letters` topic that have the sender field equal to `mom` in the message. For example, the AMPS server will send the following message, where `/sender` is `mom`: ```javascript showLineNumbers { "sender" : "mom", "text" : "Happy Birthday!", "reminder" : "Call me Thursday!" } ``` The AMPS server will not send a message with a different `/sender` value: ```javascript showLineNumbers { "sender" : "henry dave", "text" : "Things do not change; we change." } ``` --- # Controlling Blocking with Command Timeout The named convenience methods and the `Command` class provide a `timeout` setting that specifies how long the command should wait to receive a `processed` acknowledgment from AMPS. This can be helpful in cases where it is important for the caller to limit the amount of time to block waiting for AMPS to acknowledge the command. If the AMPS client does not receive the processed acknowledgment within the specified time, the client sends an `unsubscribe` command to the server to cancel the command and throws an exception. Acknowledgments from AMPS are processed by the client receive thread on the same socket as data from AMPS. This means that any other data previously returned (such as the results of a large query) must be consumed before the acknowledgment can be processed. An application that submits a set of SOW queries in rapid succession should set a timeout that takes into account the amount of time required to process the results of the previous query. --- # AMPS Programming: Working with Commands The AMPS clients provide named convenience methods for core AMPS functionality. These named methods work by creating messages and sending those messages to AMPS. All communication with AMPS occurs through messages. You can use the `Command` object to customize the messages that AMPS sends. This is useful for more advanced scenarios where you need precise control over the message, or in cases where you need to use an earlier version of the client to communicate with a more recent version of AMPS, or in cases where a named method is not available. ## Understanding AMPS Messages AMPS messages are represented in the client as `AMPS.Message` objects. The `Message` object is generic and can represent any type of AMPS message, including both outgoing and incoming messages. This section includes a brief overview of elements common to AMPS command messages. Full details of commands to AMPS are provided in the *AMPS Command Reference* (linked at the bottom of this page). All AMPS command messages contain the following elements: - **Command** - The *command* tells AMPS how to interpret the message. Without a command, AMPS will reject the message. Examples of commands include `publish`, `subscribe`, and `sow`. - **CommandId** - The *command ID*, together with the name of the client, uniquely identifies a command to AMPS. The command ID can be used later on to refer to the command or the results of the command. For example, the command ID for a `subscribe` message becomes the identifier for the subscription. The AMPS client provides a command ID when the command requires one and no command ID is set. Most AMPS messages contain the following fields: - **Topic** - The *topic* that the command applies to, or a regular expression that identifies a set of topics that the command applies to. For most commands, the topic is required. Commands such as `logon`, `start_timer`, and `stop_timer` do not apply to a specific topic, and do not need this field. - **Ack Type** - The *ack type* tells AMPS how to acknowledge the message to the client. Each command has a default acknowledgment type that AMPS uses if no other type is provided. - **Options** - The `options` are a comma-separated list of options that affect how AMPS processes and responds to the message. Beyond these fields, different commands include fields that are relevant to that particular command. For example, SOW queries, subscriptions, and some forms of SOW deletes accept the **Filter** field, which specifies the filter to apply to the subscription or query. As another example, publish commands accept the **Expiration** field, which sets the SOW expiration for the message. For full details on the options available for each command and the acknowledgment messages returned by AMPS, see the *AMPS Command Reference*. ## Creating and Populating the Command To create a command, you simply construct a command object of the appropriate type: ```python command = AMPS.Command("sow") ``` Once created, you set the appropriate fields on the command. For example, the following code creates a SOW query, setting the command, topic, and filter for the query: ```python showLineNumbers command = AMPS.Command("sow") \ .set_topic("messages-sow") \ .set_filter("/id > 20") ``` When sent to AMPS using the `execute()` method, AMPS performs a SOW query from the topic `messages-sow` using a filter of `/id > 20`. The results of sending this message to AMPS are no different than using the form of the `sow` method that sets these fields. ## Using Execute Once you've created a command, use the `execute` method to send the command to AMPS. One form of the `execute` method returns a `MessageStream` that you can use from the calling thread to process responses from AMPS. The other form, `execute_async` method, sends the message to AMPS, waits for a `processed` acknowledgment, then returns. Messages are processed on the client background thread. For example, the following snippet sends the command created above: ```python client.execute(command) ``` This returns a `MessageStream` identical to the `MessageStream` returned by the equivalent `client.sow()` method. You can also provide a message handler to receive acknowledgments, statistics, or the results of subscriptions and SOW queries. The AMPS client maintains a background thread that receives and processes incoming messages. The call to `execute_async` returns on the main thread as soon as AMPS acknowledges the command as having been processed, and messages are received and processed on the background thread: ```python showLineNumbers def handle_messages(m): print(f"{m.get_ack_type()} : {m.get_reason()}") # other message handling here client.execute_async(command, handle_messages) ``` While this message handler simply prints the ack type and reason for sample purposes, message handlers in production applications are typically designed with a specific purpose. For example, your message handler may fill a work queue, or check for success and throw an exception if the command failed. ### Using Execute to Publish Notice that the `publish` command typically does not return results other than acknowledgment messages. To send a `publish` command, use the `execute_async()` method, providing `None` for the message handler: ```python client.execute_async(publishCmd, None) ``` Since the code sets the message handler to `None`, this code does not receive acknowledgments. To detect publish failures, set the `FailedWriteHandler` for the client. ## AMPS Command Cookbook The [AMPS Command Reference](/docs/amps-command-reference) includes information on which fields and options to set on commands to get a specific result. The reference includes both reference information and a [Command Cookbook](/docs/amps-command-reference/cookbook) that provides a concise guide for commonly-used commands. --- # Providing Credentials to AMPS When a client logs on to AMPS, the client sends AMPS a username and password. The username is derived from the URI, using the standard syntax for providing a user name in a URI, for example, `tcp://JohnDoe:@server:port/amps/messagetype` to include the user name `JohnDoe` in the request. For a given user name, the password is provided by an `Authenticator`. The AMPS client distribution includes a `DefaultAuthenticator` that simply returns the password, if any, provided in the URI. A `logon()` command that does not specify an `Authenticator` will use an instance of `DefaultAuthenticator`. If your authentication system requires a different authentication token, you can implement an `Authenticator` that provides the appropriate token. ## Providing Credentials in a Connection String When using the `DefaultAuthenticator`, the AMPS clients support the standard format for including a username and password in a URI, as shown below: ```bash tcp://user:password@host:port/protocol/message_type ``` When provided in this form, the default authenticator provides the username and password specified in the URI. If you have implemented another authenticator, that authenticator controls how passwords are provided to the AMPS server. --- # Delta Publish To delta publish, you use the `delta_publish` command as follows: ```python showLineNumbers # assumes that client is connected and logged on msg = ... # obtain changed fields here client.delta_publish("myTopic", msg) ``` The message that you provide to AMPS must include the fields that the topic uses to generate the SOW key. Otherwise, AMPS will not be able to identify the message to update. For SOW topics that use a User-Generated SOW Key, use the `Command` form of `delta_publish` to set the `SowKey`, as shown below: ```python showLineNumbers # assumes that client is connected and logged on msg = ... # obtain changed fields here key = ... # obtain user-generated SOW key cmd = AMPS.Command("delta_publish") cmd.set_topic("delta_topic") cmd.set_sow_key(key) cmd.set_data(msg) # Execute the delta publish. Use None for # the message handler since any failure acks will # be routed to the FailedWriteHandler. client.execute_async(cmd,None) ``` The full behavior of `delta_publish` is described in the [AMPS User Guide](/docs/amps-user-guide) section on [Incremental Message Updates](/docs/amps-user-guide/delta-publish). --- # Delta Subscribe To delta subscribe, you simply use the `delta_subscribe` command as follows: ```python showLineNumbers # assumes that client is connected and logged on cmd = AMPS.Command("delta_subscribe") cmd.set_topic("delta_topic") cmd.set_filter("/thingIWant = 'true'") for m in client.execute(cmd): # Delta messages arrive here ``` As described in the [AMPS User Guide](/docs/amps-user-guide) section on [Receiving Only Updated Fields](/docs/amps-user-guide/delta-subscribe), messages provided to a delta subscription will contain the fields used to generate the SOW key and any changed fields in the message. Your application is responsible for choosing how to handle the changed fields. --- # Delta Publish and Subscribe Delta messaging in AMPS has two independent aspects: - **Delta Subscribe** - Allows subscribers to receive just the fields that are updated within a message. - **Delta Publish** - Allows publishers to update and add fields within a message by publishing only the updates into the SOW. This chapter describes how to create delta publish and delta subscribe commands using the AMPS Python client. For a discussion of this capability, how it works, and how message types support this capability see the [AMPS User Guide](/docs/amps-user-guide). --- # Detecting Write Failures The `publish` methods in the Python client deliver the message to be published to AMPS then return immediately, without waiting for AMPS to return an acknowledgment. Likewise, the `sow_delete` methods request deletion of SOW messages, and return before AMPS processes the message and performs the deletion. This approach provides high performance for operations that are unlikely to fail in production. However, this means that the methods return before AMPS has processed the command, without the ability to return an error in the event the command fails. The AMPS Python client provides a `failed_write_handler` that is called when the client receives an acknowledgment that indicates a failure to persist data within AMPS. As with the `last_chance_message_handler` described in the [Unexpected Messages](unexpected-messages) section, your application registers a handler for this function. When an acknowledgment returns that indicates a failed write, AMPS calls the registered handler method with information from the acknowledgment message, supplemented with information from the client publish store if one is available. Your client can log this information, present an error to the user or take whatever action is appropriate for the failure. If your application needs to know whether publishes succeeded and are durably persisted, the following approach is recommended: - Set a `PublishStore` on the client. This will ensure that messages are retransmitted if the client becomes disconnected before the message is acknowledged *and* request `persisted` acknowledgments for messages. - Install a `failed_write_handler`. In the event that AMPS reports an error for a given message, that event will be reported to the `failed_write_handler`. - Call `publish_flush()` and verify that all messages are persisted before the application exits. When no `failed_write_handler` is registered, acknowledgments that indicate errors in persisting data are treated as unexpected messages and routed to the `last_chance_message_handler`. In this case, AMPS provides only the acknowledgment message and does not provide the additional information from the client publish store. --- # Disconnect Handling Every distributed system will experience occasional disconnections between one or more nodes. The reliability of the overall system depends on an application's ability to efficiently detect and recover from these disconnections. Using the AMPS Python client's disconnect handling, you can build powerful applications that are resilient in the face of connection failures and spurious disconnects. For additional reliability, you can also use the high availability client (discussed in the following sections), which provides both disconnect handling and features to help ensure that messages are reliably delivered. --- # Error Handling In every distributed system, the robustness of your application depends on its ability to recover gracefully from unexpected events. The AMPS client provides the building blocks necessary to ensure your application can recover from the kinds of errors and special events that may occur when using AMPS. --- # Examples The AMPS Python Client includes a set of example programs that provide simple demonstrations of client functionality. The sample archive is available that includes a set of samples and a configuration file for AMPS: **[python-examples.zip](./examples/python-examples.zip)** :::tip Examples may need to be updated with the IP address or DNS name of the host running AMPS unless you are running both the samples and the AMPS server on the same system. ::: The samples archive includes samples such as: | Sample Name | Demonstrates | |---------------|---------------| | `AMPSConsoleSubscriber.py` | Simple subscriber to an adhoc topic. | | `AMPSConsolePublisher.py` | Simple publisher to an adhoc topic. | | `AMPSFIXBuilderPublisher.py` | Subscriber that uses the provided convenience class to create FIX messages | | `AMPSFIXShredderSubscriber.py` | Subscriber that uses the provided convenience class to parse FIX messages | | `AMPSNVFIXBuilderPublisher.py` | Subscriber that uses the provided convenience class to create NVFIX messages | | `AMPSNVFIXShredderSubscriber.py` | Subscriber that uses the provided convenience class to parse NVFIX messages | | `AMPSPublishForReplay.py` | Simple publisher targeting a topic in the transaction log | | `AMPSQueueConsumer.py` | Subscriber that consumes from a queue | | `AMPSQueuePublisher.py` | Simple publisher targeting a queue topic (which must also be in the transaction log) | | `AMPSSOWConsolePublisher.py` | Simple publisher targeting a topic in the State of the World | | `AMPSSOWQuerypy` | Point in time query of a topic in the State of the World | | `AMPSSOWAndSubscribeConsoleSubscriber.py` | Point in time query of and ongoing subscription to a topic in the State of the World | | `AMPSSOWandSubscribeWithOOF.py` | Point in time query of and ongoing subscription to a topic in the State of the World. This subscription also requests out of focus notifications if a message is deleted or no longer matches the subscription | | `AMPSUpdateForOOF.py` | Publisher that updates messages to generate out of focus messages in the `AMPSSOWandSubscribeWithOOF.py` example | | `AMPSSubscribeForReplay.py` | Subscriber requesting a replay from the transaction log (bookmark subscribe) | | `CompositeMessagePublisher.py` | Publisher that uses the provided convenience class to create a composite messages | | `CompositeMessageSubscriber.py` | Publisher that uses the provided convenience class to consume a composite message | --- # Exception Handling and Asynchronous Message Processing When using asynchronous message processing, exceptions thrown from the message handler are silently absorbed by the AMPS Python client by default. The AMPS Python client allows you to register an exception listener to detect and respond to these exceptions. When an exception listener is registered, AMPS will call the exception listener with the exception. See [Unhandled Exceptions](unhandled-exceptions.md) for details. --- # Exception Types The following table details each of the exception types thrown by AMPS. | Exception | When | Notes | | --------------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AlreadyConnectedException` | Connecting | Thrown when `connect()` is called on a `Client` that is already connected. | | `AMPSException` | Anytime | Base class for all AMPS exceptions. | | `AuthenticationException` | Anytime | Indicates an authentication failure occurred on the server. | | `BadFilterException` | Subscribing | This typically indicates a syntax error in a filter expression. | | `BadRegexTopicException` | Subscribing | Indicates a malformed regular expression was found in the topic name. | | `CommandException` | Anytime | Base class for all exceptions relating to commands sent to AMPS. | | `ConnectionException` | Anytime | Base class for all exceptions relating to the state of the AMPS connection. | | `ConnectionRefusedException` | Connecting | The connection was actively refused by the server. Validate that the server is running, that network connectivity is available, and the settings on the client match those on the server. | | `DisconnectedException` | Anytime | No connection is available when AMPS needed to send data to the server *or* the user's disconnect handler threw an exception. | | `InvalidTopicException` | SOW query | The topic is not configured for the requested operation. For example, a `sow` command was issued for a topic that is not in the SOW or a bookmark subscribe was issued for a topic that is not recorded in the transaction log. | | `InvalidTransportOptionsEx` `ception` | Connecting | An invalid option or option value was specified in the URI. | | `InvalidUriException` | Connecting | The URI string provided to `connect()` was formatted improperly. | | `MessageTypeException` | Connecting | The class for a given transport's message type was not found in AMPS. | | `NameInUseException` | Connecting | The client name (specified when instantiating `Client`) is already in use on the server. | | `RetryOperationException` | Anytime | An error occurred that caused processing of the last command to be aborted. Try issuing the command again. | | `StreamException` | Anytime | Indicates that data corruption has occurred on the connection between the client and server. This usually indicates an internal error inside of AMPS -- contact AMPS support. | | `SubscriptionAlreadyExists` `Exception` | Subscribing | A subscription has been requested using the same command ID string as another subscription. Create a unique command ID string for every subscription. | | `TimedOutException` | Anytime | A timeout occurred waiting for a response to a command. | | `TransportTypeException` | Connecting | Thrown when a transport type is selected in the URI that is unknown to AMPS. | | `UnknownException` | Anytime | Thrown when an internal error occurs. Contact AMPS support immediately. | | `UsageException` | Changing the properties of an object | Thrown when the object is not in a valid state for setting the properties. For example, some properties of a `Client` (such as the name) cannot be changed while that client is connected to AMPS. | --- # Exceptions Generally speaking, when an error occurs that prohibits an operation from succeeding, AMPS will throw an exception. AMPS exceptions universally derive from `AMPS.AMPSException`, so by catching `AMPSException`, you will be sure to catch anything AMPS throws, for example: ```python showLineNumbers def read_and_evaluate(client): # read a new payload from the user payload = input("Please enter a message") # write a new message to AMPS if payload: try: client.publish( "UserMessage", f'{{ "message" : "{payload}" }}' ) except AMPS.AMPSException as e: sys.stderr.write(f"An AMPS exceptionoccurred: {str(e)}") ``` In this example, if an error occurs, the program writes the error to `stderr` and the `publish()` command fails. However, `client` is still usable for continued publishing and subscribing. When the error occurs, the exception is written to the console. As with most Python exceptions, `str()` will convert the exception into a string that includes a descriptive message. AMPS exception types vary based on the nature of the error that occurs. In your program, if you would like to handle certain kinds of errors differently than others, you can handle the appropriate subclass of `AMPSException` to detect those specific errors and do something different. ```python showLineNumbers def create_new_subscription(client): message_stream = None topic_name = None while message_stream is None: # attempts to retrieve a topic name (or regular expression) from the user. topic_name = input("Please enter a topic name") try: # If an error occurs when setting up the subscription, the program decides whether # or not to try again based on the subclass of AMPSException that is thrown. In # this case, if the exception is a BadRegexTopicError, the exception indicates # that the user provided a bad regular expression. We would like to give the user # a chance to correct, so we ask the user for a new topic name. message_stream = client.subscribe( topic_name, None ) # This line indicates that the program catches the BadRegexTopicError exception # and displays a specific error to the user indicating the topic name or # expression was invalid. By not returning from the function in this except block, # the while loop runs again and the user is asked for another topic name. except BadRegexTopicError as e: print( "Error: bad topic name or regular expression " + topic_name + ". The exception was " + str(e) + "." ) # we'll ask the user for another topic # If an AMPS exception of a type other than BadRegexTopicError is thrown by AMPS, # it is caught here. In that case, the program emits a different error message to # the user. except AMPSException as e: print ( "Error: error setting up subscription to topic" + topic_name + ". The exception was " + str(e) + "." ) # At this point the code stops attempting to subscribe to the client by the return # None statement. return None return message_stream ``` --- # Changing the Filter on a Subscription AMPS allows you to update parameters, such as the content filter, on a subscription. When you replace a filter on the subscription, AMPS immediately begins sending only messages that match the updated filter. Notice that if the subscription was entered with a command that includes a SOW query, using the `replace` option can re-issue the SOW query (as described in the *AMPS User Guide*). To update the filter on a subscription, you create a `subscribe` command. You set the `SubscriptionId` provided on the `Command` to the identifier of the existing subscription and include the `replace` option on the `Command`. When you send the `Command`, AMPS atomically replaces the filter and sends messages that match the updated filter from that point forward. ```python showLineNumbers sub_cmd = AMPS.Command("sow_and_subscribe") \ .set_topic("orders-sow") \ .set_sub_id("A42") \ .set_filter("/details/items/description LIKE 'puppy'") client.execute_async(sub_cmd, message_handler) # Elsewhere in the program... replace_cmd = AMPS.Command("sow_and_subscribe") \ .set_topic("orders-sow") \ .set_sub_id("A42") \ .set_filter("/details/items/description LIKE 'kitten'") \ .set_options("replace") client.execute_async(replace_cmd, message_handler) ``` --- # Using a Heartbeat to Detect Disconnection The AMPS client includes a heartbeat feature to help applications detect disconnection from the server within a predictable amount of time. Without using a heartbeat, an application must rely on the operating system to notify the application when a disconnect occurs. For applications that are simply receiving messages, it can be impossible to tell whether a socket is disconnected or whether there are simply no incoming messages for the client. When you set a heartbeat, the AMPS client sends a heartbeat message to the AMPS server at a regular interval, and expects a response from the server within the specified amount of time. If the operating system reports an error on send, or if there is no activity received from the server within the specified amount of time, the AMPS client considers the server to be disconnected. Likewise, the server will ensure that traffic is sent to the client at the specified interval, using heartbeat messages when no other traffic is being sent to the client. If, after sending a heartbeat message, no traffic from the client arrives within a period twice the specified interval, the server will consider the client to be disconnected or nonresponsive. The AMPS client processes heartbeat messages on the client receive thread, which is the thread used for asynchronous message processing. If your application uses asynchronous message processing and occupies the thread for longer than the heartbeat interval, the client may fail to respond to heartbeat messages in a timely manner and may be disconnected by the server. --- # High Availability The AMPS Python Client provides an easy way to create highly-available applications using AMPS, via the `HAClient` class. `HAClient` derives from `Client` and offers the same methods, but also adds protection against network, server, and client outages. Using `HAClient` allows applications to automatically: - Recover from temporary disconnects between client and server. - Failover from one server to another when a server becomes unavailable. Since the `HAClient` automatically manages failover and reconnection, 60East recommends using the `HAClient` for applications that need to: - Automatically reconnect and resume work in the case of disconnection. - Ensure no messages are lost or duplicated after a reconnect or failover. - Persist messages and bookmarks on disk for protection against client failure. You can choose how your application uses `HAClient` features. For example, you might need automatic reconnection, but have no need to resume subscriptions or republish messages. The high availability behavior in `HAClient` is provided by implementations of defined interfaces. You can combine different implementations provided by 60East to meet your needs, and implement those interfaces to provide your own policies. Some of these features require specific configuration settings on your AMPS instance(s). This chapter mentions these features and describes how to use them from the AMPS Python client. You can find full documentation for these settings and server features in the *AMPS User Guide*. ## Overview of HAClient `HAClient` derives from `Client` and offers the same methods for sending commands to AMPS and receiving messages from AMPS. The `HAClient` differs from the `Client` in two ways: - The `HAClient` automatically installs a disconnect handler that reconnects to AMPS and resumes active (asynchronous) subscriptions. The disconnect handler optionally replays `publish` and `sow_delete` messages that have not been acknowledged by AMPS, using a `PublishStore`. The disconnect handler can optionally resume replays from the transaction log at a point that guarantees no messages are skipped and no duplicates are delivered to the application, using a `BookmarkStore`. - The `HAClient` includes the infrastructure needed for client failover, including a list of connection strings and their associated authentication mechanisms (provided by the `ServerChooser`), and options for controlling backoff behavior for reconnects (provided by the `DelayStrategy`). As a result, the `HAClient` provides a `connect_and_logon()` function for establishing a connection to AMPS, rather than treating these as independent steps that an application must manage itself. If your application needs to automatically reconnect to AMPS, 60East recommends using the `HAClient` and the automatically provided disconnect handler rather than using a `Client` or replacing the `HAClient` default disconnect handler. ## Reconnection with HAClient The most important difference between `Client` and `HAClient` is that `HAClient` automatically provides a reconnect handler. This description provides a high-level framework for understanding the components involved in failover with the `HAClient`. The components are described in more detail in the following sections. The `HAClient` reconnect handler performs the following steps when reconnecting: 1. Calls the `ServerChooser` to determine the next URI to connect to and the authenticator to use for that connection. If the connection fails, calls `get_error` on the `ServerChooser` to get a description of the failure, sends an exception to the exception listener and stops the reconnection process. 2. Calls the `DelayStrategy` to determine how long to wait before attempting to reconnect and waits for that period of time. 3. Connects to the AMPS server. If the connection fails, calls `report_failure` on the `ServerChooser` and begins the process again. 4. Logs on to the AMPS server. If the connection fails, calls `report_failure` on the `ServerChooser` and begins the process again. 5. Calls `report_success` on the `ServerChooser`. 6. Receives the bookmark for the last message that the server has persisted. Discards any older messages from the `PublishStore`. 7. Republishes any messages in the `PublishStore` that have not been persisted by the server. 8. Re-establishes subscriptions using the `SubscriptionManager` for the client. For bookmark subscriptions, the reconnect handler uses the `BookmarkStore` for the client to determine the most recent bookmark, and re-subscribes with that bookmark. For subscriptions that do not use a bookmark, the `SubscriptionManager` simply re-enters the subscription, meaning that it is entered at the point at which the `HAClient` reconnects. The `ServerChooser`, `DelayStrategy`, `PublishStore`, and `BookmarkStore` are all extension points for the `HAClient`. You can adapt the failover and recovery behavior by setting a different object for the behavior you want to customize on the `HAClient` or by providing your own implementation. For example, the convenience methods in the previous section customize the behavior of the `PublishStore` and `BookmarkStore` by providing either memory-backed or file-backed stores. The reconnection process runs on the thread that discovers the disconnection. This means that, in the event that an application thread discovers the disconnection as a result of a call to the Python AMPS client, that call may not return until a connection is re-established (or until the server chooser indicates failure, in which case the application will receive an exception). The Python client includes a `retry_on_disconnect` setting that controls this retry behavior when the client is disconnected. When set to `True` (the default), any call to the Client that results in a command being sent to AMPS may block until a connection is re-established. When set to `False`, the `HAClient` will retry the connection a single time and throw an exception if the connection cannot be re-established. Regardless of the `retry_on_disconnect` setting, a call to `publish` will result in the message being stored in the `PublishStore` for the client if one is set. ## Choosing Store Durability If your application needs reliable publish to AMPS, install a `PublishStore` in the `HAClient`. If your application needs to resume replays from the transaction log, install a `BookmarkStore` in the `HAClient`. These stores provide the following capabilities: - A *bookmark store* tracks received messages and is used to resume subscriptions that replay from the transaction log. - A *publish store* tracks published messages and is used to ensure that messages are persisted in AMPS. The AMPS client provides a memory-backed version of each store and a file-backed version of each store. The store interface is public, and an application can create and provide a custom store as necessary. An `HAClient` can use either a memory backed store or a file backed store for protection. Each method provides resilience to different failures, as described below: - *Memory-backed stores* provide recovery disconnection from AMPS by storing messages and bookmarks in your process' address space. This is the highest performance option for working with AMPS in a highly available manner. The trade-off with this method is there is no protection from a crash or failure of your client application. If your application is terminated prematurely or, if the application terminates at the same time as an AMPS instance failure or network outage, then messages may be lost or duplicated. The state of bookmark replays will be lost when the application shuts down. Messages in the publish store when the application shuts down will not be maintained through a restart, so the application will not be able to attempt any necessary redelivery when the application restarts. A memory-backed store should only be used by one instance of a client at a time. - *File-backed stores* provide recovery after client failure and disconnection from AMPS by storing messages and bookmarks on disk. To use this protection method, the `create_file_backed` method requests additional arguments for the two files that will be used for both bookmark storage and message storage. If these files exist and are non-empty (as they would be after a client application is restarted), the `HAClient` loads their contents and ensures synchronization with the AMPS server once connected. The performance of this option depends heavily on the speed of the device on which these files are placed. When the files do not exist (as they would the first time a client starts on a given system), the `HAClient` creates and initializes the files, and in this case the client does not have a point at which to resume the subscription or messages to republish. A store file should only be used by one instance of a client at a time. When using file-backed stores, 60East recommends periodically removing unneeded entries by calling the `prune()` method. The precise strategy that your application uses to call `prune()` depends on the nature of the application. Most applications call `prune()` when the application exits. There are two basic strategies that applications follow while the application runs: - Install a resize handler and call `prune()` after a specified number of resize operations or when the store reaches a specific size. - Call `prune()` after a specific number of messages are processed (for example, every 10,000 messages received or every 1,000 updates completed). Regardless of the strategy, it is best to call `prune()` when the application is idle, since the `prune()` call rewrites the log file. The store interface is public and an application can create and provide a custom store as necessary. While clients provide convenience methods for creating file-backed and memory-backed `HAClient` objects with the appropriate stores, you can also create and set the stores in your application code. For the AMPS Python client, stores are implemented in C++. You can implement stores using C++, and use the technique described in Chapter 12 of this guide - *Using the C++ Client*, to set the store on the client. Starting in 5.3.2.0, the underlying AMPS client contains a recovery point adapter interface to make it easier to add a custom persistence layer to a bookmark store. The distribution includes a recovery point adapter that can store bookmark recovery information in an AMPS SOW topic. The `HAClient` provides convenience methods for creating clients and setting stores. You can also construct an `HAClient` and set the store implementations you choose. In this example, we create several clients. The first client uses memory stores for both bookmarks and publishes. The second client uses files for both bookmarks and publishes. The third client uses a file for bookmarks. The third client does not set a store for publishes, which means that AMPS provides the default store (and no outgoing messages are stored). The final client does not specify any stores, so has no persistence for published messages or bookmark subscriptions, but can take advantage of the automatic failover and reconnection in the `HAClient`. ```python showLineNumbers # Memory publish store, memory bookmark store memory_client = AMPS.HAClient("lessImportantMessages") # File-backed publish store, file-backed bookmark store disk_client = AMPS.HAClient("moreImportantMessages", "/mnt/fastDisk/moreImportantMessages.outgoing", "/mnt/fastDisk/moreImportantMessages.incoming") # No-op publish store, file-backed bookmark store subscriber_client = AMPS.HAClient("subscriber", no_store=True) subscriber_client.set_bookmark_store( \ AMPS.MMapBookmarkStore("/mnt/fastdisk/bookmark.store")) # No-op publish store, no-op bookmark store # Failover behavior only. stream_reader = AMPS.HAClient("streamReader", no_store=True) ``` ### Using the SOW Recovery Point Adapter The AMPS client also includes the ability to use a SOW topic to store bookmark state for a bookmark store. This can be a useful option in a situation where an application needs a persistent bookmark store, but does not have the ability to store a file on the filesystem, or where an application has a bookmark file, but wants to have the ability to resume the subscription if the file is lost or damaged, or if the application is started on a system that does not have access to the file. To use the SOW topic recovery point adapter, you create a bookmark store of the type you would like to use for the `Client`, passing an adapter when you construct the store. You then set this bookmark store as the store for the `Client` to use. The constructor for the SOW recovery adapter allows you to customize the topic name and field names used to store the recovery point information in AMPS. As with the `RecoveryPointAdapter` interface in general, it is possible to customize the behavior of the SOW recovery point adapter by overriding the provided methods. This section describes how to use the adapter with the default settings. Should you need to change the behavior of the class, you would adjust the guidance in this section accordingly. (For example, if you override methods to produce a message with a different set of keys or a different message format, you would update the topic definition accordingly). ### AMPS Topic Configuration To store recovery point state in AMPS, the AMPS instance that will store the recovery point state must define a `SOW/Topic` to hold the recovery point data. By default, the adapter uses a topic named `/ADMIN/bookmark_store` of `json` message type, with the `/clientName` and `/subId` fields as keys, similar to the following definition: ```xml showLineNumbers /ADMIN/bookmark_store json /clientName /subId ``` You must include this definition, or an equivalent definition, in the configuration file for the AMPS instance that will host the recovery point. If you define a topic with a different configuration (for example, different key names, a different topic name or a different message type), you must ensure that the adapter that you create uses the same parameters as those configured on the server. ### Constructing a Client for the Adapter The AMPS SOW Recovery Point Adapter requires a `Client` or `HAClient` connected to the instance that contains the SOW topic. The Adapter will use this client to recover bookmark state and store bookmarks in AMPS. Notice that this client **must not** be a client that the Adapter is keeping state for. This must be a completely separate client instance, otherwise the client may deadlock while updating the store. The client must be connected and logged in to the instance that contains the SOW topic, using the message type defined for the topic. ### Capacity Planning and Store Sizing When an application uses a file-backed store, it is important to make sure that there is enough space available on the file system to be able to manage the store. For logged bookmark stores, an application needs to keep a bookmark record for each message received, each message discarded, and the persisted acknowledgments delivered by the server approximately once a second. Each bookmark entry consumes roughly 70 bytes of storage *plus* the length of the subscription ID for the subscription receiving the message. The logged bookmark store retains entries until an application explicitly calls `prune()`. The capacity needed for a logged bookmark store will depend on the strategy that the application uses for pruning the file. For a file-backed publish store, the application needs to be able to store published messages until the AMPS server that the publisher is connected to acknowledges those messages as persisted. The volume of messages that needs to be stored depends on the failover policy for the server -- that is, the maximum amount of time that the server will allow a downstream instance to fail to acknowledge a message before the server downgrades that connection to `async` acknowledgment. By default, AMPS does not downgrade connections: this policy must be set explicitly using the AMPS actions. As an example, if the server is configured to downgrade connections that are more than 120 seconds behind, then -- for disaster recovery -- the application must have the capacity to store 120 seconds of published messages at peak publishing load. However, unlike the logged bookmark store, a file-backed publish store removes messages from the store and reuses the space once AMPS has acknowledged the message. ## Connections and the Server Chooser Unlike `Client`, the `HAClient` attempts to keep itself connected to an AMPS instance at all times, by automatically reconnecting or failing over when it detects that the client is disconnected. When you are using the `Client` directly, your disconnect handler usually takes care of reconnection. `HAClient`, on the other hand, provides a disconnect handler that automatically reconnects to the current server or to the next available server. To inform the `HAClient` of the addresses of the AMPS instances in your system, you pass a `ServerChooser` instance to the `HAClient`. `ServerChooser` acts as a smart enumerator over the servers available: `HAClient` calls `ServerChooser` methods to inquire about what server should be connected, and calls methods to indicate whether a given server succeeded or failed. The AMPS Python client provides a simple implementation of `ServerChooser`, called `DefaultServerChooser`, that provides very simple logic for reconnecting. This server chooser is most suitable for basic testing, or in cases where an application should simply rotate through a list of servers. For most applications, you implement the `ServerChooser` interface yourself for more advanced logic, such as choosing a backup server based on your network topology, or limiting the number of times your application should try to reconnect to a given address. To connect to AMPS, you provide a `ServerChooser` to `HAClient` and then call `connect_and_logon()` to create the first connection: ```python showLineNumbers memory_client = AMPS.HAClient("myClient") # primary.amps.xyz.com is the primary AMPS instance, and # secondary.amps.xyz.com is the secondary chooser = AMPS.DefaultServerChooser() chooser.add("tcp://primary.amps.xyz.com:12345/fix") chooser.add("tcp://secondary.amps.xyz.com:12345/fix") memory_client.set_server_chooser(chooser) memory_client.connect_and_logon() ... memory_client.disconnect() ``` Similar to `Client`, `HAClient` remains connected to the server until `disconnect()` is called. Unlike `Client`, `HAClient` automatically attempts to reconnect to your server if it detects a disconnect and if that server cannot be connected, fails over to the next server provided by the `ServerChooser`. In this example, the call to `connect_and_logon()` attempts to connect and login to `primary.amps.xyz.com`, and returns if that is successful. If it cannot connect, it tries `secondary.amps.xyz.com`, and continues trying servers from the `ServerChooser` until a connection is established. Likewise, if it detects a disconnection while the client is in use, then `HAClient` attempts to reconnect to the server with which it was most recently connected; if that is not possible, then it moves on to the next server provided by the `ServerChooser`. The default `ServerChooser` simply provides the next URL in the sequence. This strategy works for many applications. If you need a different strategy, you can implement your own logic for failover by creating a class derived from `ServerChooser`. ### Setting a Reconnect Delay and Timeout You can control the amount of time between reconnection attempts and set a total amount of time for the `HAClient` to attempt to reconnect. The AMPS Python client includes a method for setting a delay strategy on a client, `set_reconnect_delay_strategy`. This method accepts an instance of any type that provides the methods `get_connect_wait_duration` and `reset`, as described in the API documentation. While you can easily implement your own delay strategy, the client also provides two delay strategies: - `FixedDelayStrategy` provides the same delay each time the `HAClient` tries to reconnect. - `ExponentialDelayStrategy` provides an exponential backoff until a connection attempt succeeds. To use either of these classes, you simply create an instance, set the appropriate parameters, and install that instance as the delay strategy for the `HAClient`. For example, the following code sets up a reconnect delay that starts at 200ms and increases the delay by 1.5 times after each failure. The strategy allows a maximum delay between connection attempts of 5 seconds, and will not retry longer than 60 seconds. ```python showLineNumbers client = AMPS.HAClient("myClient") client.set_reconnect_delay_strategy( \ AMPS.ExponentialDelayStrategy( \ initial_delay=200, \ backoff_exponent=1.5, \ maximum_delay=5000, \ maximum_retry_time=60000) \ ) ``` ### Implementing a Server Chooser As described above, you provide the `HAClient` with connection strings to one or more AMPS servers using a `ServerChooser`. The purpose of a `ServerChooser` is to provide information to the `HAClient`. A `ServerChooser` does not manage the reconnection process, and should not call methods on the `HAClient`. A `ServerChooser` has two required responsibilities to the `HAClient`: - Tells the `HAClient` the connection string for the server to connect to. If there are no servers, or the `ServerChooser` wants the connection to fail, the `ServerChooser` returns an empty string. To provide this information, the `ServerChooser` implements the `get_current_uri()` method. - Provides an `Authenticator` for the current connection string. This is especially important for installations where different servers require different credentials or authentication tokens must be reset after each connection attempt. To provide the authenticator, the `ServerChooser` implements the `get_current_authenticator()` method. The `HAClient` calls the `get_current_uri()` and `get_current_authenticator()` methods each time it needs to make a connection. Each time a connection succeeds, the `HAClient` calls the `report_success()` method of the `ServerChooser`. Each time a connection fails, the `HAClient` calls the `report_failure()` method of the `ServerChooser`. The `HAClient` does not require the `ServerChooser` to take any particular action when it calls these methods. These methods are provided for the `HAClient` to do internal maintenance, logging, or record keeping. For example, an `HAClient` might keep a list of available URIs with a current failure count, and skip over URIs that have failed more than 5 consecutive times until all URIs in the list have failed more than 5 consecutive times. When the `ServerChooser` returns an empty string from `get_current_uri()`, indicating that no servers are available for connection, the `HAClient` calls the `get_error()` method on the `ServerChooser`, if one is provided, and includes the string returned by `get_error()` in the generated exception. ## Heartbeats and Failure Detection Use of the `HAClient` allows your application to quickly recover from detected connection failures. By default, connection failure detection occurs when AMPS receives an operating system error on the connection. This system may result in unpredictable delays in detecting a connection failure on the client, particularly when failures in network routing hardware occur, and the client primarily acts as a subscriber. The heartbeat feature of the AMPS client allows connection failure to be detected quickly. Heartbeats ensure that regular messages are sent between the AMPS client and server on a predictable schedule. The AMPS client and server both assume disconnection has occurred if these regular heartbeats cease, ensuring disconnection is detected in a timely manner. To use the heartbeat feature, call the `set_heartbeat` method on `Client` or `HAClient`: ```python showLineNumbers memory_client = AMPS.HAClient("importantStuff") ... memory_client.set_heartbeat(3) memory_client.connect_and_logon() ... ``` Method `set_heartbeat` takes one parameter: the heartbeat interval. The heartbeat interval specifies the periodicity of heartbeat messages sent by the server: the value `3` indicates messages are sent on a three-second interval. If the client receives no messages in a six-second window (two heartbeat intervals), the connection is assumed to be dead, and the `HAClient` attempts reconnection. An additional variant of `set_heartbeat` allows the idle period to be set to a value other than two heartbeat intervals. (The server, however, will always consider a connection to be closed after two heartbeat intervals without any traffic.) Notice that, for `HAClient`, `setHeartbeat` must be called *before* the client is connected. For `Client`, `setHeartbeat` must be called *after* the client is connected. :::warning Heartbeats are serviced on the receive thread created by the AMPS client. Your application must not block the receive thread for longer than the heartbeat interval or the application is subject to being disconnected. ::: ## Considerations for Publishers Publishing with an `HAClient` is nearly identical to regular publishing; you simply call the `publish()` method with your message's topic and data. The AMPS client sends the message to AMPS, and then returns from the `publish()` call. For maximum performance, the client does not wait for the AMPS server to acknowledge that the message has been received. When an `HAClient` sets a publish store, the publish store retains a copy of each outgoing message and requests that AMPS acknowledge that the message has been persisted. The AMPS server acknowledges messages back to the publisher. Acknowledgments can be delivered for multiple messages at periodic intervals (for topics recorded in the transaction log) or after each message (for topics that are not recorded in the transaction log). When an acknowledgment for a message is received, the `HAClient` removes that message from the bookmark store. When a connection to a server is made, the `HAClient` automatically determines which messages from the publish store (if any) the server has not processed, and replays those messages to the server once the connection is established. For reliable publishers, the application must choose how best to handle application shutdown. For example, it is possible for the network to fail immediately after the publisher sends the message, while the message is still in transit. In this case, the publisher has sent the message, but the server has not processed it and acknowledged it. During normal operation, the `HAClient` will automatically connect and retry the message. On shutdown, however, the application must decide whether to wait for messages to be acknowledged, or whether to exit. Publish store implementations provide an `unpersisted_count()` method that reports the number of messages that have not yet been acknowledged by the AMPS server. When the `unpersisted_count()` reaches `0`, there are no unpersisted messages in the local publish store. For the highest level of safety, an application can wait until the `unpersisted_count()` reaches `0`, which indicates that all of the messages have been persisted to the instance that the application is connected to, and the synchronous replication destinations configured for that instance. When a synchronous replication destination goes offline, this approach will cause the publisher to wait to exit until the destination comes back online or until the destination is downgraded to asynchronous replication. For applications that are shut down periodically for short periods of time (for example, applications that are only offline during a weekly maintenance window), another approach is to use the `publish_flush()` method to ensure that messages are delivered to AMPS, and then rely on the connection logic to replay messages as necessary when the application restarts. For example, the following code flushes messages to AMPS, then warns if not all messages have been acknowledged: ```python showLineNumbers client = AMPS.HAClient("ha-publisher", "/mnt/fastDisk/moreImportantMessages.outgoing", "/mnt/fastDisk/moreImportantMessages.incoming") ... client.connect_and_logon() # Publish messages ... # We think we are done, but the server may not # have received or acknowledged all messages yet. # Wait until the server has received all messages. # The program could also specify a timeout in this # command to avoid blocking forever if the network # is down or all servers are offline. client.publish_flush() # Print warning to the console if messages have # been published but not yet acknowledged as # persisted if (client.get_unpersisted_count() > 0): print( "all messages have been published, " \ + " but not all have been persisted" ) client.disconnect() ``` In this example, the client sends each message immediately when `publish()` is called. If AMPS becomes unavailable between the final `publish()` and the `disconnect()`, or one of the servers that the AMPS instance replicates to is offline, the client may not have received a persisted acknowledgment for all of the published messages. For example, if a message has not yet been persisted by all of the servers in the replication fabric that are connected with synchronous replication, AMPS will not have acknowledged the message. Before shutting down the client, the code does two things: - First, the code flushes messages to the server to ensure that all messages have been delivered to AMPS. - Next, the code checks to see if all of the messages in the publish store have been acknowledged as persisted by AMPS. If the messages have not been acknowledged, they will remain in the publish store file and will be published to AMPS, if necessary, the next time the application connects. An application may choose to loop until `get_unpersisted_count()` returns `0`, or (as we do in this case) simply warn that AMPS has not confirmed that the messages are fully persisted. The behavior you choose in your application should be consistent with the high-availability guarantees your application needs to provide. :::warning AMPS uses the name of the `HAClient` to determine the origin of messages. For the AMPS server to correctly identify duplicate messages, each instance of an application that publishes messages must use a distinct name. That name must be consistent across different runs of the application. ::: If your application crashes or is terminated, some published messages may not have been persisted in the AMPS server. If you use the file-based store—in other words, if you provide file names for persistent storage when you create the `HAClient`—the `HAClient` will recover the messages, and once logged on, will correlate the message store to what the AMPS server has received, re-publishing any missing messages. This occurs automatically when `HAClient` connects, without any explicit consideration in your code, other than ensuring that the same file name is used to create the `HAClient` if recovery is desired. :::warning AMPS provides persisted acknowledgment messages for topics that do not have a transaction log enabled. However, the level of durability provided for topics with no transaction log is minimal. Learn more about transaction logs in the *AMPS User Guide*. ::: ## Detecting Failover Ahead of Replication AMPS replication provides two different acknowledgment modes for outgoing replication links from an instance: - For a link in `sync` acknowledgment mode, a message must be successfully acknowledged by the downstream instance of AMPS before this instance of AMPS will acknowledge the message. - For a link in `async` acknowledgment mode, this link is not considered for acknowledging the message. In this mode, the downstream side of the replication link may not have received or processed the message at the time that the publisher receives an acknowledgment. As described in the *AMPS User Guide*, a publisher must not failover from one instance of AMPS to another instance when any link between those instances uses `async` acknowledgment *unless* replication is certain to have reached that instance. (For example, if replication is taking a maximum of 1.2 seconds between the instances and the publisher has been disconnected for 30 seconds, all messages from that publisher will have been replicated). To help detect a situation where a publisher may be "jumping ahead" of messages that it has published, but which have not yet been replicated, the AMPS client allows an application to consider it to be an error to make a connection to a server that has not received messages previously published by the application. To enable this behavior, set the `set_error_on_publish_gap()` method to set this property on the `PublishStore` in use for the client. When this property is set, the client will consider it to be an error to connect to a server that has not received messages previously published by the client, and consider the connection to have failed. Notice that an application that uses this approach may need to handle situations where no server has received the message, particularly if the replication configuration uses automated replication downgrade. ## Considerations for Subscribers `HAClient` provides two important features for applications that subscribe to one or more topics: re-subscription, and a bookmark store to track the correct point at which to resume a bookmark subscription. ### Resubscription with Asynchronous Message Processing Any asynchronous subscription placed using an `HAClient` is automatically reinstated after a disconnect or a failover. These subscriptions are placed in an in-memory `SubscriptionManager`, which is created automatically when the `HAClient` is instantiated. When a re-subscription occurs, the AMPS Python client re-executes the command as originally submitted, including the original topic, options, and so on. AMPS sends the subscriber any messages for the specified topic (or topic expression) that are published after the subscription is placed. For a `sow_and_subscribe` command, this means that the client re-issues the full command, including the SOW query as well as the subscription. :::tip A `sow` command is a point-in-time query. It isn't added to the subscription manager, and isn't restarted if a disconnection happens in the middle of a query. A `sow_and_subscribe` is a subscription, and is added to the subscription manager. ::: ### Resubscription with Synchronous Message Processing The `HAClient` (starting with the AMPS Python client version 4.3.1.1) does not track synchronous message processing subscriptions in the `SubscriptionManager`. The reason for this is to preserve the iterator semantics. That is, once the `MessageStream` indicates that there are no more elements in the stream, it does not suddenly produce more elements. To re-subscribe when the `HAClient` fails over, you can simply re-issue the subscription. For example, the snippet below re-issues the subscribe command when the message stream ends: ```python showLineNumbers while still_need_to_process: # Exiting the for loop is the end of stream. # For a subscribe, this likely means that the # client has disconnected. try: for message in client.subscribe("messages"): # process messages here # check condition on still_need_to_process if still_need_to_process == False: break except AMPS.DisconnectedException as e: pass ``` ### Bookmark Stores In cases where it is critical not to miss a single message, it is important to be able to resume a subscription at the exact point that a failure occurred. In this case, simply recreating a subscription isn't sufficient. Even though the subscription is recreated, the subscriber may have been disconnected at precisely the wrong time and will not see the message. To ensure delivery of every message from a topic or set of topics, the AMPS `HAClient` includes a `BookmarkStore` that, combined with the bookmark subscription and transaction log functionality in the AMPS server, ensures that clients receive any messages that might have been missed. The client stores the bookmark associated with each message received, and tracks whether the application has processed that message; if a disconnect occurs, the client uses the `BookmarkStore` to determine the correct resubscription point, and sends that bookmark to AMPS when it re-subscribes. AMPS then replays messages from its transaction log from the point after the specified bookmark, thus ensuring the client is completely up-to-date. `HAClient` helps you to take advantage of this bookmark mechanism through the `BookmarkStore` interface and `bookmarkSubscribe()` method on `Client`. When you create subscriptions with `bookmarkSubscribe()`, whenever a disconnection or failover occurs, your application automatically re-subscribes to the message after the last message it processed. `HAClients` created by `createFileBacked()` additionally store these bookmarks on disk, so that the application can restart with the appropriate message if the client application fails and restarts. To take advantage of bookmark subscriptions, do the following: - Ensure the topic(s) to be subscribed to are included in a transaction log. See the *AMPS User Guide* for information on how to specify the contents of a transaction log. - Use `bookmark_subscribe()` instead of `subscribe()` when creating a `subscription()` and decide how the application will manage subscription identifiers (SubIds). If you are using a `Command` object, you can simply set the bookmark on that object. - Use the `discard()` method in message handlers to indicate when a message has been fully processed by the application, that is, when the application does not need to receive the message again if the application fails over. The following example creates a bookmark subscription against a transaction-logged topic and fully processes each message as soon as it is delivered: ```python showLineNumbers class MessagePrinter(object): def __init__(self, client): self._client = client def __call__(self, message): print (message.get_data()) self._client.discard(message) ... client = AMPS.HAClient( "aClient", "/logs/aClient.publishLog", "/logs/aClient.subscribeLog") # Create ServerChooser, populate chooser, connect client ... client.execute_async( \ AMPS.Command("subscribe") \ .set_topic("myTopic") \ .set_bookmark(AMPS.Client.Bookmarks.MOST_RECENT) \ .set_sub_id("MySubID"), \ MessagePrinter(client)) ``` In this example, the client is a file-backed client, meaning that arriving bookmarks will be stored in a file (`aClient.subscribeLog`). Storing these bookmarks in a file allows the application to restart the subscription from the last message processed, in the event of either server or client failure. :::info For optimum performance, it is critical to discard every message once its processing is complete. If a message is never discarded, it remains in the bookmark store. During re-subscription, `HAClient` always restarts the bookmark subscription with the oldest undiscarded message, and then filters out any more recent messages that have been discarded. If an old message remains in the store, but is no longer important for the application’s functioning, then the client and the AMPS server will incur unnecessary network, disk and CPU activity. ::: The fourth parameter, `sub_id`, specifies an identifier to be used for this subscription. Passing `None` causes `HAClient` to generate one and return it, like most other `Client` functions. However, if you wish to resume a subscription from a previous point after the application has terminated and restarted, the application must pass the same subscription ID as during its previous run. Passing a different subscription ID bypasses any recovery mechanisms, creating an entirely new subscription. When you use an existing subscription ID, the `HAClient` locates the last-used bookmark for that subscription in the local store, and attempts to re-subscribe from that point. Below are the different bookmark types that can be used to enable different recovery strategies for an application: - `Client.Bookmarks.NOW` specifies that the subscription should begin from the moment the server receives the subscription request. This results in the same messages being delivered as if you had invoked `subscribe()` instead, except that the messages will be accompanied by bookmarks. This is also the behavior that results if you supply an invalid bookmark. - `Client.Bookmarks.EPOCH` specifies that the subscription should begin from the beginning of the AMPS transaction log (that is, the first entry in the oldest journal file for the transaction log). - `Client.Bookmarks.MOST_RECENT` specifies that the subscription should begin from the last-used message in the associated `BookmarkStore`. Alternatively, if this subscription has not been seen before, it instructs the subscription to begin with `EPOCH`. This is the most common value for this parameter and is the value used in the preceding example. By using `MOST_RECENT`, the application automatically resumes from wherever the subscription left off, taking into account any messages that have already been processed and discarded. When the `HAClient` re-subscribes after a disconnection and reconnection, it always uses `MOST_RECENT`, ensuring that the continued subscription always begins from the last message used before the disconnect, so that no messages are missed. ## Conclusion With only a few changes, most AMPS applications can take advantage of the `HAClient` and associated classes to become more highly-available and resilient. Using the `PublishStore`, publishers can ensure that every message published has actually been persisted by AMPS. Using `BookmarkStore`, subscribers can make sure that there are no gaps or duplicates in the messages received. `HAClient` makes both kinds of applications more resilient to network and server outages, as well as temporary issues. By utilizing the file based `HAClient`, clients can recover their state after an unexpected termination or crash. Though `HAClient` provides useful defaults for the `Store`, `BookmarkStore`, and `ServerChooser`, you can customize any or all of these to the specific needs of your application and architecture. --- # Installation Options ## Installing From PyPI The AMPS Python client is published to the Python Package Index as `amps-python-client`. You can use `pip` to install the client directly from the repository using a command such as the following: ```bash $ pip install amps-python-client ``` ## Obtaining the Client Source The AMPS Python client is available for download from the 60East Technologies website via the below link. Download the client from the site, then extract it. AMPS Python Client Source The client source files are in the directory where you unpacked the files. By default, this is `amps-python-client-`, where `` is the current version of the python client (such as `amps-python-client-5.3.5.0`). ## Installing the Prebuilt .whl on Linux 60East provides a prebuilt `.whl` file for x64 Linux distributions using Python 3.4 and later. To install the `.whl` using the command line: 1. You will need permission to update the Python distribution on the system you are installing on. 2. Open a command prompt. 3. Run the following command, substituting the appropriate path to the release wheel that you want to install: If this command reports a permission error, you do not have permission to update the Python distribution. Run the command as a different user, or use `sudo` to run the command as `root`. ## Installing the Prebuilt .whl on Windows 60East provides a prebuilt `.whl` file for 64-bit Windows operating systems using Python 3.4 and later. Your Python distribution may include a more fully-featured package manager to assist with installing `.whl` files. To install the `.whl` using the command line: 1. You will need permission to update the Python distribution on the system you are installing on. 2. Make sure that your Python distribution includes setuptools. 3. Open a command prompt. 4. Make sure that the directory that contains python is in your path. 5. Run the following command, substituting the appropriate path to the release wheel that you want to install: If this command reports a permission error, you do not have permission to update the Python distribution. Run the command prompt from a different user, or start the command prompt with **Run as Administrator**. ## Installing the Prebuilt .whl on MacOS 60East provides a prebuilt `.whl` file for 64-bit MacOS operating systems using Python 3.4 and later. Your Python distribution may include a more fully-featured package manager to assist with installing `.whl` files. To install the `.whl` using the command line: 1. You will need permission to update the Python distribution on the system you are installing on. 2. Make sure that your Python distribution includes setuptools. 3. Open a command prompt. 4. Make sure that the directory that contains python is in your path. 5. Run the following command, substituting the appropriate path to the release wheel that you want to install: If this command reports a permission error, you do not have permission to update the Python distribution. Run the command as a different user, or use `sudo` to run the command. ## Building the Client The main AMPS Python client distribution includes the full source to the client. For most installations, you build the client with your Python distribution before using it. The process for building the client differs slightly depending on whether you are building the client for Linux or for Windows. ### Building for Linux Follow these steps to build the Linux version of the client: 1. The Python client includes all necessary C++ client sources. If you are using a different version of the C++ client than the one included with the Python client, set the `AMPS_CPP_DIR` environment variable to the location of the AMPS C++ client source. 2. Run `python setup.py build` from the AMPS Python client directory to build the client. This script uses the Python `setuptools` to build the library, which makes it easy to build the library correctly and install the library into your Python distribution. To see the options available in your environment, run `python setup.py --help`. :::info The module must be built with the same C++ compiler used to build python on your system. The `setup.py` script and `setuptools` package will generally ensure this unless you have added a different compiler to your `PATH`. If you typically use a different C++ compiler, remove the path to that compiler before running `setup.py`. ::: 3. The script builds the module to a path in the `build` directory. The exact path depends on the version of Python you are building with. 4. Add the build directory path to the PYTHONPATH environment variable. For example: `$ export PYTHONPATH=/home/AMPSdev/amps-python-client/build/lib.linux-x86_64-3.6:$PYTHONPATH` 5. Test that the module loads correctly: `$ python -c "import AMPS"` 6. Optionally, install the module to your local python installation. While this is not required, doing this makes the AMPS module available for all of the programs that use this installation of python: `sudo python setup.py install` ### Building for Windows Follow these steps to build the Windows version of the client: 1. Use the Visual Studio Command Prompt shortcut to start a command prompt window for the type of module you want to build. For example, to build a 32-bit module, you use the command prompt for x86 builds. 2. Add the Python directory (the location of the `python.exe` interpreter) to your `PATH`. :::info The platform of the python installation must match the target platform for the python module. If you want to build a 64-bit module, your PATH must include a 64-bit python installation. If you want to build a 32-bit module, your PATH must include a 32-bit python installation. The build process uses whatever python.exe is found first when searching the PATH. So, to build both 32-bit and 64-bit versions, you must build them separately, and change your PATH between builds. ::: 3. Run `python setup.py build` from the AMPS Python client directory to build the client. This script uses the Python `setuptools` to build the library, which makes it easy to build the library correctly and install the library into your Python distribution. To see the options available in your environment, run `python setup.py --help`. 4. The script builds the module to a path in the `build` directory. The exact path depends on the version of Python you are building with. 5. Add the build directory path to the PYTHONPATH environment variable. For example: `> set PYTHONPATH="C:\Users\AMPSdev\amps-python-client\build\win-amd64-3.9;%PYTHONPATH%"` 6. Test that the module loads correctly: `> python -c "import AMPS"` 7. Optionally, install the module to your local python installation.While this is not required, doing this makes the AMPS module available for all of the programs that use this installation of python: `> python setup.py install` ## About the Client Library The AMPS client library is packaged as a single binary file. The exact name of the file depends on the Python version and build environment. You can find the file under the `build` directory of your AMPS Python client install once you've completed the build process. If you have used a prepackaged `.egg` or `.whl` to install the Python client, the appropriate binary file will be installed in your Python environment. Every Python application you build will need to be able to reference the library. If you choose to install the python client into your local Python installation, then Python has access to the client library in the installation, and you do not need to include the library with each specific script. Otherwise, you will need to package and include the library with your script and ensure that the library is in the path where python looks for shared libraries, *or* ensure that the system where your application will run has installed the appropriate prepackaged build. --- # Managing Disconnection The `HAClient` class, included with the AMPS Python client, contains a disconnect handler and other features for building highly-available applications. The `HAClient` includes features for managing a list of failover servers, resuming subscriptions, republishing in-flight messages, and other functionality that is commonly needed for high availability. 60East recommends using the `HAClient` for automatic reconnection wherever possible, as the HAClient disconnect handler has been carefully crafted to handle a wide variety of edge cases and potential failures. If an application needs to reconnect or fail over, use an `HAClient`, and the AMPS client library will automatically handle failover and reconnection. You control which servers the client fails over to using an implementation of the `ServerChooser` interface, and you can control the timing of the failover using an implementation of the `ReconnectDelayStrategy` interface. :::info For most applications, the combination of the `HAClient` disconnect handler and a `ConnectionStateListener` gives you the ability to monitor disconnections and add custom behavior at the appropriate point in the reconnection process. ::: If you need to add custom behavior to the failover (such as logging, resetting an internal cache, refreshing credentials and so on), the `ConnectionStateListener` class allows your application to be notified and take action when disconnection is detected and at each stage of the reconnection process. To extend the behavior of the AMPS client during reconnection, implement a `ConnectionStateListener` as described in the section on [Monitoring Connection State](monitoring-connection-state.md) and add it to the set of connection state listeners using `add_connection_state_listener`. --- # Managing SOW Contents AMPS allows applications to manage the contents of the SOW by explicitly deleting messages that are no longer relevant. For example, if a particular delivery van is retired from service, the application can remove the record for the van by deleting the record for the van. The client provides the following methods for deleting records from the SOW: - `sow_delete` - Accepts a topic and filter, and deletes all messages that match the filter from the topic specified. - `sow_delete_by_keys` - Accepts a set of SOW keys as a comma-delimited string and deletes messages for those keys, regardless of the contents of the messages. SOW keys are provided in the header of a SOW message, and are the internal identifier AMPS uses for that SOW message. - `sow_delete_by_data` - Accepts a topic and message, and deletes the SOW record that would be updated by that message. The most efficient way to remove messages from the SOW is to use `sow_delete_by_keys` or `sow_delete_by_data`, since those options allow AMPS to exactly target the message or messages to be removed. Many applications use `sow_delete`, since this is the most flexible method for removing items from the SOW when the application does not have information on the exact messages to be removed. In either case, AMPS sends an OOF message to all subscribers who have received updates for the messages removed, as described in the previous section. The simple form of the `sow_delete` command returns a `Message`. This `Message` is an acknowledgment that contains information on the delete command. For example, the following snippet simply prints informational text with the number of messages deleted: ```python showLineNumbers msg = client.sow_delete( "sow_topic", "/id IN (42, 64, 37)" ) print(f"Got an {message.get_ack_type()} message containing {message.get_data()}: deleted {entries_deleted} SOW entries") ``` The `sow_delete` command also provides an asynchronous version that requires a message handler. This message handler is designed to receive `sow_delete` response messages from AMPS: ```python showLineNumbers def delete_handler(m): print(f"Got an {message.get_ack_type()} message containing {message.get_data()}: deleted {entries_deleted} SOW entries") client.execute_async( Command("sow_delete") \ .set_topic("sow_topic") \ .set_filter("/id IN (42, 64, 37)") \ .add_ack_type("stats"), \ delete_handler ) ``` Acknowledging messages from a queue uses a form of the `sow_delete` command that is only supported for queues. Acknowledgment is discussed in the [Using Queues](queues) chapter in this guide. --- # Manual Acknowledgment 60East generally recommends that applications use an `ack()` method to acknowledge messages during normal processing. This approach functions correctly when used within a message handler, supports batching as explained elsewhere in this chapter, and is generally both easier to code and more efficient. However, in some situations, you may need to manually acknowledge messages in the queue. This is most common when an application needs to operate on all messages with certain characteristics, rather than acknowledging individual messages. For example, an application that is doing updates to an order may want to cancel an order by both publishing a cancellation and immediately expiring all other messages in the queue for that order. With manual acknowledgment, that application can use a filter to remove all previous updates for that order, then publish the cancellation. To manually acknowledge processed messages and remove the messages from the queue, applications use the `sow_delete` command. To remove specific messages from the queue, provide the bookmarks of those messages. To remove messages that match a given filter, provide the filter. Notice that AMPS only supports using a bookmark with `sow_delete` when removing messages from a queue, not when removing records from a SOW. For example, given a `Message` object to acknowledge and a client, the code below acknowledges the message. ```python showLineNumbers # Provided as a demonstration for cases # where the message itself isn't available. # # Where the message is available, this method # is typically less efficient than # simply calling message.ack() def acknowledge_single(client,message): acknowledge = Command("sow_delete") acknowledge.set_topic(message.get_topic()).set_bookmark(message.get_bookmark()) client.execute_async(acknowledge,None) ``` In the example above, the program creates a `sow_delete` command, specifies the topic and the bookmark, and then sends the command to the server. Since the program does not need or expect a response from AMPS, this function provides `None` as the message handler. While this method works, creating and sending an acknowledgment for each individual message can be inefficient if your application is processing a large volume of messages. Rather than acknowledging each message individually, your application can build a comma-delimited list of bookmarks from the processed messages and acknowledge all of the messages at the same time. In this case, it's important to be sure that the number of messages you wait for is less than the maximum backlog -- the number of messages your client can have unacknowledged at a given time. Notice that both automatic acknowledgment and the helper method on the `Message` object take the maximum backlog into account. When constructing a command to acknowledge queue messages, AMPS allows an application to specify a filter rather than a set of bookmarks. AMPS interprets this as the client requesting acknowledgment of all messages that match the filter. (This may include messages that the client has not received, subject to the `Leasing` model for the queue.) As a more typical example of manual acknowledgment, the code below expires all messages for a given `id` that have a status other than `cancel`. An application might do this to halt processing of an order that it is about to cancel. ```python showLineNumbers def remove_pending(client, order_id): acknowledge = Command("sow_delete") acknowledge.set_topic(message.get_topic()) \ .set_filter(f"/id = '{order_id}' and /status != 'cancel'").set_options("expire") client.execute_async(acknowledge, None) ``` In the example above, the program specifies a topic and a filter to use to find the messages that should be removed. In this case, the program also provides the `expire` option to indicate that the messages have been removed from the queue rather than successfully processed (of course, whether this is the correct behavior for a canceled order depends on the expected message flow for your application). Notice that, as described in [Understanding Threading](understanding-threading.md), this method of acknowledging a message should not be used from a message handler unless the `sow_delete` is sent from a different client than the client that called the message handler. Instead, 60East recommends using the `ack()` function from within a message handler. --- # Understanding Message Objects So far, we have seen that subscribing to a topic involves working with objects of `AMPS.Message` type. A `Message` represents a single message to or from an AMPS server. Messages are received or sent for every client/server operation in AMPS. ## Header Properties There are two parts of each message in AMPS: a set of headers that provide metadata for the message, and the data that the message contains. Every AMPS message has one or more header fields defined. The precise headers present depend on the type and context of the message. There are many possible fields in any given message, but only a few are used for any given message. For each header field, the `Message` class contains a distinct property that allows for retrieval and setting of that field. For example, the `Message.get_command_id()` function corresponds to the `commandId` header field, the `Message.get_batch_size()` function corresponds to the `BatchSize header` field, and so on. For more information on these header fields, consult the *AMPS User Guide* and *AMPS Command Reference*. To work with header fields, a `Message` contains `get_xxx()` / `set_xxx()` methods corresponding to the header fields, as well as a number of `getXXX()` / `setXXX()` methods for compatibility with previous implementations of the AMPS Python client. 60East does not recommend attempting to parse header fields from the raw data of the message. ## get_data() Method Access to the data section of a message is provided via the `get_data()` method. The `data` property will contain the unparsed data of the message. Your application code parses and works with the data. The AMPS Python client contains a collection of helper classes for working with message types that are specific to AMPS (for example, FIX, NVFIX, and AMPS composite message types). For message types that are widely used, such as JSON or XML, you can use the standard Python facilities or the library you typically use in your environment. ## Message Field Reference The [AMPS Command Reference](/docs/amps-command-reference) contains a full description of which fields are available and which fields are returned in response to specific commands. --- # Monitoring Connection State The AMPS client interface provides the ability to set one or more connection state listeners. A connection state listener is a callback that is invoked when the AMPS client detects a change to the connection state. A connection state listener may be called from the client receive thread. An application should not submit commands to AMPS from a connection state listener, or the application risks creating a deadlock for commands that wait for acknowledgement from the server. The AMPS client provides the following state values for a connection state listener: |State |Indicates | |--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`Connected` |

    The client has established a connection to AMPS. If you are using a `Client`, this is delivered when `connect()` is successful.

    If you are using an `HAClient`, this state indicates that the `connect` part of the connect and logon process has completed. An `HAClient` using the default disconnect handler will attempt to log on immediately after delivering this state.

    Most applications that use `Client` will attempt to log on immediately after the call to `connect()` returns.

    An application should not submit commands to AMPS from the connection state listener while the client is in this state unless the application knows that the state has been delivered from a `Client` and that the `Client` does not call `logon()`.

    | |`LoggedOn` |

    The client has successfully logged on to AMPS. If you are using a `Client`, this is delivered when `logon()` is successful.

    If you are using an `HAClient`, this state indicates that the `logon` part of the connect and logon process has completed.

    This state is delivered after the client is logged on, but before recovery of client state is complete. Recovery will continue after delivering this state: the application should not submit commands to AMPS from the connection state listener while the client is in this state if further recovery will take place.

    | |`HeartbeatInitiated`|

    The client has successfully started heartbeat monitoring with AMPS. This state is delivered if the application has enabled heartbeating on the client.

    This state is delivered before recovery of the client state is complete. Recovery may continue after this state is delivered. The application should not submit commands to AMPS from the connection state listener until the client is completely recovered.

    | |`PublishReplayed` |

    Delivered when a client has completed replay of the publish store when recovering after connecting to AMPS.

    This state is delivered when the client has a PublishStore configured.

    If the client has a subscription manager set, (which is the default for an `HAClient`), the application should not submit commands from the connection state listener until the `Resubscribed` state is received.

    | |`Resubscribed` |

    Delivered when a client has re-entered subscriptions when recovering after connecting to AMPS.

    This state is delivered when the client has a subscription manager set (which is the default for an `HAClient`). This is the final recovery step. An application can submit commands to AMPS from the connection state listener after receiving this state.

    | |`Disconnected` |The client is not connected. For an `HAClient`, this means that the client will attempt to reconnect to AMPS. For a `Client`, this means that the client will invoke the disconnect handler, if one is specified.| |`Shutdown` |The client is shut down. For an `HAClient`, this means that the client will no longer attempt to reconnect to AMPS. This state is delivered when `close()` is called on the client or when a server chooser tells the `HAClient` to stop reconnecting to AMPS.| The enumeration provided for the connection state listener also includes a value of `UNKNOWN`, for use as a default or to represent additional states in a custom `Client` implementation. The 60East implementations of the client do not deliver this state. The following table shows examples of the set of states that will be delivered during connection, in order, depending on what features of the client are set. Notice that, for an instance of the `Client` class, this table assumes that the application calls both `connect()` and `logon()`. For an `HAClient`, this table assumes that the `HAClient` is using the default `DisconnectHandler` for the `HAClient`. |Configuration |States | |----------------------------------------------------------------------|------------------------------------------------------------------------------------------------------| |

    subscription manager

    publish store

    |

    `Connected`

    `LoggedOn`

    `PublishReplayed`

    `Resubscribed`

    | |

    subscription manager

    publish store

    heartbeat set

    |

    `Connected`

    `LoggedOn`

    `HeartbeatInitiated`

    `PublishReplayed`

    `Resubscribed`

    | |subscription manager |

    `Connected`

    `LoggedOn`

    `Resubscribed`

    | |

    subscription manager

    heartbeat set

    |

    `Connected`

    `LoggedOn`

    `HeartbeatInitiated`

    `Resubscribed`

    | |(default `Client` configuration) |

    `Connected`

    `LoggedOn`

    | --- # Performance Tips and Best Practices This chapter presents tips and techniques for writing high-performance applications with AMPS. This section presents principles and approaches that describe how to use the features of AMPS and the AMPS client libraries to achieve high performance and reliability. Specific techniques (for example, the details on how to write a message handler) are described in other parts of the AMPS documentation and referenced here. Other techniques require information specific to the application (for example, determining the minimum set of information required in a message), and are best done as part of your application design. All of the recommendations in this section are general guidelines. There are few, if any, universal rules for performance: at times, a design decision that is absolutely necessary to meet the requirements for an application might reduce performance somewhat. For example, your application might involve sending large binary data that cannot be incrementally updated. That application will use more bandwidth per message than an application that sends 100-byte messages with fields that can be incrementally updated. However, since the application depends on being able to deliver the binary payloads, this difference in bandwidth consumption is a part of the requirements for the application, not a design decision that can be optimized. ## Measure Performance and Set Goals The most important tools for creating high performance applications that use AMPS are clear goals and accurate measurement. Without accurate measurement, it's impossible to know whether a particular change has improved performance or not. Without clear goals, it's difficult to know whether a given result is sufficient, or whether you need to continue improving performance. 60East recommends that your measurements include baseline metrics for the part of your message processing that does not involve AMPS. As an example, imagine your task is to reduce the amount of time that elapses between when an order is sent and when the processed response is received from 100ms in total to 85ms in total. To achieve this reduction, you might first measure the processing that your application performs on the order. If that processing consumes 65ms, the most effective optimization may be to improve the order processing. On the other hand, if processing an order consumes 15ms, then optimizing message delivery or network utilization may be the most effective way to meet your goals. When measuring performance, simulate your production environment as closely as possible. For example, AMPS is highly parallelized, so sending a pattern of subscriptions and publishes from a single test client that would normally come from 20 clients will produce a very different performance profile. Likewise, AMPS can typically perform at rates that fill the available bandwidth. Performance measured on a 1GbE connection may be very different than performance measured over a 10GbE connection. Consider the characteristics of your data, and the number of messages you expect to store and process. A 1GB data set consisting of 1 million records will perform differently than a 1GB data set consisting of 10 million records, or a 1GB data set consisting of 100 records. When collecting information about performance, 60East recommends enabling persistence for the Statistics Database (`stats.db`), so you can easily collect historical data on both AMPS and the operating system. For example, a dip in performance correlated with high CPU and memory usage at the same time each day may be correlated with other activity on the system (such as cron jobs or close of business processing). In a situation like that, where the performance reduction is based on factors external to the AMPS application, the overall system metrics captured in `stats.db` can help you re-create the external state and understand the state of the system as a whole. AMPS collects the statistics in memory by default, and persisting that data into a database does not typically have a measurable effect on performance itself, but makes measuring and tuning performance much easier. For performance testing, 60East recommends using dedicated hardware for AMPS to eliminate the effects of other processes. If dedicated hardware is not available and other processes are consuming resources, 60East recommends disabling AMPS NUMA tuning to ensure that AMPS threads do not unnecessarily compete with other processes during performance tuning. ## Use HAClient and Heartbeating Where Appropriate Not every application that uses AMPS requires high availability and the ability to automatically fail over if connectivity is lost or an instance of AMPS is offline. For applications that do need automatic reconnection, 60East strongly recommends using the `HAClient` and setting heartbeating for the client to effectively detect disconnection. When using the `HAClient` and heartbeating, there are two important guidelines to follow: - Do not replace the disconnect handler on the `HAClient`. The disconnect handler is responsible for reconnection, resubscription, and so on. If you need to detect disconnection, use a connection state listener. - Set the interval for heartbeating to approximately one-half the time that the application can tolerate interruption in message flow. Notice that it's not possible for the `HAClient` to tell the difference between an interruption in message flow caused by a server going offline and interruptions caused by an increase in latency due to network saturation or so on, so the interval should be somewhat larger than the highest expected latency between AMPS and the application. Last, but not least, if the application uses asynchronous message handling, the interval should also be set to a value larger than the maximum amount of time expected for the message handler to process a single message. ## Simplify Message Format and Contents AMPS supports a wide range of message types, and is capable of filtering and processing large and complex messages. For many applications, the simplicity of being able to use messages that contain the full information is the most important consideration. For other applications, however, achieving the minimum possible latency and the maximum possible network utilization is important enough to warrant choosing a simplified message format. To simplify message contents, carefully consider the information that downstream processors require. If a downstream process will not use information in the message, there is no need to send the information. For example, consider an application that provides orders from a UI. In such an application, the object that represents the order often contains information relevant to the local state of the application that is not relevant to a downstream system. Rather than simply serializing the full object, your application may perform better if you serialize only the fields that a downstream system will take action on. To simplify message format, choose the simplest format that can convey the information that your application needs. The general principle is that the simpler the message format is, the more quickly AMPS and client libraries can parse messages of that type. Likewise, the more complicated the structure of each message is, the more work is required to parse the message. For the highest levels of performance, 60East recommends keeping the message structure simple and preferring message formats such as NVFIX, BFlat, or flattened JSON (structured as key/value pairs) as compared with more complicated formats such as XML or BSON. ## Measure Serialization and Deserialization When creating baseline performance numbers, measure serialization and deserialization performance independent of the AMPS server or client libraries. This can help you to: - Understand the baseline performance of creating and processing message data under ideal conditions (that is, where there is no application processing, networking, routing, etc. involved). - Easily compare the application-side performance of different message formats or different message layouts within a single format. When testing this performance, it is helpful to use data similar to the data that the application will actually process during a business day, at the volumes the application would typically process. This will help you understand the performance of serialization and deserialization for this specific application. For example, a library for working with a given message format might be less efficient when processing messages with a large number of string fields in a deeply-nested structure, but your application might exchange only numeric data in a relatively flat structure. Likewise, the library for a given format could be efficient for processing a small number of fields, but have lower performance for a message type with hundreds of fields. As with all performance testing, the more closely the test environment matches the actual data and volumes of a production environment, the more helpful those measurements will be for understanding system performance. ## Use Content Filtering Where Possible AMPS content filtering helps your application perform better by ensuring that your application only receives the messages that it needs. Wherever possible, we recommend using content filtering to precisely specify which messages your application needs. In particular, if at any point your application is receiving a message, parsing the message, and then determining whether to act on the message or not, 60East recommends using content filters to ensure that your application only receives messages that it needs to act on. ## Use Asynchronous Message Processing The synchronous message processing interface is straightforward, and presents a convenient interface for getting started with AMPS. However, the `MessageStream` used by the synchronous interface makes a full copy of each message and provides it from the background reader thread to the thread that consumes the message. This memory overhead and synchronization between the reader thread and consumer thread happens regardless of whether the application needs all of the header fields in the message or even processes the message. The `MessageStream` also does not take into account the speed at which your program is consuming messages, and will read messages into memory as fast as the network and processor allow. If your application cannot consume messages at wire speed, this can lead to increasing memory consumption as the application falls further behind the `MessageStream`. Most applications see improved performance by using a `MessageHandler`. With this approach, the `MessageHandler` does minimal work. If more extensive processing is needed, the `MessageHandler` dispatches the work to another thread: but it does this only when the work is necessary, and it only saves the part of the message needed to accomplish the work. ## Use Hash Indexes Where Possible for SOW Queries When querying a SOW, hash indexes on SOW topics are supported for exact matching on string data as described in the *AMPS User Guide*. A hash index can perform many times faster than a parallel query. If the query pattern for your application can take advantage of hash indexes, 60East recommends creating those hash indexes on your SOW topics. More recent versions of AMPS can use hash indexes for a wider variety of filters. When planning your queries, review the SOW queries section of the *AMPS User Guide* for the version you are using for guidelines on the optimizations available in that version. ## Use a Failed Write Handler and Exception Listener In many cases, particularly during the early stages of development, performance problems can point to defects in the application. Even after the application is tuned, monitoring for failure is important to keep applications running smoothly. 60East recommends always installing a failed write handler if your application is publishing messages. This will help you to quickly identify cases where AMPS is rejecting publishes due to entitlement failures, message type mismatches, or other similar problems. 60East recommends always installing an exception listener if your application is using asynchronous message processing. This will help you to identify and correct any problems with your message handler. An exception listener should typically log the message received and return. If recovery is needed, the listener should set a flag for another thread to process rather than attempting to recover on the thread that calls the exception listener. ## Reduce Bandwidth Requirements In many applications that use AMPS, network bandwidth is the single most important factor in overall performance. Your application can use bandwidth most efficiently by reducing message size. For example, rather than serializing an entire object, you might serialize only the fields that the remote process needs to act on, as mentioned above. Likewise, rather than sending one message that contains a collected set of information that processors will need to extract, consider sending a message in the units that processors will work with. This can reduce bandwidth to processors substantially. For example, rather than sending a single message with all of the activity for a single customer over a given period of time (such as a trading day), consider breaking out the record into the individual transactions for the customer. ### Tune Batch Size for SOW Queries As described in the section on [SOW Batch Size](/docs/amps-user-guide/sow-queries/batching-query-results), tuning the batch size for SOW queries can improve overall performance by improving network utilization. In addition, because the AMPS header is only parsed once per batch, a larger batch size can dramatically improve processing performance for smaller messages. The AMPS clients default to a batch size of `10`. This provides generally good performance for most transactional messages (such as order records or inventory records). For large messages, particularly messages greater than a megabyte in size, a batch size of `1` may reduce memory pressure in the client and improve performance. With smaller messages (for example, message sizes of a few hundred bytes), 60East recommends measuring performance with larger batch sizes such as `50` or `100`. For large messages, reducing the batch size may improve overall performance by requiring less memory consumption on the AMPS server. ### Conflate Fast-Changing Information If your data source publishes information faster than your clients need to consume it, consider using a conflated topic. For example, in a system that presents a user interface and displays fast-moving data, it is common for the data to change at a rate faster than the user interface can format and render the data. In this case, a conflated topic can both reduce bandwidth and simplify processing in the user interface. ### Minimize Bandwidth for Updates If your application uses a SOW and processes frequent updates, consider using delta publish and delta subscribe to reduce the size of the messages transmitted. These features are designed to minimize bandwidth while still providing full-fidelity data streams. ### Conflate Queue Acknowledgments The AMPS clients include the ability to conflate acknowledgments back to AMPS as queue messages are processed. Using these features, with an appropriate `max_backlog`, can reduce the amount of network traffic required for acknowledgments. ### Use a Transaction Log When Monitoring Publish Failures When a topic is not covered by a transaction log, AMPS returns acknowledgment messages for every publish that requests one. This ensures that each message is acknowledged, even when AMPS has no persistent record of the messages in the topic. However, acknowledging each message requires more network traffic for each publish message. When a topic is covered by a transaction log, AMPS conflates persisted acknowledgments. Conflation is possible in this case because AMPS has a full record of the messages and does not have to store additional state to conflate the acknowledgments. With conflated acknowledgments, AMPS will send a success acknowledgment periodically that covers all messages up to that point. If a message fails, AMPS immediately sends the conflated success acknowledgment for all previous messages and the failure acknowledgment for the failed message. ### Combine Conflation and Deltas In many cases, using an approach that combines delta publishes to a SOW with delta subscriptions to a conflated topic can dramatically reduce bandwidth to the application with no loss of information. ## Limit Unnecessary Copies One of the most effective ways to increase performance is to limit the amount of data copied within your application. For example, if your message handler submits work to a set of processors that only use the `Data` and `Bookmark` from a `Message`, create a data structure that holds only those fields and copy that information into instances of that data structure rather than copying the entire `Message`. While this approach requires a few extra lines of code, the performance benefits can be substantial. When publishing messages to AMPS, avoid unnecessary copies of the data. For example, if you have the data in a byte array, use the `publish` methods that use a byte array rather than converting the data to a string unnecessarily. Likewise, if you have the data in the form of a string, avoid converting it to a byte array where possible. ## Manage Publish Stores When using a publish store, the Client holds messages until they are acknowledged as persisted by AMPS, as determined by the replication configuration for the AMPS instance. In the event that an instance with `sync` replication goes offline, the publish store for the Client will grow, since the messages are not being fully persisted. To avoid this problem, 60East recommends that an instance that uses `sync` replication always configure Actions to automatically downgrade the replication link if the remote instance goes offline for a period of time, and upgrade the link when the remote instance comes back online. Further, 60East recommends that, where possible, a publisher is provisioned with enough storage to hold its complete publish stream for the amount of time that a destination may be offline or unavailable without downgrading from `sync` replication to `async` replication. For example, if the server considers a downstream system to be unreachable if it has not acknowledged a replicated message in 60 seconds, and the server checks this threshold every 10 seconds, then a publisher should plan that, at any time, the publisher may need to retain approximately 70 seconds worth of published messages. This is calculated as the 60 seconds threshold that the server has established for a destination to run behind, plus the 10 second interval at which the server checks whether the destination is within the threshold. Also notice that, with a configuration like this, a downstream replication destination could run as much as 59 seconds behind indefinitely. A publisher should be provisioned to be able to run effectively in a "worst case" (or nearly "worst case") scenario for an extended period of time. See the *High Availability and Replication* chapter in the *AMPS User Guide* for more information on replication, sync and async acknowledgment modes, and the Actions used to manage replication. ## Use the Server Logs to Help Troubleshoot When troubleshooting problems with an application that uses AMPS, the server-side logs often provide the most helpful information. For example, `trace` level logging shows the data that is flowing through AMPS. Log messages at `info` level show events as incoming connections, commands from clients, and so on. When questions arise about how the server and application interact, the server logs often contain the information. 60East recommends that an AMPS instance used for development and testing log at `trace` level, and that a server used for production log at `info` level, with the ability to log at `trace` level when necessary for investigating any problems that may arise. When a command does not have the expected result, or an application reports an error, the fastest way to understand the problem is often to review the `trace` level logging for the instance. See the *AMPS User Guide* for details on configuring logging and common patterns for searching for information in AMPS logs. ## Work with 60East as Necessary 60East offers performance advice adapted for your specific usage through your support agreement. Once you've set your performance goals, worked through the general best practices and applied the practices that make sense for your application, 60East can help with detailed performance tuning, including recommendations that are specific to your use case and performance needs. --- # Using Queues AMPS message queues provide a high-performance way of distributing messages across a set of workers. The *AMPS User Guide* describes AMPS [Queues](/docs/amps-user-guide/queues) in detail, including the features of AMPS referred to in this chapter. This chapter does not describe AMPS queues in detail, but instead explains how to use the AMPS Python client with message queues. To publish messages to a message queue, publishers simply publish to any topic that is collected by the queue. There is no difference between publishing to a queue and publishing to any other topic, and a publisher does not need to be aware that the topic will be collected into a queue. Subscribers must be aware that they are subscribing to a queue, and acknowledge messages from the queue when the message is processed. --- # Quickstart This page is intended to provide steamlined method for getting the AMPS Python Client installed and examples running. ## Installation ```bash $ pip install amps-python-client ``` Additional client install options can be found in [Installation Options](/clients/amps-client-python/installation_options). ## Examples Now that you have the client installed, here are some code examples showing how the python client is used. :::info These examples are all runnable, but they require an AMPS instance to connect to. The instructions for starting an instance of AMPS are available in the [Introduction to AMPS guide](/docs/intro-guide/getting_started/installation). ::: :::tip For more comprehensive detail on the python client see the other sections of this Developer Guide and the Python Client API Reference ::: ## Example 1: Connect and Subscribe In this example, we connect to an AMPS server running locally and initiate a subscription to the "messages" topic. As new messages are received, they are printed to the console. ```python showLineNumbers import AMPS uri = "tcp://localhost:9007/amps/json" amps = AMPS.Client("subscribe-example") amps.connect(uri) amps.logon() for message in amps.subscribe("messages"): print(message.get_data()) ``` ## Example 2: Publish a Message With AMPS, publishing is simple, as shown in this example. We connect to an AMPS server running locally, and publish a single message to the `messages` topic. To simply publish a message, there is no need to predeclare the topic or configure complex routing. Any subscription that has asked for JSON messages on the `messages` topic will receive the message. ```python showLineNumbers import AMPS uri = "tcp://localhost:9007/amps/json" client = AMPS.Client("publish-example") client.connect(uri) client.logon() client.publish("messages", '{"hi" : "Hello, world!"}') client.publish_flush() ``` ## Example 3: Query the Contents of a "SOW" Topic State-of-the-World ("SOW") topics in AMPS combine the power of a database table with the performance of a publish-subscribe system. Use the AMPS Python client to query the contents of a SOW topic. This example queries for all orders for the symbol `ROL`, and simply prints the messages to the console. ```python showLineNumbers import AMPS uri = "tcp://localhost:9007/amps/json" client = AMPS.Client("sow-example") client.connect(uri) client.logon() for message in client.sow("orders", "/symbol='ROL'"): if message.get_command() == AMPS.Message.Command.SOW: print(message.get_data()) ``` ## Example 4: Automatic Reconnection and Resubscription Rock-solid applications must be able to recover from network outages. The AMPS Python client includes an `HAClient` class that includes automatic reconnection and resubscription. Best of all, easy-to-implement interfaces control reconnection and resubscription behavior, allowing you to easily customize failover. The `HAClient` class can, optionally, also provide store-and-forward for reliable publish. With the AMPS transaction log, the class can provide resumable subscriptions that are guaranteed not to miss messages or receive duplicate messages, even in the case of failover between replicated servers. ```python showLineNumbers import AMPS client = AMPS.HAClient("reconnecting-subscriber") # The ServerChooser interface tells the HAClient which server # to connect to, both for the initial connection and failover. # The DefaultServerChooser is included with the client: many # applications implement a ServerChooser to control failover # behavior. chooser = AMPS.DefaultServerChooser() chooser.add("tcp://amps-server:9007/amps/json") chooser.add("tcp://amps-failover-server:9007/amps/json") client.set_server_chooser(chooser) client.connect_and_logon() def handle_messages(message): print(message.get_data()) # Subscribe. If the connection to the server # is lost, the HAClient will restore the connection to # the server or the failover partner, and restore # subscriptions. client.subscribe(handle_messages, "messages") ``` ## Example 5: Automatic Failover The AMPS Python client includes both a basic client, and a high availability client with additional features, including the ability to automatically failover if the client is disconnected. This example shows how to set up a high availability client for failover. This example creates an HA client and a server chooser for the client. The code then populates the server chooser with the list of failover servers, adds the chooser to the client, and then connects. Once the client is connected, you can use the HAClient object just like a regular AMPS client. You can take advantage of the extended features, such as durable publish, duplicate message protection, and so on -- see the Developer's Guide for more information! ```python showLineNumbers import AMPS client = AMPS.HAClient("haclient-with-failover") # create a server chooser chooser = AMPS.DefaultServerChooser() # add the addresses to use for failover chooser.add("tcp://primary.amps.xyz.com:12345/amps/json") chooser.add("tcp://secondary.amps.xyz.com:12345/amps/json") # set the server chooser for the client client.set_server_chooser(chooser) # connect and logon client.connect_and_logon() # now, use the client: if the client detects # a disconnection, it will reconnect. ... # at the end of the program, disconnect client.disconnect() ``` ## Example 6: Command interface and async vs sync execution The named methods such as `subscribe()`, `sow()`, and `publish()` are convenience methods for common commands. When you want more direct control over the command sent to AMPS, use the `AMPS.Command` interface described in [Creating Commands](/clients/amps-client-python/creating-commands). This interface is the most flexible way to set fields such as `topic`, `filter`, `options`, `ack_type`, `bookmark`, and `sub_id`. Use `execute()` when you want a `MessageStream` that you can iterate over on the calling thread. Use `execute_async()` when you want AMPS to process results on the client receive thread and dispatch messages to a handler function. ```python showLineNumbers import AMPS uri = "tcp://localhost:9007/amps/json" client = AMPS.HAClient("command-example") chooser = AMPS.DefaultServerChooser() chooser.add(uri) client.set_server_chooser(chooser) client.connect_and_logon() # Build a command explicitly and process the returned MessageStream # on the calling thread. sow_cmd = AMPS.Command("sow") \ .set_topic("orders") \ .set_filter("/symbol = 'ROL'") \ .add_ack_type("processed") for message in client.execute(sow_cmd): print(message.get_data()) # Using the same command, but dispatch messages to a handler # on the client receive thread. def on_message(message): print(message.get_data()) client.execute_async(sow_cmd, on_message) ``` ## Example 7: `sow_and_subscribe` with OOF Using `AMPS.Command` In this example, a `Command` is used to query a SOW topic, then receive updates. The `oof` option tells AMPS to also send out-of-focus messages when records no longer match the filter. ```python showLineNumbers import AMPS uri = "tcp://localhost:9007/amps/json" client = AMPS.HAClient("sow-and-subscribe-example") chooser = AMPS.DefaultServerChooser() chooser.add(uri) client.set_server_chooser(chooser) client.connect_and_logon() def on_message(message): if message.get_command() in (AMPS.Message.Command.Publish, AMPS.Message.Command.SOW): print(f"{message.get_command()}:", message.get_data()) elif message.get_command() == AMPS.Message.Command.OOF: print("OOF:", message.get_data()) sow_sub_cmd = AMPS.Command("sow_and_subscribe") \ .set_topic("orders") \ .set_filter("/status = 'OPEN'") \ .set_options("oof") client.execute_async(sow_sub_cmd, on_message) ``` ## Example 8: Subscribe to a Queue Topic with `max_backlog` For queue consumers, the `Command` interface makes it easy to request queue-specific behavior such as a larger `max_backlog`. This allows AMPS to pipeline queue messages to the client more efficiently and is the key to getting maximum performance with queues as explained in [Queues](/docs/amps-user-guide/queues) section of the User Guide. In this example, `add_ack_type("processed")` requests a `processed` acknowledgment which sets the `threading.Event` allowing the application wait until AMPS has confirmed that the subscription is active before continuing. ```python showLineNumbers import AMPS import threading uri = "tcp://localhost:9007/amps/json" client = AMPS.HAClient("queue-consumer-example") chooser = AMPS.DefaultServerChooser() chooser.add(uri) client.set_server_chooser(chooser) client.connect_and_logon() subscription_ready = threading.Event() def on_message(message): if message.get_command() == AMPS.Message.Command.Ack: if message.get_ack_type() == "processed": subscription_ready.set() else: print(message.get_data()) message.ack() queue_sub_cmd = AMPS.Command("subscribe") \ .set_topic("sample-queue") \ .set_options("max_backlog=10") \ .add_ack_type("processed") client.execute_async(queue_sub_cmd, on_message) subscription_ready.wait(timeout=5) ``` ## Example 9: Bookmark Subscribe with a Timestamp Bookmark If a topic is recorded in the AMPS transaction log, you can use a bookmark subscription to replay messages from a specific point in time and then cut over to the live stream. A timestamp bookmark uses the format `YYYYmmddTHHMMSSZ`. This command begins the subscription just after the provided timestamp bookmark. The `completed` acknowledgment is returned when the replay portion of the bookmark subscription is complete, and the handler class shows how to keep state such as an event or message counters in the handler itself. ```python showLineNumbers import AMPS import threading uri = "tcp://localhost:9007/amps/json" client = AMPS.HAClient("bookmark-subscribe-example") chooser = AMPS.DefaultServerChooser() chooser.add(uri) client.set_server_chooser(chooser) client.connect_and_logon() class MessageHandler(object): def __init__(self): self.replay_complete = threading.Event() self.message_count = 0 def __call__(self, message): if message.get_command() == AMPS.Message.Command.Ack: if message.get_ack_type() == "completed": self.replay_complete.set() else: self.message_count += 1 print(message.get_bookmark(), message.get_data()) bookmark_sub_cmd = AMPS.Command("subscribe") \ .set_topic("messages-history") \ .set_bookmark("20250401T153000Z") \ .add_ack_type("completed") handler = MessageHandler() client.execute_async(bookmark_sub_cmd, handler) handler.replay_complete.wait(timeout=5) ``` ## Example 10: Other Useful `AMPS.Command` Options Once you are using `AMPS.Command`, you can use the same interface to request other AMPS features. For example, you can request conflation, pagination with `top_n` and `skip_n`, or aggregation with `projection` and `grouping`. This example requests a page of `25` matching records after skipping the first `50`. Other useful command options include conflation for subscriptions and aggregation with `projection` and `grouping`. ```python showLineNumbers import AMPS uri = "tcp://localhost:9007/amps/json" client = AMPS.HAClient("paged-sow-example") chooser = AMPS.DefaultServerChooser() chooser.add(uri) client.set_server_chooser(chooser) client.connect_and_logon() sow_cmd = AMPS.Command("sow") \ .set_topic("orders") \ .set_filter("/status = 'OPEN'") \ .set_options("top_n=25,skip_n=50") for message in client.execute(sow_cmd): print(message.get_data()) ``` ## Additional Examples More examples can be found in the other section of this Developer Guide as well as in the API Reference. --- # Regular Expression Subscriptions Regular Expression (Regex) subscriptions allow a regular expression to be supplied in the place of a topic name. When you supply a regular expression, it is as if a subscription is made to every topic that matches your expression, including topics that do not yet exist at the time of creating the subscription. To use a regular expression, simply supply the regular expression in place of the topic name in the `subscribe()` call. For example: ```python showLineNumbers def message_handler(msg): topic = msg.get_topic() subscription_id = client.subscribe( message_handler, "orders.*" ) ``` In this example, messages on topics `orders-north-america`, `orders-europe`, and `new-orders` would match the regular expression. Messages published to any of those topics will be sent to our `message_handler` function. As in the example, you can use the `get_topic()` method to determine the actual topic of the message sent to the function. --- # Returning a Message to the Queue A subscriber can also explicitly release a message back to the queue. AMPS returns the message to the queue and redelivers the message just as though the lease had expired. To do this, the subscriber sends a `sow_delete` command with the bookmark of the message to release and the `cancel` option. When using automatic acknowledgments and the asynchronous API, AMPS will cancel a message if an exception is thrown from the message handler. To return a message to the queue, you can build a `sow_delete` acknowledgment using the `Command` class, or pass an option to the `ack()` method on the message. | Option | Result | | -------- | ----------------------------------------------- | | `cancel` | Returns the message to the queue. | | `expire` | Immediately expires the message from the queue. | For example, to return a message to a queue, call `ack()` on the message and pass the `cancel` option. ```python message.ack("cancel") ``` --- # Setting Batch Size The AMPS clients include a batch size parameter that specifies how many messages the AMPS server will return to the client in a single batch when returning the results of a SOW query. The 60East clients set a batch size of 10 by default. This batch size works well for common message sizes and network configurations. Adjusting the batch size may produce better network utilization and produce better performance overall for the application. The larger the batch size, the more messages AMPS will send to the network layer at a time. This can result in fewer packets being sent, and therefore less overhead in the network layer. The effect on performance is generally most noticeable for small messages, where setting a larger batch size will allow several messages to fit into a single packet. For larger messages, a batch size may still improve performance, but the improvement is less noticeable. In general, 60East recommends setting a batch size that is large enough to produce few partially-filled packets. Bear in mind that AMPS holds the messages in memory while batching them, and the client must also hold the messages in memory while receiving the messages. Using batch sizes that require large amounts of memory for these operations can reduce overall application performance, even if network utilization is good. For smaller message sizes, 60East recommends using the default batch size, and experimenting with tuning the batch size if performance improvements are necessary. For relatively large messages (especially messages with sizes over 1MB), 60East recommends explicitly setting a batch size of 1 as an initial value, and increasing the batch size only if performance testing with a larger batch size shows improved network utilization or faster overall performance. --- # SOW and Subscribe Imagine an application that displays real time information about the position and status of a fleet of delivery vans. When the application starts, it should display the current location of each of the vans along with their current status. As vans move around the city and post other status updates, the application should keep its display up to date. Vans upload information to the system by posting messages to the `van_location` topic, configured with a key of `van_id` on the AMPS server. In this application, it is important to not only stay up-to-date on the latest information about each van, but to ensure all of the active vans are displayed as soon as the application starts. Combining a SOW with a subscription to the topic is exactly what is needed, and that is accomplished by the `Client.sow_and_subscribe()` method. As with the other methods for receiving messages, the AMPS Python client provides a basic, synchronous form of `sow_and subscribe` that provides you with a `MessageStream` to iterate over, and an asynchronous form that requires a message handler. First, let's look at an example that uses the basic form of `sow_and_subscribe`: ```python showLineNumbers def report_van_position(client): # sow_and_subscribe command to begin receiving information about all of the active # delivery vans in the system. All of the vans in the system now are returned as # Message objects whose get_command returns sow. New messages coming in are # returned as Message objects whose get_command returns publish. for message in client.execute(Command("sow_and_subscribe") \ .set_topic("van_location") \ .set_filter("/status = 'ACTIVE'") \ .set_batch_size(100) \ .set_options("oof")): # Notice here that we specified the oof option to the command. Setting this # option causes AMPS to send Out-of-Focus (OOF) messages for the topic. # OOF messages are sent when an entry that was sent to us in the past no # longer matches our query. This happens when an entry is removed from the # SOW cache via a sow_delete operation, when the entry expires (as specified # by the expiration time on the message or by the configuration of that topic # on the AMPS server), or when the entry no longer matches the content filter # specified. In our case, if a van’s status changes to something other than # ACTIVE, it no longer matches the content filter, and becomes out of focus. # When this occurs, a Message is sent with Command set to oof. We use OOF # messages to remove vans from the display as they become inactive, expire, # or are deleted. if (message.get_command() == Message.Command.SOW or message.get_command() == Message.Command.Publish): # For each of these messages we call add_or_update_van(), that presumably adds # the van to our application’s display. As vans send updates to the AMPS server, # those are also received by the client because of the subscription placed by # sow_and_subscribe(). Our application does not need to distinguish between # updates and the original set of vans we found via the SOW query, so we use # add_or_update_van() to display the new position of vans as well. add_or_update_van(message) elif message.get_command() == Message.Command.OOF: remove_van(message) ``` Now we will look at an example that uses the asynchronous form of `sow_and_subscribe`: ```python showLineNumbers def update_van_position(message): if (message.get_command() == Message.Command.SOW or message.get_command() == Message.Command.Publish): add_or_update_van(message) elif message.get_command() == Message.Command.OOF: remove_van(message) def subscribe_to_van_location(client): client.execute_async( Command("sow_and_subscribe") \ .set_topic("van_location") \ .set_filter("/status = 'ACTIVE'") \ .set_batch_size(100) \ .set_options("oof"), \ update_van_position) ``` Notice that the two forms have the same result. However, one form performs processing on a background thread, and blocks the client from receiving messages while that processing happens. The other form processes messages on the main thread and allows the background thread to continue to receive messages while processing occurs. In both cases, the calls to `add_or_update_van` and `remove_van` receive the same data. --- # State of the World (SOW) AMPS State of the World (SOW) allows you to automatically keep and query the latest information about a topic on the AMPS server, without building a separate database. Using SOW lets you build impressively high-performance applications that provide rich experiences to users. The AMPS Python client lets you query SOW topics and subscribe to changes with ease. ## Performing SOW Queries To begin, we will look at a simple example of issuing a SOW query. ```python showLineNumbers for message in client.sow("orders", "/symbol='ROL'"): if message.get_command() == AMPS.Message.Command.GroupBegin: print("--- Begin SOW Results ---") if message.get_command() == AMPS.Message.Command.SOW: print(message.get_data()) if message.get_command() == AMPS.Message.Command.GroupEnd: print("--- End SOW Results ---") ``` In the example above, the `Client.sow()` convenience method is used to execute a SOW query on the `orders` topic, for all entries that have a symbol of `'ROL'`. As the query executes, messages containing the data of matching entries have a `Command` of value `"sow"` (provided as a constant value, `AMPS.Message.Command.SOW`), so as those arrive, we write them to the console. As with `subscribe`, the `sow` command also provides an asynchronous mode, where you provide a message handler. ```python showLineNumbers def on_message_handler(message): if message.get_command() == AMPS.Message.Command.GroupBegin: print("--- Begin SOW Results ---") if message.get_command() == AMPS.Message.Command.SOW: print(message.get_data()) if message.get_command() == AMPS.Message.Command.GroupEnd: print("--- End SOW Results ---") def execute_sow_query(client): return client.execute_async( \ AMPS.Command("sow") \ .set_topic("orders") \ .set_filter("/symbol='ROL'"),\ on_message_handler) execute_sow_query(client) ``` In the example above, the `execute_sow_query()` function invokes `Client.execute_async()` to initiate a SOW query on the `orders` topic, for all entries that have a symbol of `'ROL'`. As the query executes, the `on_message_handler()` function is invoked for each matching entry in the topic. Messages containing the data of matching entries have a `Command` of value `sow`, so as those arrive, we write them to the console. --- # Subscriptions Messages published to a topic on an AMPS server are available to other clients via a subscription. Before messages can be received, a client must subscribe to one or more topics on the AMPS server so that the server will begin sending messages to the client. The server will continue sending messages to the client until the client unsubscribes, or the client disconnects. With content filtering, the AMPS server will limit the messages sent to only those messages that match a client-supplied filter. In this chapter, you will learn how to subscribe, unsubscribe, and supply filters for messages using the AMPS Python client. ## Subscribing The AMPS client makes it simple to subscribe to a topic. You call `client.subscribe()` with the topic to subscribe to and the parameters for the subscription. The client submits the subscription to AMPS and returns a `MessageStream` that you can iterate over to receive the messages from the subscription. Below is a short example: ```python showLineNumbers from AMPS import Client # Here, we create a Client object and connect to an AMPS server. client = Client("test") client.connect("tcp://127.0.0.1:9007/amps/json") client.logon() # Here we subscribe to the topic messages. We do not provide a filter, so AMPS # does not content-filter the topic. Although we don't use the object explicitly # here, the subscribe method returns a MessageStream object that we iterate over. # If, at any time, we no longer need to subscribe, we can break out of the loop. # When there are no more active references to the MessageStream, the client sends # an unsubscribe command to AMPS and stops receiving messages. for message in client.subscribe("messages"): # Within the loop, we process the message. In this case, we simply print the # contents of the message. print(message.get_data()) ``` AMPS creates a background thread that receives messages and copies them into the `MessageStream` that the `for` loop iterates over. This means that the client application as a whole can continue to receive messages while you are doing processing work. The simple method described above is provided for convenience. The AMPS Python client provides convenience methods for the most common form of AMPS commands. The client also provides an interface that allows you to have precise control over the command. Using that interface, the example above becomes: ```python showLineNumbers from AMPS import Client from AMPS import Command # Here, we create a Client object and connect to an AMPS server. client = Client("test") client.connect("tcp://127.0.0.1:9007/amps/json") client.logon() # Here, we create a Command object for the subscribe command, specifying the topic # messages. We do not provide a filter, so AMPS does not content-filter the # subscription. Although we don't use the object explicitly here, the execute # method returns a MessageStream object that we iterate over. If, at any time, we # no longer need to subscribe, we can break out of the loop. When we break out of # the loop, there are no more references to the MessageStream and the AMPS client # sends an unsubscribe message to AMPS. for message in client.execute(Command("subscribe").set_topic("messages")): # Within the body of the loop, we can process the message as we need to. In this # case, we simply print the contents of the message. print(message.get_data()) ``` The `Command` interface allows you to precisely customize the commands you send to AMPS. For flexibility and ease of maintenance, 60East recommends using the `Command` interface (rather than a named method) for any command that will receive messages from AMPS. For publishing messages, there can be a slight performance advantage to using the named commands where possible. --- # Synchronous Message Processing As mentioned [earlier](subscriptions), one way for an application to receive messages is to have the AMPS Python client return a `MessageStream` object that can be used to iterate over the results of the command. The `MessageStream` object makes copies of the incoming messages. When there is no message available, the `MessageStream` will block. A `MessageStream` will only remain active while the client that produced it is connected. If the client disconnects, the `MessageStream` will continue to provide any messages that have not yet been consumed, then throw an exception. The advantages of using a `MessageStream` that it provides a simple processing model, that receiving messages from a `MessageStream` does not block the client receive thread (see [Understanding Threading](understanding-threading) ) and that a copy of the message is automatically made for the application. In return for these advantages, a `MessageStream` has higher overhead than [Asynchronous Message Processing](async-message-processing), it will not be resumed if the client disconnects, and, by default, it will use as much memory as necessary to hold messages coming from the AMPS server. --- # Understanding Threading The first time a command causes an instance of the `Client` or `HAClient` to connect to AMPS (typically, the `logon()` command), the client creates a thread that runs in the background. This background thread is responsible for processing incoming messages from AMPS, which includes both messages that contain data and acknowledgments from the server. When you call a command on the AMPS client, the command typically waits for an acknowledgment from the server and then returns. (The exception to this is `publish`. For performance, the `publish` command does not wait for an acknowledgment from the server before returning.) In the simple case, using synchronous message processing, the client provides an internal handler function that populates the `MessageStream`. The client receive thread calls the internal handler function, which makes a deep copy of the incoming message and adds it to the `MessageStream`. The `MessageStream` is used on the calling thread, so operations on the `MessageStream` do not block the client receive thread. When using asynchronous message processing, AMPS calls the handler function from the client receive thread. Message handlers provided for *asynchronous* message processing must be aware of the following considerations: - The client creates one client receive thread at a time, and the lifetime of the thread lasts for the lifetime of the connection to the AMPS server. A message handler that is only provided to a single client will only be called from a single thread at a time. If your message handler will be used by multiple clients, then multiple threads will call your message handler. In this case, you should take care to protect any state that will be shared between threads. Notice that if the client connection fails (or is closed), and the client reconnects, the client will create a different thread for the new connection. - For maximum performance, do as little work in the message handler as possible. For example, if you use the contents of the message to update an external database, a message handler that adds the relevant data to an update queue, that is processed by a different thread, will typically perform better than a message handler that does this update during the message handling. - While your message handler is running, the thread that calls your message handler is no longer receiving messages. This makes it easier to write a message handler because you know that no other messages are arriving from the same subscription. However, this also means that you cannot use the same client that called the message handler to send commands to AMPS. Acknowledgments from AMPS cannot be processed and your application will deadlock waiting for the acknowledgment. Instead, enqueue the command in a work queue to be processed by a separate thread or use a different client object to submit the commands. - The AMPS client resets and reuses the `Message` provided to this function between calls. This improves performance in the client, but means that if your handler function needs to preserve information contained within the message, you must copy the information (either by making a copy of the entire message or copying the required fields) rather than just saving the message object. Otherwise, the AMPS client cannot guarantee the state of the object or the contents of the object when your program goes to use it. Likewise, a message handler should not modify the `Message` -- this will result in modifying the message provided to other handlers (including handlers internal to the AMPS client). --- # Unexpected Messages The AMPS Python client handles most incoming messages and takes appropriate action. Some messages are unexpected or occur only in very rare circumstances. The AMPS Python client provides a way for clients to process these messages. Rather than providing handlers for all of these unusual events, AMPS provides a single handler function for messages that can't be handled during normal processing. Your application registers this handler by setting the `last_chance_message_handler` for the client. This handler is called when the client receives a message that can't be processed by any other handler. This is a rare event, and typically indicates an unexpected condition. For example, if a client publishes a message that AMPS cannot parse, AMPS returns a failure acknowledgment. This is an unexpected event, so AMPS does not include an explicit handler for this event, and failure acknowledgments are received in the method registered as the `last_chance_message_handler`. Your application is responsible for taking any corrective action needed. For example, if a message publication fails, your application can decide to republish the message, publish a compensating message, log the error, stop publication altogether, or any other action that is appropriate. --- # Unhandled Exceptions When using the asynchronous interface, exceptions can occur that are not thrown to the user. For example, when an exception occurs in the process of reading subscription data from the AMPS server, the exception occurs on a thread inside of the AMPS Python client. Consider the following example using the asynchronous interface: ```python showLineNumbers class MyApp: def on_message_handler(self, message): print(message.get_data()) def wait_to_be_poked(self, client): client.subscribe( self.on_message_handler, "pokes", f"/Pokee LIKE '{getpass.getuser()}'", timeout=5000) input("Press enter to exit") ``` In this example, we set up a subscription to wait for messages on the pokes topic, whose Pokee tag begins with our user name. When messages arrive, we print a message out to the console, but otherwise our application waits for a key to be pressed. Inside of the AMPS client, the client creates a new thread of execution that reads data from the server, and invokes message handlers and disconnect handlers when those events occur. When exceptions occur inside this thread, however, there is no caller for them to be thrown to and by default they are ignored. In applications that use the asynchronous interface, and where it is important to deal with every issue that occurs in using AMPS, you can set an `ExceptionHandler` via `Client.set_exception_listener()` that receives these otherwise unhandled exceptions. Making the modifications shown in the example below, to our previous example, will allow those exceptions to be caught and handled. In this case we are simply printing those caught exceptions out to the console. :::tip In some cases, the AMPS Python client may wrap exceptions of unknown type into an `AMPSException`. Your application should always include an except block for `AMPSException`. ::: If your application will attempt to recover from an exception thrown on the background processing thread, your application should set a flag and attempt recovery on a *different* thread than the thread that called the exception listener. :::info At the point that the AMPS client calls the exception listener, it has handled the exception. Your exception listener must not rethrow the exception (or wrap the exception and throw a different exception type). ::: ```python showLineNumbers class MyApp: def on_exception(self, e): print(f"Exception occurred: {str(e)}") def on_message_handler(self, message): print(message.get_data()) def wait_to_be_poked(self, client): client.set_exception_listener(self.on_exception) # Use the advanced interface to be able to # accept input while processing messages. client.subscribe( self.on_message_handler, "pokes", f"/Pokee LIKE '{getpass.getuser()}'", timeout=5000) input("Press enter to exit") ``` In this example we have added a call to `client.set_exception_listener()`, registering a simple function that writes the text of the exception out to the console. If exceptions are thrown in the message handler, those exceptions are written to the console. AMPS records the stack trace and provides it to the exception handler, if the provided method includes a parameter for the stack trace. The sample below demonstrates one way to do this. (For sample purposes, the message handler always throws an exception.) ```python showLineNumbers import AMPS import time import traceback def handler(message): print (message) raise RuntimeError("in my handler") def exception_listener(exception, tb): print ("EXCEPTION RECEIVED", exception) if tb is not None: traceback.print_tb(tb) client = AMPS.Client("client") client.set_exception_listener(exception_listener) client.connect("tcp://localhost:9007/amps/json") client.logon() client.subscribe(handler, "topic") client.publish("topic", "data") time.sleep(1) client.close() ``` --- # Ending Subscriptions The AMPS server continues a subscription until the client explicitly ends the subscription (that is, *unsubscribes*) or until the connection to the client is closed. With a `MessageStream`, AMPS automatically unsubscribes to the topic when there are no more references to the `MessageStream`. You can also call the `close()` method on the `MessageStream` object to remove the subscription. With asynchronous message processing, when a subscription is successfully made, messages will begin flowing to the message handler function and the `subscribe()` or `execute_async()` call returns a string that serves as the identifier for this subscription. A `Client` can have any number of active subscriptions, and this subscription ID is how AMPS designates messages intended for this subscription. To unsubscribe, we simply call `unsubscribe` with the subscription identifier, as shown below: ```python showLineNumbers client = Client("exampleClient") # Register an asynchronous subscription sub_id = client.execute_async( Command("subscribe").set_topic("messages"), on_message_printer ) ... # when the program is done with the subscription, unsubscribe client.unsubscribe(sub_id) ``` In this example we use the `execute_async()` method to create a subscription to the `messages` topic. When our application is done listening to this topic, it unsubscribes (on the last line) by passing in the subscription identifier returned by the `subscribe` command. After the subscription is removed, no more messages will flow into our `on_message_printer` function. When an application calls `unsubscribe()`, the client sends an explicit `unsubscribe` command to AMPS. The AMPS server removes that subscription from the set of subscriptions for the client, and stops sending messages for that subscription. On the client side, the client unregisters the subscription so that the `MessageStream` or message handler for that subscription will no longer receive messages for that subscription. Notice that calling `unsubscribe` does not destroy messages that the server has already sent to the client. If there are messages on the way to the client for this subscription, the AMPS client must consume those messages. If a `last_chance_message_handler` is registered, the handler may receive the messages. Otherwise, they will be discarded since no message handler matches the subscription ID on the message. --- # Utility Classes The AMPS Python client includes a set of utilities and helper classes to make working with AMPS easier. ## Composite Message Types The client provides a pair of classes for creating and parsing composite message types: - `CompositeMessageBuilder` allows you to assemble the parts of a composite message and then serialize them in a format suitable for AMPS. - `CompositeMessageParser` extracts the individual parts of a composite message type. For more information regarding composite message types, refer to the [Message Types](/docs/amps-user-guide/message-types) chapter in the *AMPS User Guide*. ### Building Composite Messages To build a composite message, create an instance of `CompositeMessageBuilder`, and populate the parts. The `CompositeMessageBuilder` copies the parts provided, in order, to the underlying message. The builder simply writes to an internal buffer with the appropriate formatting, and does not allow you to update or change the individual parts of a message once they've been added to the builder. The snippet below shows how to build a composite message that includes a JSON part, constructed as a string, and a binary part consisting of the bytes from an `array.array` that contains doubles. ```python showLineNumbers json = '{"data":"sample"}' data = array.array('d') # populate data ... # Create the payload for the composite message builder = AMPS.CompositeMessageBuilder() # Construct the composite builder.append(json) builder.append(data.tostring()) # Send the message client.publish("messages", builder.get_data()) ``` ### Parsing Composite Messages To parse a composite message, create an instance of `CompositeMessageParser`, then use the `parse()` method to parse the message provided by the AMPS client. The `CompositeMessageParser` gives you access to each part of the message as a sequence of bytes. For example, the following snippet parses and prints messages that contain a JSON part and a binary part that contains an array of doubles. ```python showLineNumbers parts = parser.parse(message) json = parser.get_part(0) data = array.array('d') data.fromstring(parser.get_part(1)) print(f"Received message with {parts} parts.") print(json) datastring = "" for d in data: datastring += f"{d}" print(datastring) ``` Notice that the receiving application is written with explicit knowledge of the structure and content of the composite message type. ## NVFIX Messages The client provides a pair of classes for creating and parsing NVFIX messages: - `NVFIXBuilder` allows you to assemble an NVFIX message and then serialize it in a format suitable for AMPS. - `NVFIXShredder` extracts the individual fields of an NVFIX message type. ### Building NVFIX Messages To build an NVFIX message, create an instance of `NVFIXBuilder`, then add the fields of the message using `append()`. `NVFIXBuilder` copies the fields provided, in order, to the underlying message. The builder simply writes to an internal buffer with the appropriate formatting, and does not allow you to update or change the individual fields of a message once they've been added to the builder. The snippet below shows how to build an NVFIX message and publish it to the AMPS client. ```python showLineNumbers # create the payload for the NVFIX Message builder = AMPS.NVFIXBuilder() # construct the NVFIX message builder.append("sample","data") builder.append("even", "more data") ... # display the data print(builder.get_string()) # publish the message client.publish("messages-sow", builder.get_string()) ``` ### Parsing NVFIX Messages To parse an NVFIX message, create an instance of `NVFIXShredder`, then use the `to_map()` method to parse the message provided by the AMPS client. The `NVFIXShredder` gives you access to each field of the message in a map. The snippet below shows how to parse and print an NVFIX message. ```python showLineNumbers # create the shredder for the message and subscribe to the topic shredder = AMPS.NVFIXShredder() message = client.subscribe(topic="messages-sow", timeout=5000) # shred the message to a map message_map = shredder.to_map(message.next().get_data()) # display the values of the message for key in message_map: print(key + " " + message_map[key]) ``` ## FIX Messages The client provides a pair of classes for creating and parsing FIX messages: - `FIXBuilder` allows you to assemble a FIX message and then serialize it in a format suitable for AMPS. - `FIXShredder` extracts the individual fields of a FIX message. ### Building FIX Messages To build a FIX message, create an instance of `FIXBuilder`, then add the fields of the message using `append()`. `FIXBuilder` copies the fields provided, in order, to the underlying message. The builder simply writes to an internal buffer with the appropriate formatting, and does not allow you to update or change the individual fields of a message once they've been added to the builder. The snippet below shows how to build a FIX message and publish it to the AMPS client. ```python showLineNumbers # create the payload for the FIX Message builder = AMPS.FIXBuilder() # construct the FIX message builder.append(0,"data") builder.append(1, "more data") ... # display the data print(builder.get_string()) # publish the message client.publish("messages-sow", builder.get_string()) ``` ### Parsing FIX Messages To parse a FIX message, create an instance of `FIXShredder`, then use the `to_map()` method to parse the message provided by the AMPS client. The `FIXShredder` gives you access to each field of the message in a map. The snippet below shows how to parse and print a FIX message. ```python showLineNumbers # create the shredder for the message and subscribe to the topic shredder = AMPS.FIXShredder() message = client.subscribe(topic="messages-sow", timeout=5000) # shred the message to a map message_map = shredder.to_map(message.next().get_data()) # display the values of the message for key in message_map: print(str(key) + " " + message_map[key]) ``` --- # Welcome & the AMPS Philosophy Welcome to our Blog! We are 60East Technologies and we’re the creators of AMPS, the world’s fastest real-time streaming analytic database. A product like AMPS isn’t born overnight, but rather grown over years of development and discovery. In this first blog entry, we’d like to introduce you to our core beliefs and philosophy for building high-quality software that’s not only resilient to rapidly changing technology but also benefits from it. Nurture Your Code ----------------- Software wants to run fast. As developers, it’s our job to show it how. At 60East we take great care making sure our code is properly nurtured to be the very best it can be. The 4 things we’re continually teaching our products: 1. **Reduce complexity**: Developers are reluctant to change complex code. Reduce complexity and you reduce reluctance. With proper controls, code should only get better by changing it, so making developers feel comfortable changing the code is critically important. Given the same performance and functionality, the simpler code always wins. 2. **Increase Performance**: We’re always experimenting with ways to tease out more performance, whether it be a new type of lock-free data structure or trying to get a O(log n) algorithm to run in constant time. 3. **Simplify maintenance and support**: We know customers adopt us for the performance, but they stick with us because of our commitment to quality and their success. 4. **Be Fit!**: We don't like fat, lazy code. That's why we're always kicking our code off its idle-state loving butt and sending it through the obstacle course. We don't test to show software is working. We test to show that it's BROKEN! Run it until the wheels fall off. All software has a breaking point and we always want to know where ours is. The “Cutting Edge” isn’t a Band-Aid ----------------------------------- It’s tempting to use technology on the “cutting edge” as a Band-Aid for the software of yore. After all, it’s easy to get a 10-20% performance bump by just buying new hardware. Unfortunately, not all software is capable of taking advantage of the capacity offered by the newest hardware. We know that the next generation of hardware could yield us a modest gain for very little work. However, we know from experience that if we spend a bit of effort we might find performance that is orders of magnitude better than the effortless, “Just Upgrade It” path. That’s why you’ll see us “unwrapping” a new vendor spec with the enthusiasm of a small child opening their first gift. Behind that cheesy grin, we’re thinking: Can we use that new SSE instruction? How could we use that new futex() flag? Which calls bypass the OS kernel? Expect and Embrace Technological Change --------------------------------------- When we first started building AMPS, dual-core CPUs were all the rage and enterprises were spending a fortune on 15K RPM hard disks. Now we have dual-core cell phones in our pockets and flash storage technology that makes spinning storage seem silly. Knowing that the next best thing is just around the corner, we do our best to write elastic code that takes full advantage of increasing hardware capacity. Sometimes we nail it and end up with an algorithm that works well for generations of technology change. Occasionally we get it wrong. Either we didn’t plan for such a rapid improvement in technology or we need to refactor to make full use of new vendor APIs. Regardless of how often we’re correct, we always write code in a modular way so that we can be nimble and “upgrade” the code when the next greatest thing comes out. Develop Meaningful Relationships -------------------------------- Developing quality relationships with our customers and vendors is just as important as developing the best code. We think of our customers as partners and will do everything we can to help them succeed in developing high-quality solutions for their own customers. We also have great relationships with our vendors because we have genuine enthusiasm for their technology and our plans for using it. We work closely with vendors to experiment, evaluate, test and report on their, sometimes “bleeding edge”, technologies. We get a lot from these relationships as well, because we use these technologies as a “crystal ball” into the future. This helps us build software that fully exploits the capacity offered today as well as being capable of unlocking the gains of the future. Remain Positive --------------- Even though the future has always been riddled with vaporware and products that don't deliver on their performance promises, we value the lessons of what doesn’t work just as highly as what does. Sometimes we’ll drool over a product specification for months and when it’s finally released it doesn’t meet our expectations. This can be frustrating, but we know that something better is just around the corner and we’re going to be ready when it gets here. We’re always working on perfecting our craft and exercising the utmost discipline in enhancing our products, whether it’s documentation, testing, or writing lock-free data structures. Whether it’s ripping through online blogs to learn how others use “fuzzing” to test their products or reading up on the latest instructions supported by a CPU, we’re doing all we can to be responsible stewards of our code. We hope this introduction gives you some insight into our methodologies and philosophy. We’re looking forward to sharing lessons we’ve learned and going into more detail on the methodologies listed above in the coming months. If you’re interested in getting future updates as soon as they’re published, please subscribe to our mailing list and/or RSS feed. Until next time, happy coding! --- # Using AMPS in XAML Applications Introduction ------------ AMPS provides an ideal environment for writing Windows applications that display real-time data. The combination of content filtering and state-of-the-world ("SOW") caching in AMPS makes it perfect as a "view server" that delivers updates to desktop applications. The AMPS Client for C# provides built-in support for XAML applications in the AMPS.Xaml library. In this article, I walk through the steps involved in configuring AMPS as a view server, and how to build a sample app, OrdView, with the AMPS.Xaml components. Sample code for OrdView is available for [download here](/downloads/ordview.zip). OrdView: An Order-Display Application ------------------------------------- Our application, OrdView, will allow users to view order information from a trading system. After launching OrdView, users specify the orders they are interested in watching via search criteria, and OrdView queries AMPS and displays the results in a grid. OrdView updates the grid in real-time to reflect the current state of the system, because AMPS continues to send information about new and changed orders. At any time, users can enter new search criteria to observe a different set of orders. Configuring AMPS ---------------- Before building OrdView, let's review the features in AMPS we'll need. AMPS is a topic based publish/subcribe server with unique features that make it ideal for building this type of application. *Content filtering* enables OrdView to realize high performance over a large trading system. While the trading system processes millions of trades, no one user of OrdView wants to see the entire set. AMPS allows filters to be applied at the server for any subscription. Only messages matching the filter are sent over the network to the subscriber. Any message field can be used to filter, without any additional AMPS configuration. No special configuration is required to enable content filtering; our application will simply construct and supply an appropriate filter when it places the subscription. *State-of-the-world* ("SOW") allows AMPS to store the last values seen in a given topic. OrdView displays all of the current orders once a user specifies search criteria, and uses AMPS to retrieve the current orders in the system matching the criteria by performing a *sow_and_subscribe* query. This special kind of query returns all of the results stored by AMPS, and causes AMPS to begin sending updates to this result set as soon as the result set has been sent. OrdView displays the results sent, and begins applying updates to the grid immediately thereafter. AMPS guarantees that the initial results and updates are synchronized: no updates can "slip through the crack" between the initial result set and the beginning of the update stream. Configuring SOW requires specification of what fields from messages should be used as the primary key for the SOW store. This metadata is specified in the AMPS server configuration file. For OrdView, we will use the following metadata configuration: ```xml showLineNumbers sow/%n.sow ORDERS /Key /Symbol nvfix ``` For more information on configuring SOW topics, review "State of the World (SOW)" in the [User Guide](/docs/amps-user-guide). OOF Processing sends out-of-focus ("OOF") messages to OrdView as orders are removed or change in such a way as to no longer match the user's criteria. For example, if the user filters to only show orders whose "state" tag is "PENDING", AMPS will send an OOF message to OrdView as soon as an order is removed, or its state changes from "PENDING" to any other value. OrdView uses this opportunity to delete records from the grid. Prerequisites ------------- In addition to having AMPS installed, you'll need a computer running Microsoft Windows, with Microsoft Visual Studio 2010 or 2012 installed. You'll also need to install the AMPS C# Client, available for download at [/documentation/client-apis/c-sharp/](/clients/amps-client-csharp). The Client includes binaries, documentation, and reference materials for both the AMPS.Client and AMPS.Client.Xaml assemblies. Creating OrdView ---------------- To get started, we'll create a new, empty C# WPF application in Microsoft Visual Studio. Click "New Project..." and then choose "WPF Application" in the "Visual C#" templates: ![New Project](/img/blog/newproject_498x330.jpg) Once we've created the project, Visual Studio displays the XAML editor for the generated main window of our application. Since we'll be using the AMPS C# Client to work with AMPS, choose "Add Reference..." from the "Project" menu, navigate to the bin directory of the AMPS C# Client, and select the AMPS.Client.dll and AMPS.Xaml.dll assemblies: ![Add Reference](/img/blog/addreference.png) Create AMPS XAML Objects ------------------------ XAML is an XML dialect that allows object to be created and manipulated declaratively. Windows applications use XAML extensively to define user interfaces and create bindings between UI components and data. Most data models are not created with direct binding in mind, so an extra layer between the true data model and the bound UI controls is common, and is often referred to as the "view model" of the application. The AMPS C# Xaml library acts as a view model for AMPS topics, bridging the gap between the AMPS C# Client and data bound controls. You can use it directly in your applications, or integrate it into a view model scheme. In this application, we use it directly. Once the application is created and references to the AMPS assemblies are in place, it's time to begin working on our user interface. In OrdView, we'll display our AMPS topic in the main window of our application, so we add references to AMPS objects into the main window's XAML. First, change the namespace declarations on the Window element to include AMPS. In the XAML view, add attributes to the Window element to make it look like this: ```xml showLineNumbers ``` The addition of the "amps" namespace prefix allows us to refer to AMPS.Client.Xaml objects from our window's XAML. There are a number of approaches for creating resources, both in XAML directly, or by adding properties to our window and referring to them later. In OrdView, we'll add our objects to the main window's resources using the Resources element. Inside the Window element, create a Window.Resources element, substituting your server's parameters for the ones given here: ```xml showLineNumbers ``` This code creates two objects in our window's resource dictionary: an AMPS.Client.Xaml.Server object with the key "MyServer", and an AMPS.Client.Xaml.Subscription object with the key "MySubscription". The Server object specifies the parameters to connect to an AMPS server. In this example, we connect to a server running on port 9005 of the host "ampServer.local" and use the NVFIX message type. These are hardcoded here for simplicity; you could also construct and set these properties programmatically. The Subscription object specifies the parameters for a subscription to an AMPS SOW topic. The Server attribute specifies an AMPS.Client.Xaml.Server to use, and here we've used XAML data binding to bind this attribute to the Server we created on the previous line. The Subscription subscribes to the ORDERS topic with a filter of `/Key=0`. Later, we'll present the user with the ability to change the filter and update the subscription. Binding AMPS Subscriptions to our UI ------------------------------------ When our OrdView application starts, it will now establish a connection to AMPS and place a subscription to the ORDERS topic. The subscription is a SOW subscription, and will utilize content filtering and out-of-focus ("OOF") messages to keep itself up-to-date. Next, we use data binding and the Subscription object to bind our subscription to a data grid. First, let's work on the layout of our application. To keep things simple, we'll remove the Grid that is automatically generated for our window's layout, and replace it with a DockPanel. We set the DockPanel's DataContext to the subscription, as we'll add multiple bindings to the subscription before OrdView is complete. Inside the DockPanel, we create a DataGrid bound to our Subscription's data, with the two columns we're most interested in. Our Window's content looks like this: ```xml showLineNumbers ``` Publish Test Data ----------------- Before running OrdView, we should find or create test data to publish into our topic. Sample data for this example can be downloaded [here](/downloads/ordview.zip). Use the AMPS command-line tool, 'spark', for example: ```bash ./spark publish -file testdata.nvfix -server localhost:9005 -topic ORDERS ``` Run OrdView ----------- That's it! Run OrdView, and you should see a window displaying the data in the ORDERS topic: ![First Run](/img/blog/firstrun_500x333.jpg) Our OrdView application not only displays new, changed and deleted data in real time, but the grid supports operations users expect, such as sorting, column reordering, and copy-and-paste. #### A Word About Columns Unlike a traditional database, AMPS allows each message to contain different or new columns. You don't configure AMPS with message schemas, except for defining one or more Key fields. To determine the set of columns, the AMPS.Client.Xaml.Subscription examines the fields specified on each message, and adds columns whenever a new field is seen in a message. Earlier we explicitly named the columns we want to show, but we can also utilize DataGrid's AutoGenerateColumns attribute to specify that a new grid column will be displayed any time a new field is observed in a message. These columns will be automatically named based on the field title: ```xml ``` In addition to setting AutoGenerateColumns to True, we need to create an event handler so that, as new columns are added to the underlying DataTable, the grid is refreshed automatically. First, navigate to the Window's events in the property grid. Create a new event handler for the "Loaded" event on the window by double-clicking. Add the following code to the event handler: ```csharp showLineNumbers private void Window_Loaded(object sender, RoutedEventArgs e) { ((System.Data.DataView)_grid.ItemsSource).Table.Columns.CollectionChanged += new System.ComponentModel.CollectionChangeEventHandler(Columns_CollectionChanged) } ``` And add the following member function to the Window (referenced by our event handler, above): ```csharp showLineNumbers void Columns_CollectionChanged(object sender, System.ComponentModel.CollectionChangeEventArgs e) { _grid.AutoGenerateColumns = false; _grid.AutoGenerateColumns = true; } ``` Content Filtering ----------------- Our example hardcodes a filter to start with, but we'd like to give the user the ability to change the filter. In OrdView, we accomplish this by binding a TextBox to the Filter property on our subscription, allowing users to type any filter string they choose. Keeping the UI simple, we bind a single text box to the filter: once the user changes it and clicks on another control, the Subscription's filter is automatically changed. Here's the new XAML for the content of the Window: ```xml showLineNumbers ``` WPF TextBox default behavior updates its source when it loses focus. In our app, we've changed the behavior to update the Filter property with every keystroke, but for a more sophisticated UI, you could easily add a button that "applies" changes to the Filter. Run the program again, and you should see a small textbox above the grid where you can manipulate the filter using AMPS filter expressions. Remember that as you make these changes, the filter is updated on the server, so that your client receives no messages except those that match the filter. Real-Time Status ---------------- Unlike most data sources, AMPS maintains a live connection to "push" new data at OrdView. Sometimes errors occur, such as when a disconnect occurs or an invalid filter is specified, and a full-featured OrdView ought to show the status of the connection in real-time. To facilitate this, each AMPS Subscription contains a read-only Error property that may be bound to a control on the UI, so that users can observe the current health of the connection. We'll use a StatusBar at the bottom of the window for this purpose: ```xml showLineNumbers ``` This StatusBar is empty most of the time, but when errors occur, either on the connection or as a result of an invalid filter, it indicated the problem: ![Bad filter](/img/blog/badfilter.png) Conclusion ---------- Here's what OrdView looks like now that we're finished: ![ItsAlive](/img/blog/itsalive_500x333.jpg) In this article we've taken a look at how to use AMPS and the AMPS.Client.Xaml classes to create an efficient application that views real-time data. Underneath the covers, we use AMPS SOW, OOF, and Content Filtering features to make sure the application only displays the data we're interested in. In a future post, we'll take a look under the covers of AMPS.Client.Xaml to see how it bridges the world of AMPS and Windows. --- # Pushing the Limits of the Windows DataGrid with AMPS In some applications, it's critical to query and use huge amounts of data on the client. What if you want to visually display and work with a million rows of data? Using AMPS and the WPF DataGrid, you can build applications that do just that: filter and display over a million rows in a few seconds, and keep that display up to date with thousands of changes a second. In this blog post, we'll show you how to do it. Prerequisites ------------- Before following the steps included in this document, ensure you have the following prerequisites installed and running: * AMPS 3.3.1.0 (or greater), available from [here](/evaluate). * An appropriate x86_64 server to run AMPS. * The AMPS C# client (version 3.2.0.0 or greater), available from [here](/clients/amps-client-csharp). * Microsoft Visual Studio 2010 * Python (2.x) Note: In this post, we've used an AMPS server with more than 100GB of memory; if you have less, scale back the initial size from 200 million records to 3 million records. Sample Files ------------ To get started with this app, you can download the example project [here](/downloads/ordview.zip), which contains a zip file of an AMPS configuration, source code, and a utility for loading and updating test data in your AMPS server. ### Configure AMPS An example AMPS configuration file is provided in “amps-config” that matches the expectations of the rest of this example. Modify this configuration as necessary to represent your environment and start an ampServer instance once configured. ### Run the Sample Workload A python client to publish and update ORDERS data is provided in the "testdata" directory, "sample-workload.py". For best performance in initial load, run sample-workload.py on the same computer as your AMPS instance. To publish 200 million records to the instance running on localhost: ```sh ./sample-workload.py init localhost 200 ``` After publishing the initial dataset, use the 'update' command to begin publishing updates to the base data. To update the 200 million records published before at a rate of 1,000 records per second: ```sh ./sample-workload.py update localhost 200 1000 ``` ### Build and Run the OrdView Application Open the `OrdView.sln` solution located in this package. Add references from OrdView to the AMPS.Client.dll and AMPS.Client.XAML.dll assemblies located in the “bin” directory where the AMPS C# client is installed. Build and run the application, and you will see a window like the following: ![Ordview Shot1](/img/blog/ordviewshot1_500x363.jpg) In this window, the top bar contains the AMPS hostname and a filter expression, and the space below is dedicated to the result grid. The status bars below indicates the status of the subscription: * Time to first: the time elapsed between the beginning of the query and the arrival of the first result row * Time to last: the time elapsed between the beginning of the query and the arrival of the last result row. * Rows in view: the number of rows displayed in the grid and the number of rows in the underlying ORDERS topic, once queried. * Message Rate: the incoming message rate over the last second. * Peak Rate: the peak message rate achieved during the last query. Change the Hostname to the name of your AMPS server, and type a filter expression that yields results. Press the Enter key (or click the Update button), and after a few moments, you will see a result: ![Ordview Shot2](/img/blog/ordviewshot2_499x363.jpg) This grid is “live”, and able to reflect thousands of updates every second to the underlying AMPS topic, even on top of a grid containing more than a million records. ### Reviewing the Code The code for this application makes extensive use of the data model classes provided in the AMPS.Client.XAML assembly of the AMPS C# client. #### MainWindow.xaml The XAML for this application is quite simple: ```xml showLineNumbers localhost Rows in view: of Message Rate: Peak Rate: Time to first: Time to last: ``` In this XAML, after declaring namespace references for the amps namespace, we create an amps:Subscription that connects and subscribe to an AMPS topic. The subscription is initialized with a filter, 1 = 0, that returns no rows, so that the data grid begins empty. Inside the code for the application, the Subscription is bound to a Server when Update is pressed. Our window uses a Grid layout to lay out the status bars, hostname and filter text controls, update button, and data grid. Each of these controls are bound to properties of the subscription: the filter text box is directly bound to the Filter property, the standard WPF DataGrid to the Data property, and StatusBar items are bound to the various properties of the subscription, to indicate the state of the subscription to the user. ### MainWindow.xaml.cs ```csharp showLineNumbers using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Windows; using System.Windows.Controls; using System.Windows.Data; using System.Windows.Documents; using System.Windows.Input; using System.Windows.Media; using System.Windows.Media.Imaging; using System.Windows.Navigation; using System.Windows.Shapes; using AMPS.Client.Xaml; using System.Data; using System.Threading; using System.ComponentModel; namespace OrdView { public partial class MainWindow : Window { public MainWindow() { InitializeComponent(); AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException); } void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e) { MessageBox.Show(e.ExceptionObject.ToString(), "Unhandled exception"); } // Since we have a large query to process, we don't // want to run queries without an explicit command private void Update_Click(object sender, RoutedEventArgs e) { _filter.GetBindingExpression(TextBox.TextProperty).UpdateSource(); this.Cursor = Cursors.Wait; var sub = (Subscription)this.Resources["MySubscription"]; if (sub.Server == null || sub.Server.Hostname != _server.Text) { try { ((Subscription)this.Resources["MySubscription"]).Server = new Server { Hostname = _server.Text, Port = 9005, MessageType = "nvfix" }; } catch (Exception ex) { this.Cursor = Cursors.Arrow; MessageBox.Show(ex.Message, "Error", MessageBoxButton.OK, MessageBoxImage.Error); } } this.Cursor = Cursors.Arrow; } } } ``` Most of the logic of this application is implemented declaratively through the XAML, so this part of the application is minimal. In the code here, the handler for the Update button forces an update to the Filter property on the Subscription. Next, the hostname in the text box is compared to the current Server's hostname, and if they are different, a new Server is constructed and bound to the Subscription. Error handling is minimal; the application displays a message box with the content of any Exception thrown by constructing the Server. Setting the Cursor provides a nicer user experience, since connecting/not connecting to a server may take a few moments. ### Summary In this post, we've showed you how you can use AMPS as a large scale view server and build a simple query GUI that's capable of displaying more than a million rows with thousands of updates per second. In our environment (details below), we were able to query 200 million records inside of AMPS and return and display a million records in under 1.5s. If your application requires a similar low-latency, high-scale user interface, try out this demo, and let us know what you find! ### Configuration Details Operating System: * Type: Linux x86_64 * Kernel: 2.6.32-5 Processor: * 2 x Intel Xeon E5-2680 Memory: * 256 GB Storage: * FusionIO ioDrive2 Duo 2.4TB * Driver Ver: 3.2.3 Network: * Intel 1Gb --- # Light-Weight AMPS bridge > Learn how to build an AMPS Bridge that solves different problems than AMPS Replication, and gives you a light-weight, customizable way to move, transform, and enrich messages. ## The AMPS Bridge What if your application needs a light-weight way to move messages between AMPS instances? AMPS Replication already reliably propagates messages to downstream instances. Individual replication links are configured with settings to filter out which topics and messages are replicated. Replication makes use of transaction logging to ensure that no messages are missed during temporary server or network outages, and ensures that messages are delivered as received. Some applications have different requirements, though. For example, you might need to enrich data as it is moved from one instance to another, using reference data in a database or local cache. Or you might simply want to make a live stream of data from one instance available on another, without the logging overhead associated with replication (and without the reliability guarantees). The AMPS Bridge, a new sample application built on the AMPS C++ Client, meets these needs and more. AMPS Bridge uses the AMPS C++ library to establish connections to a set of source and destination instances, and bridges messages from the message source to one or more destinations. Source code is included so you can transform or enrich messages as they move through AMPS Bridge. Since AMPS Bridge is built on our C++ client, it automatically takes care of detecting disconnects and reconnecting to the server when a connection is interrupted. AMPS Bridge allows configuration of the each source/destination pair in a number of ways. First, you can choose which topic or topics to subscribe to, by supplying a topic name or regular expression – the AMPS server takes care of identifying which topics you mean when you supply a regular expression. You can choose any downstream topic you’d like to receive the messages. You can also supply an AMPS filter string to filter the messages that are bridged; that filter is processed on the source AMPS instance, so destination instances only receive relevant messages. Finally, you can opt-in to using bookmark subscriptions and store-and-forward publishing, if your source and destination are enabled with a transaction log. Included here are the source code, a makefile, and a configuration file. You’ll need the AMPS C++ Client, version 3.2 or greater, installed and built on your machine; change the `AMPS_CPP_DIR` value in the makefile to point to where your client is installed. You’ll also need one or more AMPS servers, version 3.3 or greater, installed and running. Try running it as-is by modifying config.csv to point to an amps instance and topic you’d like to bridge, and then just execute: ```bash ./amps_bridge config.csv ``` to start it. If you open the code, you’ll find a few classes of interest: * `amps_bridge` is the “main” class of the application; it manages a map of client connections and kicks off subscriptions. * `bridge_subscription` reads a line from the CSV configuration file supplied, and then subscribes to the specified topic to get things started. Note that when HA is specified in the configuration file, we use `bookmarkSubscribe()` instead of regular `subscribe()`: this is a heavier weight operation on the server and client, but ensures there are no gaps between messages when reconnecting to a server after a disconnect. Also notice that when we use `bookmarkSubscribe()`, there is an additional, required, call to `discard()` in the message handler to indicate that the message has been processed. Finally, the `filter()` method is called by the message handler to create an outgoing message from the incoming one. If you’d like to add more complex transformation or enrichment logic to AMPS Bridge, `filter()` is the place to do it. * `bridge_server_chooser` is an `AMPS::ServerChooser` implementation that produces messages to `stderr` when a disconnect occurs and ten consecutive attempts to reconnect have all failed. This is a simple implementation: you could make this far more intricate, choosing to shut down the program after a period of disconnect, fire off an alert to a monitoring system, etc. Both “HA” and regular subscriptions use the `HAClient`, which provides automatic reconnection and resubscription even when using plain subscriptions. `HAClient` interacts with a `ServerChooser` to indicate failed connections and acquire additional servers for fail-over. In this implementation, we use the `ServerChooser` interface to find out when the `HAClient` loses a connection, but AMPS Bridge only supports one server URI per subscription. To wrap it up, the AMPS Bridge solves different problems than AMPS Replication, and gives you a light-weight, customizable way to move, transform, and enrich messages. Let us know how you use it! ## Download Download the AMPS Bridge source code and makefile here: [amps_bridge.tar.gz](/downloads/amps_bridge.tar.gz) --- # Not Using Content Filtering in Your Messaging Application? You're Doing it WRONG. > How do AMPS applications perform so well? Here we take look into how AMPS filtering creates message selectivity that speeds the performance of end users' application. Naive messaging systems broadcast all messages to subscribers. This style of message delivery can cause resource over-utilization on subscribers who are only interested in a subset of the entire message flow. Even worse, such a message delivery system can quickly bog down or even oversaturate a network. When this happens, the messaging system can no longer scale, at least not without costly upgrades to infrastructure. We've seen first-hand how broadcast publish-subscribe systems can place the heavy burden of message filtering on trading GUI applications. In one case in particular, a subscribing application was consuming 80% of it's CPU - the vast majority of this time was spent discarding unwanted messages. This made for unhappy users that were angry that "their" data was being delayed and the overall GUI experience was "sluggish". ![this is bad CPU utilization](/img/blog/CPU-Utilization.png "this is bad CPU utilization") With AMPS topic and content filtering, throughput requirements of the network can be reduced by delivering to subscribers only the messages they need. Additionally, end-to-end message delivery latency can be improved by reducing the time spent discarding messages that the subscriber has no desire in processing. ![your message pipeline with broadcast](/img/blog/Message-Queue-opaque.png "your message pipeline with broadcast") The same system was reimplemented using AMPS, leveraging topic and content filtering - moving the heavy burden of message filtering from the client application to an AMPS instance. This had the net effect of reducing the client's CPU utilization from 80% to nearly 8%, all while improving the latency profile of message delivery and, most importantly, their over-all satisfaction. The one-two punch of network capacity improvements and reduced client resource consumption enables applications to run more efficiently. If your messaging system expects your end-user application to filter the results that are sent to them, it could mean that a significant burden is being placed on every one of your end-user desktops. It could also mean that your messaging solution is *doing it wrong*. Implementing topic filtering and content filtering in AMPS can improve your application from end-to-end and make certain that you continue to handle messaging *the right way*! Below we'd like to show you how simple it is to use one of the most powerful features of AMPS. PCRE Primer ---------- Before we can dive into how AMPS filtering works, we need to give a quick primer on a select group of Perl Compatible Regular Expression (PCRE) expressions supported by AMPS. If you are unfamiliar with PCRE, fear not, the [AMPS User Guide](/docs/amps-user-guide) has a chapter that covers supported characters and their definitions. To get started quickly, below are a couple of examples to illustrate PCRE and how we can use it in AMPS topic filtering. Regular Expression | Definition --------------------|------------ `"^Hello"` | match all strings that begin with the string 'Hello'. `"World$"` | match all strings that end with the string 'World'. AMPS contains support for these and other regular expression symbols, but for the sake of brevity, we will only discuss these few. Topic filtering in AMPS ----------------------------- With AMPS, a client can use a regular expression to subscribe to topics that match the given pattern. This feature can be used in two different ways: * subscribe to topics without knowing the topic names in advance. With regular expressions, you can subscribe to all topics ( `Topic='.*'` ), or ad-hoc topics that match a pattern, like all topic that start with the letter 'B' ( `Topic=B.*'` ). This allows a subscriber to subscribe to topics with little or no understanding of existing topics in the message stream. * subscribe to topics that only match a very selective pattern. An example of this type of subscription is the search for a specific token in the topic, for example, subscribing to all topics that end with the string "@Foo" ( `Topic='.*@Foo$'` ). Subscription topics are interpreted as regular expressions if they include special regular expression characters. Otherwise, they must be an exact match. Some examples of regular expressions within topics are included in table below. Topic | Behavior ------|---------- `trade` | matches only “trade”. `^client.*` | matches “client”, “clients”, “client001”, etc. `.*trade.*` | matches “NYSEtrades”, “ICEtrade”, etc. AMPS topic filtering allows subscribers to receive messages from the topics they are interested, eliminating the need to filter out unwanted messages. Content filtering in AMPS -------------------------------- We have demonstrated how topic filtering is a powerful feature in AMPS, but now we're going to go a step further and examine the most powerful feature of AMPS - content filtering. Content filters add the query power of syntax similar to SQL-92 to create the filter, which provides a greater level of selectivity than topic filters alone. Content filtering is used in a similar manner to a `WHERE` clause in a SQL `SELECT` statement. It enables filtering on the message body, and uses an XPath syntax to support queries of nested items in message types that support them. Much like the SQL `WHERE` clause, an AMPS content filter supports logical operators (`AND`, `OR`), arithmetic operators (`+`, `-`, `*`, `/`), comparison operators (`<=`, `=`, `BETWEEN`, `IN`), and the conditional operator `IF`. Additionally, the `LIKE` operator is supported to search for a pattern within the data. For the following examples, we're going to use messages that are composed of name-value pairs. Our message stream will consist of the following messages: ```bash Topic=NYSE.Technology;Symbol=MSFT,Price=34 Topic=NYSE.Technology;Symbol=TIBX,Price=14 Topic=NYSE.Technology;Symbol=IBM,Price=180 Topic=NYSE.Utilities;Symbol=XOM,Price=87 Topic=NYSE.Utilities;Symbol=XOM,Price=88 Topic=NYSE.Technology;Symbol=TIBX,Price=15 Topic=NYSE.Technology;Symbol=HP,Price=24 Topic=NYSE.Technology;Symbol=MSFT,Price=31 Topic=NYSE.Technology;Symbol=TIBX,Price=17 Topic=NYSE.Utilities;Symbol=XOM,Price=90 Topic=NYSE.Technology;Symbol=MSFT,Price=34 Topic=NYSE.Technology;Symbol=TIBX,Price=16 Topic=NYSE.Utilities;Symbol=XOM,Price=86 Topic=NYSE.Technology;Symbol=IBM,Price=185 Topic=NYSE.Utilities;Symbol=XOM,Price=87 Topic=NYSE.Utilities;Symbol=XOM,Price=88 ``` For the first example, let's create a client that is only interest in `MSFT` symbols. That client's subscription would look like: ```bash Topic=NYSE.Technology;Filter="/Symbol='MSFT'" ``` This subscription would only return results to the subscriber that contained the 'Symbol' matching 'MSFT' and using the stream from above would only contain the following messages: ```bash Topic=NYSE.Technology;Symbol=MSFT,Price=34 Topic=NYSE.Technology;Symbol=MSFT,Price=31 Topic=NYSE.Technology;Symbol=MSFT,Price=34 ``` In another example, a client is only interested in trades that are trading between $15 and $20 per share. This subscription would look like: ```bash Topic=NYSE.Technology;Filter="/Symbol='TIBX' AND /Price BETWEEN 15 and 20 ``` The range used in the `BETWEEN` operator is inclusive of both operands, meaning the expression `/A BETWEEN 0 AND 100` is equivalent to `/A >= 0 AND /A <= 100` This subscription would return the following messages to the subscriber: ```bash Topic=NYSE.Utilities;Symbol=TIBX,Price=15 Topic=NYSE.Utilities;Symbol=TIBX,Price=17 Topic=NYSE.Utilities;Symbol=TIBX,Price=16 ``` Like topic filtering, using content filtering can reduce the number of superfluous messages sent to a subscriber even further. This has the net result of even further reducing demand on the network and reducing the overall resource requirement on the client. Taking things a step further, topic filtering and content filtering can both be applied to a message stream to provide message filtering so reliable, you'll never need to build another client-side filter again. Using topic and content filtering will make your network happy, your users happy and your developers happy. Conclusion ---------- Implementing topic and content filtering is a simple and highly effective way that AMPS can deliver improved network performance, reduce CPU and memory consuming burden of filtering messages from a broadcast stream, and *reduce latency in your messaging platform*. In the next blog post, we'll look at using topic-only filtering to implement a request / response system using AMPS. We'll also provide an example implementation of a heartbeat monitor and agent that uses this system. Stay tuned! --- # Easy Request/Response Recipe for AMPS The power of AMPS filtering allows your application to receive the messages it wants to process, and none of the messages it doesn't. Client-side filtering is no longer required with AMPS. The ability to use Perl Compatible Regular Expressions (PCRE) to define filters on a topic, a message, or both allows your application to achieve an unrivaled level of precision in message delivery. One example of this precision is an application which implements topic filtering to facilitate request-response messaging between a monitoring system and an agent for a large scale heartbeat system. Heartbeats are a pattern often used to monitor the health of one or more systems or services in a group of systems. In a heartbeat system a monitoring application can send out a heartbeat message (request) to one or more monitoring agents. The monitoring agents then send a message back to the monitoring application (response) indicating the health of the system. If a monitoring agent fails to respond within a specified window of time, the monitoring application will assume that the service is unavailable. In a [previous article](/blog/not-using-content-filtering-in-your-messaging-application-youre-doing-it-wrong), we introduced the selectivity granted by the content filtering and topic filtering in AMPS. It is recommended to read the article to be familiar with the concepts introduced there before diving into this article. In this post, we give a brief overview of how AMPS implements topic filtering, and then build upon that knowledge to describe how to implement a simple request / response heartbeat monitoring system using AMPS topic filtering. We also provide an example in Java, and discuss how to enhance the system to be more (or less) selective by using different topic filters. ## AMPS Topic Filtering With AMPS, a client can use a regular expression to subscribe to topics that match a given pattern. This feature can be used in two different ways: * to subscribe to topics without knowing the topic names in advance. This is known as a "greedy subscription" and can be used to subscribe to many topics simultaneously. For example, the subscription to the topic `foo.*` would match topics "foo", "fool", "food" and "footie". * to subscribe to topics that only match a very selective pattern. The simplest example of this is matching on an exact pattern, for example - we only want to return messages from the topic "Client". For our request-response heartbeat monitoring system, we will be focusing on greedy subscriptions. The messages themselves have no notion of a topic pattern. The topic for a given message is unambiguously specified using a literal string. From the publisher's point of view, it is publishing a message to a topic; it is never publishing to a topic pattern. Subscription topics are interpreted as regular expressions if they include special regular expression characters. Otherwise, they must be an exact match. Some examples of regular expressions within topics are included in table below. Topic | Behavior ------|---------- `trade` | matches only "trade". `^client.*` | matches "client", "clients", "client001", etc. `.*trade.*` | matches "NYSEtrades", "ICEtrade", etc. ## Setting up the Example Heartbeat messages consist of two parts - the request message (requesting a heartbeat) and the response message (letting the requester know that it is still alive). In each of these messages, there is a "source" and a "destination". In the heartbeat request message, the *source* is a heartbeat monitoring application and the *destination* is the agent that is being monitored by the monitoring application. ![Monitor subscribes to `^A<-.*`, while the agent subscribes to `.*->A$`](/img/blog/request_response.png "Sample message flow" ) With this information, we want to conceive of a format for our message topic that can be used to control the flow of messages from the source to the destination, regardless if it is a request or a response message. This lends nicely to a topic format that follows: `` `` `` In this topic format, we can define `MACHINE_1` to be a *source* or a *destination* depending on the `ROUTING_INSTRUCTION` we use. We will use the following convention to define the `ROUTING_INSTRUCTION` * When `->` is the `ROUTING_INSTRUCTION`, this is a *request* message and the topic can be read as: `` `->` `` Where `` is the monitoring application and `` is the heartbeat agent. * When `<-` is the `ROUTING_INSTRUCTION`, this is a *response* message and the topic can be read as: `` `<-` `` Where `` is the heartbeat agent and the `` is the monitoring application. With these tokens our routing message topics would look like `M->A` for the request messages, and `M<-A` for the response messages - where `M` represents the heartbeat monitoring system, and `A` represents the heartbeat agent. We have included an implementation of the `Monitor` and `Agent` classes in Java along with our sample, and will be presenting code samples from that project. The full project, along with instructions in the README file for building and running are available here: [01\_simple\_request\_response.zip](/downloads/01\_simple\_request\_response.zip) With our topic format defined, we're ready to create the subscriptions for our monitoring system and our agent. The monitoring system will subscribe to topic `^M<-.*` - which can be interpreted as "subscribe to all messages where the topic begins with the string `M<-`". Similarly, the agent will subscribe to the topic `->A$` - which can be interpreted as "subscribe to all messages where the topic ends with the string `->A`". Regarding our message format, it's worth pointing out is that we chose to implement our Request/Response system using topic filtering, as opposed to content filtering. This decision is based entirely on the premise we would like to keep the routing rules in the message's topic separate from the message contents. In our particular example, our routing doesn't need to know anything about our message contents when making decisions about delivery. ## An Example By now you should see the beginnings of how such a system would work, but let's walk through the lifespan of a single request-response to see the system in action! The code samples included below come from the included demonstration. These are used to highlight some of the features of the heartbeat request-response system. Note that the Monitor and Agent classes are implemented in their entirety in the included demonstration. Only highlights from those classes are shown here. To begin the lifespan of the heartbeat message, the system is put into motion when the `Monitor` class (M) publishes a heartbeat with a timestamp message to topic `M->A`. ```java showLineNumbers // Monitoring System publish heartbeat message. Date d = new Date(); SimpleDateFormat sdf = new SimpleDateFormat( "MM/dd/yyyy h:mm:ss a"); msg = "Heartbeat! Timestamp: " + sdf.format(d); System.out.println( "**************************\n" + "Monitor - sending message:" + "\n\tmessage: " + msg + "\n" + "**************************\n"); monitorClient.publish("M->A", msg); ``` The `Agent` class (A)- which is subscribed to Topic `->A$` - receives the message, recognizes this is a request topic because of the presence of the `->` token, parses the prefix of the `Topic` field, and then constructs a response message to be returned to M. A publishes the response message to topic `M<-A`. ```java showLineNumbers // Agent subscription. AgentMessageHandler mh = new AgentMessageHandler(); agent.subscribe(mh, "->A$", 10000); ... static class AgentMessageHandler implements MessageHandler{ public void invoke(Message m){ // Parse the incoming message. String topic = m.getTopic(); TopicParse parse = TopicParser.parse(topic); System.out.println( "**************************\n" + "Agent - received message:" + "\n\t type: " + (parse.type == "<-" ? "response" : "request") + "\n\t message: " + m.getData() + "\n" + "**************************\n"); try{ // Construct and send the response message. Date d = new Date(); SimpleDateFormat sdf = new SimpleDateFormat( "MM/dd/yyyy h:mm:ss a"); String msg = "Response: " + sdf.format(d); System.out.println( "**************************\n" + "Agent - Sending response: " + "\n\t message: " + msg + "\n" + "**************************\n"); agent.publish(parse.prefix + "<-" + parse.postfix, msg); } catch(Exception ex){ System.err.println("Agent exception: " + ex.toString()); ex.printStackTrace(); } } } ``` `Monitor` M - who is subscribed to Topic `^M<-.*`- receives the response message, recognizes this as a response topic because of the presence of the `<-` symbol, parses the postfix of the Topic field to see that it is from `A` and completes processing the heartbeat. ```java showLineNumbers // Monitoring System subscription. MonitorMessageHandler mh = new MonitorMessageHandler(); monitorClient.subscribe(mh, "^M<-.*", 10000); ... static class MonitorMessageHandler implements MessageHandler{ public void invoke(Message m){ String topic = m.getTopic(); TopicParse parse = TopicParser.parse(topic); System.out.println( "**************************\n" + "Monitor - received message:" + "\n\t type: " + (parse.type == "<-" ? "response" : "request") + "\n\t message: " + m.getData() + "\n" + "**************************\n"); } } ``` In our example, processing the message means that we write a message to the console noting that the heartbeat request and response message have been received. In a more sophisticated system, additional logging and processing could take place. ```java showLineNumbers class TopicParser { public static TopicParse parse(String topic) { TopicParse parse = new TopicParse(); if (topic.indexOf("->") >= 0) { // We have a request message parse.type = "->"; parse.prefix = topic.substring(0, topic.indexOf("->")); parse.postfix = topic.substring( topic.indexOf("->") + 2, topic.length()); } else if (topic.indexOf("<-") >= 0) { // We have a response message. parse.type = "<-"; parse.prefix = topic.substring(0, topic.indexOf("<-")); parse.postfix = topic.substring( topic.indexOf("<-") + 2, topic.length()); } else { // We don't recognize this message type. parse.type = "unknown"; parse.prefix = null; parse.postfix = null; } return parse; } } ``` For completeness and to show how simple it is to create a class to parse the messages manually using our routing tokens, the `TopicParser` class has been included above. In our example, each of the three cases are enumerated in the if / else structure, and populate a TopicParse object, which is then returned as a result of the `parse()` method. The actions taken as a result of the parse function can easily be replaced with more advanced features, but that will be an exercise which is left to the reader. ## Conclusion While the example here is a simple one, it would be trivial to add even more powerful features such as: listener notifications when heartbeats are received / missed, processing messages once and only once, detecting messages received out of order. Through the powerful combination of AMPS Topic filtering, PCRE, and some creativity in combining these two features - we are able to architect a solution that provides a simple heartbeat request-response system in AMPS. ## Next Steps Try it yourself to see how easy request/response messaging can be with AMPS! Download the sample project in Java, along with instructions in the README file for building and running the sample: [01\_simple\_request\_response.zip](/downloads/01_simple_request_response.zip) --- # Break the Chains of Version Dependency > Want to see how we build our AMPS product to run on any modern Linux/x86 system? We share the secrets in this blog article. At 60East, we strive to make AMPS the most powerful, high-performance, real-time messaging database ever. This philosophy extends to every aspect of AMPS, including installation and packaging. We want you to get up and running as fast as possible, with a zero-friction install process. Since our customers run AMPS on a wide variety of operating system versions, from quite old to very new, we strive to make a single install image that works everywhere. This means we ship a single set of binaries that run on a range of Linux kernels and library versions. And, since AMPS provides extensibility via a C api and shared library modules, it is important that customers are able to use the latest C and C++ features when writing extension modules. To meet these challenges -- ease of install, version independence, and extensibility -- we've made some specific technical choices that have produced excellent results. This information isn't always easy to find or discover, so in an attempt to make a lower-latency world for all of us, we've described our approach here. Packaging up prerequisites -------------------------- Even though AMPS is an extensible product, many of our customers use the functionality we provide without extending. Many of our customers also keep very trim systems in production -- very few extra libraries installed. As a consequence, many systems have no `libstdc++.so` installed, a support library for C++ applications. If you build an application and send your customer the resulting executable, and they attempt to run it on such a machine, they will of course see: ```bash joe@somecomputer:~$ ./myapp ./myapp: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory ``` A natural reaction to this problem might be to suggest they obtain this library on their own. If they do so, or they already have it installed, your application may still not function: ```bash joe@somecomputer:~$ ./myapp ./myapp: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15` not found ( required by myapp ) ``` Why did this happen? Not every `libstdc++.so` is the same. As gcc evolves, so do the contents of this library. When version X of a compiler is used to produce a binary, assuming that binary is dynamically linked to `libstdc++.so`, *at least* version X of `libstdc++.so` must be present and loaded as well. This error tells us that the `libstdc++.so` in `/usr/lib64` is too old for our application: our application has used features of the library that aren't available in the version this computer has installed. A common way to fix this is is to ship `libstdc++.so` along with `myapp`. `libstdc++.so` depends on `libgcc_s.so`, so we've copied both into the `libs/` directory just below `myapp`. Let's see how this turns out: ```bash joe@somecomputer:~$ LD_LIBRARY_PATH=libs/ ./myapp Hello! ``` That worked! We can even see exactly which libraries are loaded by myapp, using `ldd`: ```bash joe@somecomputer:~$ LD_LIBRARY_PATH=libs/ ldd ./myapp linux-vdso.so.1 => (0x00007fffbe9f2000) libstdc++.so.6 => libs/libstdc++.so.6 (0x00007fcb13b85000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcb1379b000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fcb13496000) /lib64/ld-linux-x86-64.so.2 (0x00007fcb13e8b000) libgcc_s.so.1 => libs/libgcc_s.so.1 (0x00007fcb13280000) ``` Notice how these libraries are now loaded from the `libs` directory instead of our OS. That's exactly what we want. That LD_LIBRARY_PATH setting is troublesome, though. Sure, we could wrap that up in a shell script that the user runs, instead of running our application. Instead, let's use another feature of the linker to eliminate the need for LD_LIBRARY_PATH. rpath to the rescue ------------------- So far, when we build `myapp`, we haven't done anything special, just `g++ -o myapp myapp.cpp`. The compiler and linker build an application that, when run, looks in the default system search path (see the man page for `ld.so` and `ldconfig` to learn more) and LD_LIBRARY_PATH to find shared libraries. Using the linker's `rpath` option, we can build an application that looks in a more specific path for libraries before searching the defaults. ```bash joe@somecomputer:~$ g++ -Wl,-rpath=\$ORIGIN/libs/ myapp.cpp -o myapp joe@somecomputer:~$ ./myapp Hello! joe@somecomputer:~$ ldd myapp linux-vdso.so.1 => (0x00007fff164fe000) libstdc++.so.6 => libs/libstdc++.so.6 (0x00007f2fb6d3c000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2fb6952000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2fb664d000) /lib64/ld-linux-x86-64.so.2 (0x00007f2fb7042000) libgcc_s.so.1 => libs/libgcc_s.so.1 (0x00007f2fb6437000) ``` Great -- now users of `myapp` no longer need to think about `LD_LIBRARY_PATH`. In the example, note how we also used a relative path for the `-rpath` argument along with the special `$ORIGIN` value. When the program is run, this special `$ORIGIN` value acts as a special token to the runtime loader, `ld.so`. This token causes `ld.so` to locate shared libraries relative to the location of `myapp`, even if we launch `myapp` from some other directory. As long as we maintain this same relative location between our binary and libraries, we can redistribute this directory structure and the runtime loader will do the right thing. Problem solved. libstdc++ has needs, too ------------------------ You've been successful so far distributing your application, when one day, someone emails you asking for assistance. Here's the error they see: ```bash myapp: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by libs/libstdc++.so.6) ``` Uh-oh. What is this? Just like your application is built with and now depends on a certain version of `libstdc++.so.6`, the same is true for `libstdc++.so.6`. It is built against a particular version of the C runtime library, `libc.so`. If `libstdc++` uses features from `libc` that aren't there on the much older OS where you try to run the application, you'll see this kind of error. Chances are, this customer's libc, and likely their Linux kernel, are much older than the one where you built your application. It is tempting to apply the same technique we did before. Just ship `libc.so.6` with our application. This might work, but chances are, you'll run into a new kind of error when you run the app, with your libc, on an older machine. Here are some of the ones we've seen, heading down this road: ```bash oldjoe@oldcomputer:~$ ./myapp ``` FATAL: kernel too old Segmentation fault Eventually, you run into a thing you can't simply package up and ship yourself: the Linux kernel. You could try to ship an older libc that works on old kernels -- but even if you succeeded at shipping libc, and it worked across your target OS versions, that's not necessarily a wise choice. Shipping an old libc means shipping old [security vulnerabilities](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CCoQFjAA&url=http%3A%2F%2Fwww.cvedetails.com%2Fvulnerability-list%2Fvendor_id-72%2Fproduct_id-767%2FGNU-Glibc.html&ei=4yEEU86EFc2BogSE1YHoCQ&usg=AFQjCNHNG4bJlje6g0G02kpST_JQ6m1g6g&sig2=4QHbV6UhIFHQH9B6TGattw&bvm=bv.61535280,d.cGU), for example, and doing so just for the sake of supporting old operating systems. Instead of all that, we chose to do something recommended by many: build our product, AMPS, on the oldest supported operating system, and then distribute the resultant binaries. How low can you go? -------------------- An important question to ask yourself up-front: what is the oldest Linux version I can reasonably support for my application? Presumably you have an idea of what your customer base needs: perhaps a specific older version is required, or you're looking to match the support matrix of another important piece of software. What kernel features does your software, and its dependencies, need? That can be a harder question to answer. If you've been developing and testing on relatively new Linux versions, you'll have work to do, to make sure your software not only functions but performs acceptably on your target OS. For us, these considerations culminated in choosing Linux 2.6.9 as our low-end. Now the solution to everything is clear: install an old 2.6.9-based distribution, build AMPS, and distribute. Except... That's a really old compiler ------------------------------------------- The 2.6.9-based distribution we used, CentOS 4.8, bundles gcc 3.4.6. That's really old. C++ has changed a lot over the years, both in terms of language features and library features. It would be one thing if that choice only impacted the author of the application -- we would just bear the brunt of using the subset of C++ features available to us. But things get trickier when we want to allow extensibility. In AMPS, users are allowed to write their own plug-in modules in C or C++, for things like authentication, entitlements, and custom actions. These modules are shared libraries, and by way of configuration, AMPS loads them shortly after start-up. These modules are often quite sophisticated, and many customers desire to use the latest language and library features available to them. This is where our solution needs a little more work. We've already seen the issues with including the `libstdc++` from a recent compiler and Linux version. Everything works fine if we build and distribute binaries from our old OS. But if, inside our application, we attempt to `dlopen()` the customer's extension module, _and_ that extension module uses newer C++ features, we're right back to this error message: ```bash joe@somecomputer:~$ ./myapp ./myapp: libs/libstdc++.so.6: version `GLIBCXX_3.4.15` not found ( required by customer_extension_module.so ) ``` To review, our application, which distributes its own libstdc++, loaded just fine. But the customer's extension module, which uses new C++ features, doesn't load. When `myapp` attempts to load it, the runtime loader sees that `libstdc++` is already loaded into the process, and attempts to resolve needed symbols from it, rather than trying to find a newer, better `libstdc++`. `customer_extension_module.so`, depending on new features, doesn't find what it needs in the old `libstdc++` we shipped. Telling customers not to use recent compilers is clearly unsatisfactory. So is telling them to replace the `libstdc++` we ship with a newer one. And because some customers don't have one at all, or only have old ones, we still need to ship one. The best solution we've found is a hybrid: ship a new `libstdc++` from the newest released gcc (currently 4.8.2). However, we need to build a custom version of this library, and indeed of the compiler, on our oldest target machine: CentOS 4.8. At the end of that process, we'll have a `libstdc++` that contains the latest C++ features, but still loads on the oldest `libc` and kernel we can support. Building your own gcc --------------------- Building a new GCC for the first time on your legacy OS can be intimidating. This document is not meant to replace nor contradict any of the excellent information regarding how to do so, but instead provides more specific guidance about how to do so for this particular scenario. Make sure to review the [official installation documentation](http://gcc.gnu.org/install/) and the installation [wiki](http://gcc.gnu.org/wiki/InstallingGCC). Both resources are invaluable in understanding the process of producing a gcc build that is usable for your environment. Note the following: 1. If you have the choice, start out with a Linux distribution that has a GCC and Make that is recent enough to build the newest GCC. (CentOS 4.8 had adequate versions of all of these prerequisites.) 1. You'll likely want to download the prerequisite libraries for GCC and build them from source, rather than using the old ones on your legacy OS. The `contrib/download_prerequisites` script mentioned in the [wiki](http://gcc.gnu.org/wiki/InstallingGCC) makes this easy. 2. You can greatly decrease build time by disabling unnecessary features when configuring. Since I only needed x64 libraries, my `configure` command looked like this: ```bash joe@oldcomputer:~/gcc-build$ ../gcc-4.8.2/configure --disable-multilib --enable-languages=c,c++ ``` 3. You probably do not want to replace the default compiler on your old OS. You can specify a different directory to install gcc and the built libraries into using the `DESTDIR` variable: ```bash joe@oldcomputer:~/gcc-build$ make DESTDIR=/home/joe/gcc-output-tree install ``` I ran into mysterious errors when I used a relative path for `DESTDIR`; it appears you must specify an absolute path. If the build process succeeds, your output tree contains a compiler and a set of shared libraries that will load on platforms at least as old as the one you're on, and because of the backwards compatibility requirements of libc and the Linux kernel, should work on the most recent OS distributions. Build your application with this compiler, using the `rpath` technique to ensure it depends on the libraries you will ship. Then, when you're satisfied with the result, package up the libraries and application together. The final product should work on operating systems as old as the one you built gcc on, all the way to the most recent, and (if applicable) can be extended by customers using the latest gcc and C++ features. Summary ======= We accomplished what we set out to do: build a distributable binary package that works on a wide variety of Linux versions, with a minimum of prerequisites, and allow customers to write extension modules with the most recent gcc. To get there yourself: 1. Identify the *oldest* Linux version/distribution you need to support, and use it for the following steps. 2. Install and build the *latest* gcc on that platform -- at least the C and C++ languages. 3. Build your application there using the `-Wl,-rpath` flag with a relative path and the `$ORIGIN` token to specify where your prerequisite libraries should be found at runtime, relative to your binaries. 4. Package up your application and the `libstdc++` and additional prerequisites you built in step (2), and distribute them together. Make sure the directory structure once installed matches the relative path specified in step (3). It might sound like a lot of work, but the results are worth it. Your customers can get up and running with your application quickly, and extend it with whatever compiler version they'd like. How Do You Do It? ================= In this post, we talk about the best way we've found so far to ship version-independent binaries and libraries on Linux. Do you have a tip or trick we've missed? Curious about something we didn't go into here? Let us know in the comments! --- # Real-time Streaming JOINS, Reinvented! Why does CEP have to be complex? Real-time streaming JOINs don't have to be complicated to code, and they don't have to be the bottleneck in your system. In AMPS, we designed our JOINs from the ground up to be easy to use and highly performant. AMPS is unique in that it provides a hybrid approach to data movement and storage. Thanks to AMPS’ revolutionary State of the World cache (SOW), we are able to provide unbounded, windowless JOIN support in real-time! Companies rely on AMPS every day to process data at incomparable speeds, but now with JOIN support, we have eliminated the need for latency-adding (and pricey) secondary CEP systems. Joining data upstream, relieves expensive downstream work, thus lowering overall latency. Decisions due to Embellishment ============================== In real-time systems, the very mention of the word "JOIN" brings to mind one thing - increased latency. JOINs in the database world are a necessity, whereas in the streaming world, they should be considered a luxury. All of this super-fast streaming data comes at a typically unacknowledged cost - the data still needs to be consumed. Streaming data must be embellished, interpreted, and applied to some business process. Whether that process is manual or assumed by other software system, it is added to the end-to-end latency. All this extra data has a remarkable impact to performance, which is unacceptable in any serious venue. Then why would one even consider the use of a JOIN when streaming real-time data? As it turns out, you can get by perfectly fine without joining disparate entities. That is, until the data needs to be ingested into a decision-making process. To obtain a clear picture on what we mean, let's begin with a seemingly simple example. Publisher/subscriber systems process Orders in real-time, whereas the Customers and Limits are not processed with the orders, but rather stored in a historical database. To extract meaningful insight out of the orders, they must be combined with the customer and limit data. There are many ways to do this, but we tend to see complex event processing (CEP) systems assuming this role. Traditional complex event processors are great for such aggregation, but they lose their luster when you decouple them from the ultra-fast streaming engines. ![Embellishment with a CEP](/img/blog/embellish_joins-opaque.png "Embellishment with a CEP")
    Figure 1: Embellishment with a conventional CEP
    Even the fastest CEP engines still need to ingest data from the publisher via an adapter and transform that stream into workable data. An introduction to *windowless* real-time, streaming joins. =========================================================== Traditional CEP systems require bounded JOIN support; this means that you typically must provide a time-based window in which to exercise your JOIN statement. In CEP systems, the streaming JOIN operation is temporal, meaning that disparate streams of events must have overlapping lifetimes. It is not to be confused with the joining of historical and embellishment data, which is typically stored in a separate database system. The streaming JOIN operation of traditional CEP systems are conceptually different than historical database systems, where a JOIN is evaluated over the entirety of your source data. The reason why these systems differ in their support of JOINs is due to how they fundamentally access data. CEP systems work on “streams” of data, whereas databases work on stores of data. Due to AMPS’ revolutionary State of the World cache (SOW), we are able to provide a hybrid approach – one where you can JOIN data streams with historical data in real-time. But what does it mean when we say “real-time?” Well, CEP systems, while operating on streams of data, tend to operate over specified windows of time. Whether utilizing sliding windows, hopping windows, or some variation of the two, the entirety of the window’s specified time must elapse before a calculation can occur. The “windowed real-time” operation of the CEP JOIN is now delayed by the elapsed time of that window. AMPS operates in “real-time” in that we JOIN data as it arrives, rather than waiting for some expiration. Additionally, as mentioned above, your streaming data can be joined with your historical data in the same real-time latency. Let’s now examine how we would implement the previous example in AMPS. The differences should be immediately clear; there are no adapters, there are no delayed windows, and there is only one system from which to aggregate data – this is AMPS! All of your data, whether real-time, historical, or embellishment is treated exactly the same. Not only does this greatly simplify development and administration due to its conceptually clean implementation, but it also allows for slowly updating sources (Limits and Customers). ![Integrated streams](/img/blog/real_time_stream-opaque.png "Integrated streams")
    Figure 3: AMPS embellishment
    ![View definition](/img/blog/joins_view_definition.png "View definition")
    Figure 4: View definition in AMPS

    ```bash ./spark subscribe -type nvfix -server localhost:9003 -topic CUSTOMER_TOTAL_OF_LIMIT ```
    Figure 5: Subscription to current order stream in AMPS

    Back to the Future ================== In the previous example, we saw how we can enrich the order data with its customer and their daily limits. But what if we would like to examine a customer’s activity over the past year? In your CEP system, you probably need to create another input adapter to your historical data store. If you would like to combine the customer’s history with their current activity, you’ll need to create a CEP stream that JOINs the two data inputs and figure out some way to negate the overlapping data. At this point, you have the following: 1. JOINs within your current activity stream, 2. JOINs within your historical activity stream, 3. A JOIN to link your current and historical activity streams into a single stream, 4. A expensive de-duplicating operation, 5. A window, expressing your current activity stream ![Adding historical data to the CEP Stream](/img/blog/historical_data_with_cep-opaque.png "Adding historical data to the CEP Stream")
    Figure 6: Adding historical data to the traditional CEP Stream
    Let’s compare this with how you would accomplish the same thing in AMPS. In AMPS, you already have the query that JOINs orders, customers, and their daily limits. Since the SOW stores current and past data, we only need to query the SOW for the old Orders and subscribe to the new Orders. We do this through the use of AMPS' unique `sow_and_subscribe` command: ```bash ./spark sow_and_subscribe -type nvfix -server localhost:9003 -topic CUSTOMER_TOTAL_OF_LIMIT ```
    Figure 7: SOW (query) and Subscribe to historical Orders and the current Order stream in AMPS

    Notice that I did not include a new architectural diagram. The architecture here is the same as it was in Figure 3. That's because, unlike a traditional CEP, AMPS doesn't require a hodgepodge of disjoint systems to accomplish conceptually simple tasks. Its power and simplicity derive from a smart, robust, and extensible architecture - one that puts developers first! Try it yourself! ================ In this article, I have made some grand statements on our performance, comparisons with other products, and our ease of use. If you’re intrigued by them, whether they spark your interest or you simply disagree, we invite you to [download a free evaluation](/evaluate) and try it yourself! --- # Joining BSON Data with XML Data and Aggregating in JSON -- Making it Easy and Natural Data aggregation as easy as it is in the movies: Find out how! We’ve all seen television’s expectation of middleware – real-time streams of data, arriving from all over the world, effortlessly joined and available in an instant, where combining new information is as easy as a few keyboard clicks – even in the middle of the night, from an underpowered laptop, while the clock is counting down to a major disaster. The reality is that the systems of today are not how they are presented on television, where data feeds are ubiquitous and are easily aggregated in “the cloud.” Most likely, you rely on systems that are of varying age and communicate via disparate messaging types. These message types, whether XML, JSON, BSON, FIX, or even CSV, need to be converted to a common format so they can be consumed. **Where does this conversion occur?** Unfortunately, conversion typically happens just before consumption in aggregation systems, which tends to limit the usefulness of the data. Furthermore, this conversion is a separate step, one that is either hand-rolled or part of a different process inside an aggregator. Such a disjoint operation is both clumsy and inefficient. It’s expensive, in both development time and processing time. No one wants to do things this way, but it’s seen as a necessary bottleneck in a data stream. **At 60East, we’re breaking the rules.** Rather than converting back-and-forth between message types, we JOIN the messages **directly**. This allows us to do some rather unique things, such as cross-typed joins into entirely different message types, in real time! It’s like Jack Bauer meets the Matrix. But let’s not just stand around while CTU gets overtaken by sentinels, it’s time to pick up that phone and dive in for an example. Suppose we have a live feed of XML GPS check-in data for all the taxicabs in NYC. Each taxi transmits vital information every second. ![Taxi Data](/img/blog/MultiType-Join-Blog-XML.png) Using the real-time aggregation capabilities in AMPS, we’re able to determine where each cab is, whether the driver is speeding, the total trip time for each pickup, the cab fare compounded with the gallons of fuel used, etc. ![Taxi Data Topic Definition](/img/blog/MultiType-Join-Blog-XML-Topic.png) Now suppose that we own multiple taxicab garages in the city and want to determine when to shut each down for the night to maximize profits. A NoSQL database contains the garage fleet information, but the feed is in the BSON message type. ![BSON Data](/img/blog/MultiType-Join-Blog-BSON.png) An additional caveat is that our front-end system wants the data in JSON format. Since AMPS supports the cross joining of discrete message types (and projects into any other message type), we simply need to define the query. ![Combined Topic Definition](/img/blog/MultiType-Join-Blog-Combined-XML.png) And there you have it! As garage owners, we are able to see which garages are the most profitable as the taxis are accumulating fares. This allows us to react in real-time and close any garages that are underutilized. Using our unique JOIN technology, we’ve demonstrated how easy it is for you to join disparate entities, which you can use to save your company countless hours of development time. But, as always, don’t take our word for it. Download an [evaluation of AMPS for free](/evaluate) and try it yourself! Then try to do the same thing in your current system – and make it real-time. --> --- # AMPS: The Ultimate Shock Absorber AMPS can help you get new life out of your existing messaging system, while providing more capacity and functionality. A shock absorber is a buffer between two systems. It protects each system from the other and helps both systems to run smoothly. AMPS provides stunning throughput, expressive filtering, high availability, slow consumer protection, and unlimited record-and-replay of messages. All of these capabilities are necessary for a shock absorber. The whitepaper describes what makes a good shock absorber, presents how the capabilities of AMPS can be used as a shock absorber, and shows the kind of performance results that 60East gets when testing customer scenarios that use AMPS as a shock absorber. Click [here](/downloads/documentation/AMPS-Ultimate-Shock-Absorber.pdf) to download and start reading! [![Ultimate Shock Absorber Cover](/img/Ultimate-Shock-Absorber-Cover.png "Ultimate Shock Absorber Cover")](/downloads/documentation/AMPS-Ultimate-Shock-Absorber.pdf)


    --- # AMPS versus Santa: Who's Faster? Santa checks out AMPS performance At 60East Technologies, we think about delivery **a lot**! We’re always thinking about what can be learned by studying real-world delivery technologies used elsewhere. Perhaps the grand-daddy of all delivery companies is Santa Claus. He’s delivering packages to approximately 526 million children in 150 million households over a 31 hour period. That’s very impressive! Just how fast is Santa and how does Santa compare to what we’ve built at 60East Technologies? Let’s see in this feature breakdown: | Attribute | Santa | AMPS | | --- | --- | --- | | Stamina | Infinite[i] | Infinite[ii] | | Horsepower[iii] | 402[iv] | 6[v] | | Durable Delivery Throughput | 1144 per second[vi] | 4,500,000 per second[vii] | | Durable Median Latency | 620 milliseconds[viii] | 250 microseconds (µs) | | Delivery Furry Creatures | 10 | 9[ix] | | Customers believe it FLIES! | Yes | Yes[x] | | Unbelievable Performance | Yes | Yes | | World-wide Coverage | Yes | Yes | | Delivery Cardinality | M:1 | M:N | | Content Filtering[xi] | No | Yes | | Queryable Transaction Log[xii] | No | Yes | | Concurrency | 1 | Infinite | | Redundancy | 0 | Unlimited Asynchronous, 64 Synchronous | | 526M Record Regex Query Performance | ~2.2 seconds[xiii] | < 1 second | | Fair Delivery | No | Yes; Everyone gets what they want. | | Time Travel | Not that we know. | Coming in v4.0! | We hope this comparison doesn’t get us on the Naughty List! We really do think Santa is doing an amazing job – magic and all! In early 2015 (just a couple of weeks away) we’ll be launching AMPS v4.0, which includes features for traveling through time – this is something not even Santa can do. Please join our [newsletter](/newsletter) to stay up to date on all of the cool things coming from our lab. And, who knows? We just might have a little more information about Santa on the way, too. Happy New Year!     [i] [Reindeer Are Part of Santa's Magic](http://www.durangoherald.com/apps/pbcs.dll/article?AID=/20111208/COLUMNISTS12/712089989&template=printpicart) from the 12/08/2011 edition of the [Durango Herald](http://www.durangoherald.com/) [ii] Ok; We don’t really know, but we have customers with production instances with uptimes of more than 1 year, and AMPS is working _everyday_ not just a 31 hour period once per year! [iii] Arguably, as long as you have the power to achieve your goals, you should be seeking to minimize the horsepower required for delivery. [iv] Assuming Santa uses large Finnish Forest Reindeer, Rudolf, and magic acorns. [v] Estimate Horsepower driven by our Shock Absorber pattern: [http://www.crankuptheamps.com/blog/2014/09/24/ultimate-shock-absorber/](/blog/ultimate-shock-absorber/) [vi] 526 million children in Santa’s coverage zone, avg 3.5 children per household, 31 hours to complete delivery (thanks to timezones!), and assuming no naughty children. [vii] You can push AMPS harder, but we’re going easy on Santa and quoting numbers from our Shock Absorber pattern. [viii] Assuming Santa travels at 904 miles per second, .56 miles median distance between households, infinite acceleration, and includes the time it takes to travel through chimney, eat cookies, stuff stockings, and do other things most people need to do in a 31 hour period. [ix]Meet the AMPS Furry Creatures: [/about/](/about) [x] We’ve had many customers that have said AMPS flies. Compared to other messaging systems, it certainly appears AMPS is endowed with some magical powers. [xi] You don’t know what’s in Santa’s bag until after delivery. With AMPS, you always get delivered what you want! [xii] NORAD tries really hard at keeping track of Santa, but there’s no history of where he’s been.  With AMPS, you can track the full history of what’s happening in your systems! [xiii] Assuming a speedy 160 words per minute to say the question and answer, 200ms Wernicke’s latency, and 600ms of memory latency for the recall. We’re taking for granted that Santa can remember the 526 million children and whether or not they were naughty/nice -- though we're working with Santa to help with that problem. (Stay tuned for details!) --- # Santa Officially Chooses To Crank Up the AMPS Santa cranks up the AMPS FOR IMMEDIATE RELEASE
    2014-12-26 North Pole -- Santa Clause, Inc. (SANTA) announced today that it will be choosing 60East Technologies' AMPS product as their strategic messaging platform. SANTA expects the demands on delivery to continue to increase at record rates given the improvements in the economy and world population increases. With the increase in demand, having a best-of-breed messaging with a SQL database and analytics platform is critical -- AMPS combines all of these into a single cohesive product. SANTA deployed AMPS version 4.0 Beta to implement their new Real-time Naughty List in early 2014. With the Real-time Naughty List, SANTA was able to deliver gifts to more than 526 million children in less than 6 hours -- a 4x reduction over 2013. SANTA CIO Mortimer J. Elf says, "Using AMPS, we launched our Real-time Naughty List, which allowed us to get a real-time view of inventory needs up to the actual delivery. We were running AMPS within a Virtual Machine on a laptop in the sleigh, getting hourly updates for each of the 526 million children -- that's about 150K updates per second. That capacity allows us to accurately deliver to children everywhere. Before deploying AMPS, we'd have difficulties tracking who was naughty or nice within the last month of delivery -- and sleepovers and family visits made tracking kids impossible. We had no Wrong-Chimney events this year, whereas these events accounted for 3% of all deliveries in 2013. The NORAD tracker couldn’t keep up." A spokesperson for 60East Technologies said, “We’re happy to have had the opportunity to partner with SANTA in the building of their Real-time Naughty List. The feature set and capacity found in AMPS uniquely suits it to the world’s most demanding applications.” SANTA Founder, Head Logistics Officer and CEO Kris Kringle summed up his satisfaction with the project and AMPS overall. “Ho, ho, ho”. About 60East Technologies Inc. ------------------------------ 60East Technologies was founded in 2010 by a world-class tandem of system programming experts with a proven track record of delivering systems to the most demanding customers. The team’s experience and expertise building high performance systems over the past twenty years, has led to the development of a technology, AMPS, that will revolutionize how real-time messaging is used to build modern applications that scale into the future. 60East Technologies is committed to building the fastest real-time streaming database software and helping application developers deliver solutions that outperform their peers. About Santa Clause, Inc. ------------------------ Santa Clause, Inc. (SANTA) is recognized worldwide as one of the industry leaders in holiday-focused parcel distribution. Founded at some point in the distant mythic past, SANTA is best known for delivering packages for upwards of 526 million customers in a single 31-hour period each and every year. We are experts in both conventional and modern logistics, and our operation combines cutting-edge technology with tradition. We pride ourselves on providing an unparalleled parcel delivery experience and groundbreaking customer satisfaction for the price of a glass of milk, a plate of cookies, and a few carrots for the reindeer. Maybe some hot chocolate, too. --- # Toolbox: Regular Expression Testing Tool The regex developer toolkit If you’re a multi-discipline developer (like us!) it can be difficult bouncing between languages with different regular expression grammars. There are often command line utilities to help and you can always write code to test your patterns against test data. That said, there are some great regular expression testing tools online that make it really easy to write and test your regular expression grammars. Our favorite online tool is [regex101.com](http://regex101.com), because it supports PCRE grammars, which is what we use in our AMPS product and surrounding utilities. Let’s say I have a list of regionalized topics I publish to for both ORDERS and EXECUTIONS, but want to place a subscription that only selects the all ORDERS topics (`^ORDERS_.*$`) or everything from the TKO region (`^.*_TKO$`). Here’s how you’d use the regex101.com tool to write a fancy regular expression: 1. Goto [regex101.com](http://regex101.com)! 2. Add the list of the topics you publish to in the “TEST STRING” box: ![List of topics](/img/blog/regex-images/list_of_topics.png) 3. Add the test pattern to the “REGULAR EXPRESSION” test box, notice I’ve also selected the “pcre” regular expression flavor, because PCRE is the grammar used in AMPS topic regular expression matching: ![Test pattern](/img/blog/regex-images/test_pattern.png) 4. You’ll notice that nothing in my pattern matches, but that’s because I need to add a couple of special modifiers to get the multiline matching to work in this sandbox. If you add `mg` to the modifier box like in the next snapshot, you’ll see that the correct topics are matching. The `m` is for multiline matches where ^ and $ operate on the line rather than the entire input string, which is handy for a list of topics like we’re using. Also, the `g` says you want to match globally, not just the first match. ![Test pattern with mods](/img/blog/regex-images/test_pattern_with_mods.png) 5. Notice also the right-hand side of the page that explains the grammar to you in English with descriptions of all modifiers and a list of frequently used tokens – a nifty feature! ![Explanation](/img/blog/regex-images/explanation.png) Summary: We hope you find this tool as useful as we do. It really makes testing regular expressions easy and the ability to bounce between Javacript, Python, and PCRE is a real time saver. --- # AMPS 4.0: Back from the Future Back from the future with never-before-seen technology It’s great to be back! We just returned from a trip to 2025. While we were there, we took a look at what messaging systems and databases were like in 2025. We brought a bunch of cool stuff back and added it to AMPS 4.0. We’ve revealed some of these secrets before: * *Historical Database Query*. Travel back in time. AMPS includes the ability to save the state of a State of the World (SOW) database and query the state of the SOW at any point in time. For an explanation of how this feature enables unique time-based queries, see Jeffrey M. Birnbaum's presentation [What Would You Build with a Data Time Machine?](https://www.youtube.com/watch?v=3hcO3KVgzko)

    [![Data aggregation as easy as it is in the movies: Find out how!](/img/blog/thumbnails/matrix-24.png "Data aggregation as easy as it is in the movies: Find out how!")](/blog/joining-json-bson-xml/) * *All your data works together*. In the future, all your data works together, regardless of the format. In AMPS 4.0, you can create views and aggregates that JOIN data from different message types and produce data in any message type you want: [just the way data works in the movies](/blog/joining-json-bson-xml/). We’re still making sure that none of the details in the blog posts will [contaminate the timeline](http://en.wikipedia.org/wiki/Temporal_paradox#Branching_universe_hypothesis), but here are some of the things that are available in AMPS 4.0 now that we’ll be talking about in the weeks to come: * *Use any message format, even formats that haven’t been invented yet*. We can’t reveal exactly what message formats we used in 2025, but we can tell you that some of the popular formats haven’t been invented yet. AMPS 4.0 allows anyone to create a message type plugin that seamlessly works with existing message types. We’re ready, just as soon as the formats are. And we know that at least one of you is working on [that format](http://xkcd.com/927/), [already](http://en.wikipedia.org/wiki/Bootstrap_paradox). * *You control your query results*. In the future, you can get exactly the query results you need. AMPS 4.0 provides `OrderBy` and `TopN` on SOW queries to let you control how query results return. * *Mind-blowing performance that scales up and down with your hardware*. We’ve made a host of futuristic performance enhancements, including a "slow lane” to further isolate the effect of slow clients on the rest of the system. * *Keep track of things*. Some classic techniques are still classic. AMPS now allows you to set a user-provided `CorrelationId` on messages, to help applications that need to correlate messages without parsing the message body. * *Choice in programming*. In the future, APIs provide the ability to easily choose both fine-grained control and ease of use. The 4.0 AMPS clients include synchronous interfaces to make them easier to use for simple applications. The clients also include a new Command interface to give you precise control over how your program interacts with AMPS. Best of all? You can use either of those interfaces independently, or use them both together for both simplicity and control. That’s not all we brought back. We’ve also made lots of improvements that may not make for a fascinating blog post, but still make AMPS easier to use. Here are a few: * New JSON, BSON and uninterpreted binary message types * Operations enhancements, including out-of-the-box support for running as a service * Statistics changes to focus on the statistics that are most easy to take action on * Performance and stability enhancements to help AMPS scale to the hardware, speeds, and capacities common in 2025 In 2025, software reliability and performance is even more important than they are in 2015. We’ve taken that to heart and done extensive work extending and improving our Quality Assurance process for AMPS 4.0. We regularly run performance and stress tests that simulate the most demanding environments – both now, and for the next ten years. Many of these tests detected failures or reduced performance on previous releases of AMPS. There’s a blog post on our QA improvements coming up, too. Ready to live the future now with AMPS 4.0? Want a sweet shirt with a lime green DeLorean on it? And be kept up to date on what's going on in our labs? The first 100 people to sign up for our newsletter will be styling! (Please let us know what size shirt and where to ship the shirt, too.) And we can tell you (without contaminating the timeline) that these shirts are even cooler in 2025. UPDATED March 20 2015: The shirts we mentioned above are gone, but you can still stay up to date with 60East by [signing up for our newsletter](/newsletter). The improvements in 4.0 are just the beginning! Oh, and last thing. SPOILER ALERT: There are still no flying cars in 2025. Surprised us, too. --- # Yuck! Stateless Poison Message Handling Danger: Keep your application away from poison! It's January, and many people (including me) are thinking about the food they've eaten over the last few months. If you’ve ever eaten too much Halloween candy, indulged yourself in holiday food that doesn’t quite agree with you, had too many cookies or gone back for that one last sliver of pumpkin pie, you may know how much trouble eating the wrong thing can cause. That’s also a problem in messaging applications – if your application is unable to process a message because it’s malformed, oversized, or contains nonsense data, problems can arise. In messaging terms, a message that can’t be processed by an application is called a _poison message_. These messages can cause big problems: imagine if you have a set of applications showing active orders when a poison message comes in. Each of the clients fails to parse that message and restarts. The clients then use AMPS bookmarks to pick up processing where they left off. They receive the message again, and restart again. What happens next? The clients pick up where they left off, and the whole process starts over. The clients keep crashing and restarting, and no work is getting done. That’s a serious problem. Here’s a technique to help applications protect against poison messages. The technique is spelled out in this post, and there's also a sample available for [download](/downloads/poison_message.zip). There are two parts to the technique. First, we define a SOW topic to hold a list of poison messages: ```xml showLineNumbers ./sow/%n.sow ADMIN_PoisonMessages json /bookmark ``` We’ll use this topic to hold the bookmark for each bad message. In addition, in cases where we have an exception, we’ll record the contents of the exception. Applications use a `sow_and_subscribe` to get the current list of bad messages. If a message comes in that’s on that list, the application skips the message rather than processing it. If the application fails to process a message, it publishes that message to the SOW topic. Here’s a simple wrapper, written using the AMPS Python client, which demonstrates how to wrap an existing message handler to provide poison message protection. First, we define the wrapper class and create an initializer: ```python showLineNumbers class MisterYuck: # Initialize the class with the template client, # and the handler to wrap. def __init__(self, client, handler): self.skipMessages = [] self.handler = handler self.doneLoading = False self.currentBookmark = None #create a new client to use for the admin subscription self.client = AMPS.Client("poison_monitor-%s" % uuid.uuid1()) # Copy the connect string from the provided client self.client.connect(client.get_uri()) self.client.logon() self.client.sow_and_subscribe(self.updatePoisonMessages, "ADMIN_PoisonMessages") # wait for the admin SOW query to complete while(self.doneLoading == False): time.sleep(.25) ``` The initializer takes a client and the handler that will do the actual work of processing the messages. The initializer saves the handler and uses the connect string from the client to create a new client to use to monitor the `ADMIN_PoisonMessages` topic. To keep the sample simple for this post, we don’t do all of the things that might be necessary for a production client (such as setting an Authenticator, or creating an HA client that can handle failover). Notice that we create a new client to avoid the deadlock that would occur when calling publish for a client from within a message handler for that client. The initialize method issues a sow_and_subscribe command to populate the list of poison messages, then waits for the SOW query that populates the list of poison messages to complete. This delay avoids the situation where a poison message could arrive before the list of poison messages is fully populated. The snippet below shows the message handler for the `sow_and_subscribe` that manages the subscription to the `ADMIN_PoisonMessages` topic. ```python showLineNumbers def updatePoisonMessages(self, m): if (m.get_command() == "group_begin"): return if (m.get_command() == "group_end"): self.doneLoading = True return print "Trace: Updating poison message dictionary!" self.skipMessages.append(json.loads(m.get_data())["bookmark"]) ``` Again, the handler is straightforward. The handler ignores the `group_begin` message. The `group_end message` sets the flag that indicates the SOW query is complete, so the `__init__` method can return. For any other method, we extract the `“bookmark”` field of the message and add the value of that field to the skipMessages list. Notice that the wrapper uses `sow_and_subscribe`. This means that any time there’s an update to the poison message list, the wrapper receives that update, so the poison message list is always current. Once the `skipMessage` list is populated, the wrapper is ready to use. The wrapper implements the `__call__` method so that you can easily pass the wrapper to methods in the AMPS client. For each incoming message, the wrapper checks to be sure that the bookmark isn’t on the list of bad messages. If so, the wrapper skips the message. Otherwise, the wrapper passes the message to the handler provided when the class was created: ```python showLineNumbers def __call__(self, m): self.currentBookmark = m.get_bookmark() # Process the message if it's not on the skip list if self.currentBookmark in self.skipMessages: print "Trace: skipping bookmark: %s -- I know it's bad" % bookmark return try: self.handler(m) self.currentBookmark = None except Exception as e: self.client.publish("ADMIN_PoisonMessages", json.dumps( { "bookmark" : self.currentBookmark, "why" : "%s" % e } )) self.currentBookmark = None raise ``` If an exception occurs during processing, we publish the bookmark and reason to the `ADMIN_PoisonMessages` SOW topic and rethrow the exception. Notice that this method doesn’t update the skipMessageList itself. The sow_and_subscribe will receive the update and automatically add the bookmark to the skip messages list, so there’s no need to do that work here. Last, but not least, we provide a pair of cleanup methods for the class. The close method can be used to deliberately shut down the class. The `__del__` function cleans up the client. Notice that if the `__del__` function is called while a bookmark is active, the wrapper assumes that message processing caused a problem and attempts to publish that bookmark to the list of poison messages. This helps to protect against a class of problems (for example, a handler detecting a fatal error and calling sys.exit()) that indicate a failure without throwing an exception: ```python showLineNumbers def __del__(self): # If the wrapper is being deleted while currentBookmark # is set, that indicates a serious error. if (self.curentBookmark != None and self.client != None): self.client.publish("ADMIN_PoisonMessages", json.dumps( { "bookmark" : self.currentBookmark, "why" : "Fatal error." } )) if (self.client != None): self.client.close() self.client = None self.handler = None def close(self): self.currentBookmark = None self.__del__() ``` That’s the full wrapper. Here’s a simple sample program that demonstrates using the wrapper. To simulate errors, the sample program throws an exception if it receives a message where the “id” field is evenly divisible by 13. ```python showLineNumbers from MisterYuck import MisterYuck def no13Handler(m): dict = json.loads(m.get_data()) if dict["id"] % 13 == 0: raise ValueError("I'm superstitious...") print "Whew, a value of %d is fine." % dict["id"] def main(): client = AMPS.Client("handler_demo") client.connect("tcp://localhost:9007/amps") client.logon() poisonprotector = MisterYuck(client, no13Handler) client.bookmark_subscribe(poisonprotector, "ContentTopic", bookmark=AMPS.Client.Bookmarks.EPOCH) while True: time.sleep(100) main() ``` To keep things simple, we've used the `AMPS.Client` and started each subscription from the beginning of the transaction log, `EPOCH`. A production application would generally use the HAClient to resume the subscription at `MOST_RECENT`, and the wrapper would call `discard()` for poison messages so the client doesn't read them again on restart. With those small changes, the same techniques work. In this post, we’ve covered an easy way to use SOW topics to create a poison message list. The concept is simple: create a SOW topic in AMPS to hold the list of bad messages. Each instance of the application loads the list and uses `sow_and_subscribe` to keep the list current as other clients update the list. AMPS SOW topics provide an easy way to build reliable, flexible poison message protection without keeping any persistent state on the application side. Even better, because the state is stored in AMPS, if a new instance of the application starts, that new instance never has to process messages that are known to be bad. Each new instance gets the benefit of the work the other instances have done. --- # New Feature: Keep It All Together With CorrelationId Say hello to a new feature! One of the best things about AMPS is the way that it keeps publishers completely independent from subscribers. Publishers don't need to know how many subscribers are listening for a message, where they are, or even whether they're connected at a given point in time. That flexibility pays off: once publishers are set up, you can use the same message stream for any number of different applications, without changing the publisher. Every now and then, though, this decoupling has a disadvantage. Like helping subscribers identify which messages belong together, even if they're on a different subscription. Or communicating metadata, like a response address, that isn't part of the original message. One way to do this is to enrich the message. With this approach, the publisher can add an extra field with the correlation information to the message before sending it. The subscriber then uses the extra field to figure out which messages go together. There are some real advantages to message enrichment: because the correlation information is part of the message, AMPS can filter on that field, which can increase the precision of your subscriptions. There are some drawbacks to enrichment, though. You need to add that field to every message when you publish the message. If your AMPS publisher isn’t the original source of the message, this adds extra work because your publisher needs to parse and reassemble the message. Likewise, your subscriber may need that correlation information to figure out how to handle the message, which may mean parsing the message before parsing is required, or (even worse) parsing the message more than once. For lots of applications, enrichment adds minimal overhead, and the added filtering capability makes enrichment a great option. For other applications, though, it’s expensive or impossible to enrich messages. What then? In AMPS 4.0, there’s another option. We’ve added an optional `CorrelationId` header for messages published to AMPS. The publisher sets the `CorrelationId` before sending the message. AMPS passes along the `CorrelationId` to the subscriber, without parsing, altering, changing, folding, spindling, or mutilating the `CorrelationId`. ## Code it Up Enough introduction, let’s look at the code. First, a simple publisher sets a value on the CorrelationId: ```python showLineNumbers # Set up a subscription to wait for replies. def get_replies(message): print "Got a response %s on topic %s" % \ (msg.get_data(), msg.get_topic()) client.subscribe(get_replies,"reply-to-[0-9]") # Publish messages command = AMPS.Command("publish").set_topic("all-the-things") # use the CorrelationId as a reply to topic for i in range(1,10): command.set_correlation_id("reply-to-%d" % i) command.set_data('{"message":"hello, subscriber!"}') # because we don't check for a response, we use # execute_async with no message handler client.execute_async(command, None) # wait for replies while True: time.sleep(1) ``` First, the publisher sets up a subscription to process replies. It's important to make sure that the subscription is running before sending any messages, because subscribers may begin replying before all of the messages are sent. To send the messages, the publisher first creates a publish Command that will hold the common information for all o the publish requests. Each time through the loop, the publisher fills in a distinct `CorrelationId` and the message data, then publishes the message. Once all the messages are published, the subscribe waits for replies -- each on a different topic communicated to the subscriber in the `CorrelationId`. That’s all there is to it. AMPS doesn’t process the CorrelationId, so there’s no need to provide it in any particular format. For the subscriber, we just receive the messages and print what we get. Because, in this case, we use the CorrelationId as the topic to reply to, we just send a message back to the specified topic: ```python showLineNumbers for message in client.subscribe("all-the-things"): print "Correlation: '%s' on message '%s'" % \ (message.get_correlation_id(), message.get_data()) client.publish(message.get_correlation_id(), \ '{"response":"hi, world!"}') ``` The output of the subscriber is shown below: ```sh Correlation: 'reply-to-1' on message '{"message":"hello, subscriber!"}' Correlation: 'reply-to-2' on message '{"message":"hello, subscriber!"}' Correlation: 'reply-to-3' on message '{"message":"hello, subscriber!"}' Correlation: 'reply-to-4' on message '{"message":"hello, subscriber!"}' Correlation: 'reply-to-5' on message '{"message":"hello, subscriber!"}' Correlation: 'reply-to-6' on message '{"message":"hello, subscriber!"}' Correlation: 'reply-to-7' on message '{"message":"hello, subscriber!"}' Correlation: 'reply-to-8' on message '{"message":"hello, subscriber!"}' Correlation: 'reply-to-9' on message '{"message":"hello, subscriber!"}' ``` And, once the subscriber replies to the messages, the publisher produces this output: ```sh Got a response {"response":"hi, world!"} on topic reply-to-1 Got a response {"response":"hi, world!"} on topic reply-to-2 Got a response {"response":"hi, world!"} on topic reply-to-3 Got a response {"response":"hi, world!"} on topic reply-to-4 Got a response {"response":"hi, world!"} on topic reply-to-5 Got a response {"response":"hi, world!"} on topic reply-to-6 Got a response {"response":"hi, world!"} on topic reply-to-7 Got a response {"response":"hi, world!"} on topic reply-to-8 Got a response {"response":"hi, world!"} on topic reply-to-9 ``` It's just that simple. ## Are there provisos? Caveats? Quid pro quos? No! Absolutely not! It’s just that simple. Well, most of the time, anyway. There are a few things you should know about how AMPS treats `CorrelationId`. These mostly boil down to a simple principle of “do the right thing when the right thing is clear, otherwise do nothing”. * For **SOW records**, the `CorrelationId` of the record is the `CorrelationId` of the most recent message that updated the record. Exactly what you would get if the `CorrelationId` were a field in the SOW. What if there’s no `CorrelationId` on that SOW record? Then AMPS doesn’t provide one – the message from AMPS doesn’t include that header. * For **transaction log replay**, AMPS includes the `CorrelationId` on the replayed messages. To make this possible, AMPS stores the `CorrelationId` for a message in the transaction log when the message includes a `CorrelationId`. * For **delta publish**, the `CorrelationId` of the record is updated if there’s a `CorrelationId` on the update. Otherwise, AMPS preserves the existing `CorrelationId`. * For **delta subscribe**, AMPS provides the `CorrelationId` on the message if the record in the SOW has a `CorrelationId`, or if the new publish adds one. Otherwise, no `CorrelationId`. This is the same principle as SOW records. * For **views** (including JOINs and aggregations), AMPS never provides a `CorrelationId`. In this case, the results of the view might come from messages that have different `CorrelationId` values. Which one is the best? Who’s right and who’s wrong? How can AMPS choose a favorite? There’s no good answer, so AMPS doesn’t provide a `CorrelationId`. * For **out-of-focus messages**, the `CorrelationID` AMPS provides depends on why the message went out of focus. If the message has gone out of focus because it no longer matches the subscription, AMPS provides the `CorrelationID` of the update. If the message has been deleted, or the subscriber is no longer entitled to see the new message, AMPS provides the `CorrelationID` of the previous message. * For **topics where AMPS creates messages**, such as `/AMPS/ClientStatus`, AMPS never provides a `CorrelationId`. Since AMPS doesn’t use the contents of the `CorrelationId` at all, there’s no good answer for how to fill those in. ## Keep it All Together That’s the story of `CorrelationId`. It’s a simple feature that does one thing, only one thing, and does it well. But what will you do with it? How do you use it to keep track of messages and get them where they’re going? That’s up to you. Because AMPS doesn’t change the `CorrelationId`, you can put whatever routing or processing information you need to in that field. The sky’s the limit1! [1] Within some common sense guidelines. The `CorrelationId` is part of the message headers sent to AMPS and is stored as part of a SOW record. The precise limits for those depend on your configuration and the data involved, but there's no special limit on a `CorrelationId`. --- # How Fast Can You Go? Built for speed AMPS is built from the ground up to go fast. AMPS tries to deliver messages at the fastest rate that an individual consumer can handle. AMPS has sophisticated machinery to try to find the fastest possible delivery rate for an individual consumer, and AMPS works hard1 to keep slower consumers from causing problems for faster consumers. These techniques apply to historical replay (bookmark subscriptions2) as well as subscriptions to current publishes. AMPS is designed to provide messages as fast as possible, regardless of the source. ### Why would anyone Ever Want to go Slow? _As fast as possible_ is always a good thing in our world here at 60East, but, what if you need to slow the pace of messages? And why would you ever want to do that? We recently worked with a developer who wanted to simulate real-time conditions for capacity planning and testing purposes. In the simulation, this developer wanted the ability to replay messages at any rate from the actual publish speed all the way up to the maximum throughput the client could handle. By controlling the replay speed, the simulation could exactly replicate activity peaks throughout the day, or run at 2x speed, or 4x speed, or anything up to the full speed of the system. As it turns out, this is simple to do with AMPS 4.0, a topic backed by a transaction log, and a bookmark subscription. The key ingredient is that AMPS 4.0 records the timestamp at processing time for each message and then makes that available when you replay messages. For regular subscriptions, the timestamp is close to the current time, but, for bookmark subscriptions, you get the original timestamp of when the message was processed. True-to-form, AMPS will continue to send messages as fast as your client can consume them. With a little bit of code, though, you can use the timestamps to replay messages to your client at any speed you like. Real time? Twice as fast? Four times as fast? Half speed? You've got full control. Here is a sample bookmark subscription using the Java client and the Command Interface (new to AMPS 4.0 clients!): ```java showLineNumbers public void subscribe(float pace, String bookmark) { try { Client client = new Client("Paced-Replay-Client"); client.connect(uri); client.logon(); // Use your own message handler SOWAndSubscribeMessageHandler ssh = new SOWAndSubscribeMessageHandler(); PacedBookMarkSubscribeHandler pacedh = new PacedBookMarkSubscribeHandler(pace, ssh); Command command = new Command("subscribe") .setTopic(topic) .setBookmark(bookmark) .setOptions("timestamp,oof"); client.executeAsync(command, pacedh); } catch(AMPSException e) { System.out.println("exception in Main: " + e.toString()); } } ``` This is a pretty typical subscription, which includes the Option of `timestamp`. This option is key as it will return a timestamp on published messages. We'll use that timestamp to pace the messages. To set the pace, we used a PacedBookMarkSubscribeHandler and pass our Message Handler of choice along with the desired replay pace. ```java showLineNumbers public class PacedBookMarkSubscribeHandler implements MessageHandler { final TimeZone gmt = TimeZone.getTimeZone("GMT"); SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd'T'HHmmss"); long previousTx = 0; float pace; MessageHandler wrappedHandler; public PacedBookMarkSubscribeHandler(float pace, MessageHandler h) { System.out.printf("Replaying at %fx of real time.\n", 1/pace); this.pace = pace; this.sdf.setTimeZone(gmt); this.wrappedHandler = h; } private long getDelta(String timestamp) { long delta = 0, currentTx = 0; try { currentTx = sdf.parse(timestamp).getTime(); } catch(ParseException e) { System.out.println("exception in getDelta: " + e.toString()); } if(previousTx > 0) delta =(long) (( currentTx - previousTx ) * pace); previousTx = currentTx; return delta; } public void invoke(Message m) { switch (m.getCommand()) { case Message.Command.SOW: wrappedHandler.invoke(m); break; case Message.Command.OOF: case Message.Command.Publish: try { Thread.sleep(getDelta(m.getTimestamp())); }catch(InterruptedException e) { System.out.println("Paced Handler Exception: " + e.toString()); } wrappedHandler.invoke(m); break; } } } ``` As you can see, when each message comes in, we use the timestamp on the message to determine the elapsed time between this message and the previous message, scaled by the speed of the replay. We sleep for that amount of time, and then process the message. ### Set the Pace You can use this simple technique to set the speed of a replay, and use the `PacedBookmarkSubscribeHandler` (or your own variation of it!) in place of an existing message handler. How fast can your application process messages? It's up to you. [1]See the [AMPS User Guide](/docs/amps-user-guide), where Slow Client Management is discussed in section 23.4. We're also working on a blog post to talk more about the techniques AMPS uses to make efficient use of resources -- [stay tuned](/newsletter)! [2]Bookmark subscribe allows you to begin a subscription at any point in the AMPS transaction log. It's one of the most commonly-used features of AMPS. For more information, see the [AMPS User Guide](/docs/amps-user-guide) chapter 18, or the documentation for your AMPS client library of choice. --- # No Filesystem? No Problem! Keeping State in AMPS Never lose track of your state AMPS makes a great platform for distributing messages to worker processes. The combination of low latency delivery, the SOW last value cache, message replay, and powerful content filtering make it easy to build a scalable grid of workers. **Update: This post describes an approach that was used with older versions of the AMPS clients. Current client versions include a recovery point adapter interface that can be used to store recovery points. Current clients also include an out-of-the-box recovery point adapter that can be used with a SOW topic.

    For current client versions, 60East recommends using a recovery point adapter, as described at [Bookmark State Without a Filesystem: Ultimate Director's Cut](/blog/sow-recovery-point-adapter), rather than the approach in this blog.**
    In this post, we show how to extend the AMPS client to provide a bookmark store for workers that don't maintain persistent state locally. The post assumes a good working knowledge of resumable subscriptions (covered in detail in the AMPS User Guide and the AMPS Java Developer Guide), and also assumes some familiarity with the implementations of the AMPS clients. Complete source code for this post, including an AMPS configuration file and a class that loads sample data into AMPS, is available [for download](/downloads/amps_bookmark_store.zip). ## Keeping State in AMPS To keep state in AMPS, we use the following three steps: 1. Load state from AMPS when the application starts 2. Periodically persist state to AMPS as the application performs work 3. Flush the saved state to AMPS when the application shuts down One way to use this technique is to maintain the point at which a particular worker should resume a subscription. To do this, we implement a BookmarkStore that uses AMPS for persistence. There are lots of other ways to use this technique, of course. [Yuck! Stateless Poison Message Handling](/blog/yuck-stateless-poison-message-handling/) shows a way to use the same technique to track messages that can't be processed by a worker. You can use variations on this technique to save the current state of a calculation or any other state that you need to track. ## About Stores The AMPS client libraries use _stores_ to provide reliable publication and resumable subscriptions. Stores, as the name suggests, are used by the client to maintain state. The stores preserve the current state of the client. Bookmark stores save the state of incoming subscriptions. Publish stores save outgoing messages. For each type of store, the client library provides a simple interface, allowing you to choose the specific store implementation you want to use, or to write your own. ## Bookmark Stores Bookmark stores provide the following major functions: * Add a bookmark to a subscription, indicating that a message has been received * Remove a bookmark from a subscription, indicating that the message has been processed * Return the most recent bookmark for a subscription, which is the point at which the subscription should resume With bookmark live subscriptions, bookmark stores have one additional responsibility: * Track which received messages have been persisted, and use those to help calculate the most recent bookmark The AMPS clients include two varieties of bookmark stores. _Memory-backed_ bookmark stores allow clients to resume after losing a connection to the server. In this case, the client loses state if the application that uses the client restarts. _File-backed_ bookmark stores allow clients to resume after restarting. Some applications, though, need to be able to resume subscriptions without maintaining local state. For example, a set of worker tasks running on a virtual machine that is periodically re-provisioned can't rely on persistent files to maintain state. Your application may want to present a consistent view of information whether the user is connecting from a desktop, a mobile phone, or through a website, without presenting messages that the user has already acted on. The AMPS SOW is one way to preserve state for a client without relying on having access to a filesystem, or being able to access any resources other than AMPS itself. In this blog post, we'll show you how to use AMPS as a bookmark store. There are a few constraints to consider for creating the bookmark store. The solution needs to minimize the number of messages and the overall amount of data published to AMPS. While publishing to AMPS is fast, each message sent to AMPS consumes bandwidth between AMPS and the client. That bandwidth is often the most constrained resource for the application, so we need to use as little as possible. Last, but not least, it's important to keep the solution simple, and use what's already provided in the client wherever possible. In addition, when a subscription uses the `live` option, the subscription receives messages that have not yet been persisted to the transaction log. This means that, if the server fails over, it is possible that the client has received messages that are not stored in the transaction log. In this case, AMPS periodically sends persisted acknowledgments on the subscription, which indicates the most recent point at which messages in the topic have been persisted. The bookmark store implementations provided with the AMPS clients track these acknowledgments, and the most recent method for those stores returns the latest persisted message rather than the latest discarded message. Using this strategy, the bookmark store guarantees that the client can restart from a valid message and will not miss messages even when using the `live` option. To meet these constraints, the AMPS bookmark store takes this approach: * Progress for the clients is stored in a SOW topic. This SOW topic need not be on the same server that publishes the messages. The SOW topic can be replicated, as well, to provide highly-available storage. * Rather than persisting the entire state of the store to AMPS, keep the store in local memory and persist only the bookmark value for MOST_RECENT. When persisting the value, indicate whether the subscription is receiving persisted acknowledgments or not. * Persist the most recent value to AMPS periodically, based on the number of messages processed for the subscription. This interval is configurable. * Derive from MemoryBookmarkStore to take advantage of the logic that's already written for duplicate handling, finding the correct value of MOST_RECENT, and maintaining quick in-memory access. The rest of this post describes the implementation in detail. ## Configuring The SOW The records that hold the progress will contain the clientName, the subscription ID for every subscription tracked by that client, the last bookmark the client has persisted, and whether the subscriber is receiving persisted acknowledgments. Represented as JSON, each message will look something like this: ```json showLineNumbers {"clientName":"resumableClient", "subId":"sample-replay-id", "bookmark":"10620414156524534001|3836|", "persisted":"false"} ``` Messages in the SOW are uniquely identified using the `clientName` and `subId` fields. Because each client can have multiple subscriptions, each processing different bookmarks, the SOW definition creates a compound key where each unique combination of clientName and subId is a unique message. We define the SOW topic as follows: ```xml showLineNumbers ./sow/%n.sow /ADMIN/bookmark_store json /clientName /subId ``` For this example, we use the JSON message type to make it easy to read the messages. This means that the instance that hosts the bookmark store needs to accept connections from clients that use the JSON message type. We use the `/ADMIN/` prefix as a way of indicating that this topic is used for application record keeping, and is not a topic that contains data. This is a convention to help with logging and troubleshooting, and is also intended to make it unlikely than any existing applications that use regex topic subscriptions will accidentally subscribe to this topic. However, the prefix has no meaning for AMPS itself, and the implementation could choose a different name. ## Working with the BookmarkStore Interface To get the results we need, there are three sets of methods we need to worry about on the BookmarkStore interface: * `log()` registers a message with the bookmark store to register the subscription and bookmark on the message. We don't need to override this, since we're planning to use the functionality that's already provided, but we'll call the `log()` function when we load the current state of the bookmarks from the SOW. * `discard()` marks a message as processed, and allows the bookmark store to discard the message. Our new AMPS-based bookmark store will override this method. We'll call the `discard()` method for the MemoryBookmark store and add code for persisting bookmarks to AMPS periodically. * `persisted()` is called when the client receives a persisted acknowledgment. This marks a message as persisted, and allows the bookmark store to discard the message. For our implementation, we track that the subscription is receiving persisted acknowledgment and then call the MemoryBookmarkStore implementation. For all of the other functionality of the bookmark store, the MemoryBookmarkStore does exactly what we need. ## Defining the BookmarkStore Class Next, it's time to define the class for the bookmark store and a constructor. The constructor for the class takes the client to use to manage the bookmark store -- which must already be connected -- and the name of the client to track subscriptions for. Notice that the client used to track subscriptions doesn't need to be connected to the same AMPS instance as the subscriptions being tracked. ```java showLineNumbers public AMPSBookmarkStore(Client bookmarkClient, String trackedClientName) throws AMPSException { if (bookmarkClient == null) { throw new AMPSException("Null client passed to bookmark store."); } _internalClient = bookmarkClient; _trackedName = trackedClientName; _bookmarkPattern = Pattern.compile("\"bookmark\" *: *\"([^\"]*)\""); _subIdPattern = Pattern.compile("\"subId\" *: *\"([^\"]+)\""); _persistedAckPattern = Pattern.compile("\"persisted\" *: *\"([^\"]+)\""); MessageStream ms = null; try { // Message to use for logging bookmarks Message logmsg = _internalClient.allocateMessage(); // Retrieve state for this client. ms = _internalClient.sow("/ADMIN/bookmark_store", "/clientName = '" + _trackedName + "'"); for (Message msg : ms) { if (msg.getCommand() != Message.Command.SOW ) { continue; } String data = msg.getData(); Matcher bookmarkMatch = _bookmarkPattern.matcher(data); if(! bookmarkMatch.find()) continue; Matcher subIdMatch = _subIdPattern.matcher(data); if (! subIdMatch.find()) continue; Matcher persistedAckMatch = _persistedAckPattern.matcher(data); if (! persistedAckMatch.find()) continue; // Get the bookmark string, the subId, and whether this subscription // receives persisted acks. String bookmark = bookmarkMatch.group(1); String subId = subIdMatch.group(1); String persistedAcks = persistedAckMatch.group(1); // Extract individual bookmarks from the record if necessary String[] bookmarks = new String[1]; if (bookmark.contains(",")) { bookmarks = bookmark.split(","); } else { bookmarks[0] = bookmark; } for (String b : bookmarks) { // Create a message with the subId and bookmark // to use for logging with the MemoryBookmarkStore. logmsg.reset(); logmsg.setSubId(subId); logmsg.setBookmark(bookmark); // Register the bookmark from the SOW as the // last successfully processed bookmark for this subId. if (! super.isDiscarded(logmsg)) { super.log(logmsg); super.discard(logmsg); } if (persistedAcks == "true") { super.persisted(logmsg.getSubIdRaw(), logmsg.getBookmarkRaw()); _persistedAcks.add(logmsg.getSubId()); } } _discardCounter.put(logmsg.getSubIdRaw(), 0); } } catch (AMPSException e) { System.err.println(e.getLocalizedMessage()); e.printStackTrace(System.err); throw e; } finally { if (ms != null) ms.close(); } // Start the worker to asynchronously handle updates _workerThread = new Thread( new UpdatePublisher(_internalClient, _trackedName, _workQueue), "Bookmark Update for " + _trackedName); _workerThread.start(); } ``` The constructor stores the client and the name of the client to track. The constructor then runs a SOW query on the topic that stores the persisted bookmarks and processes the results. The persisted message for the subscription contains either a single bookmark, or a comma-delimited set of bookmarks. If the message contains a list, the constructor processes the bookmarks one at a time. For each persisted bookmark, the constructor creates a message that contains the subscription ID and bookmark. The constructor logs the message, then immediately discards it. If the subscription has been receiving persisted acknowledgments, the constructor also logs an acknowledgment for the bookmark. This has the result of ensuring that the underlying MemoryBookmarkStore records the bookmark from the SOW query as the most recent bookmark for that subscription. The relevant lines are duplicated below: ```java showLineNumbers // last successfully processed bookmark for this subId. super.log(logmsg); super.discard(logmsg); if (persistedAcks == "true") { super.persisted(logmsg.getSubIdRaw(), logmsg.getBookmarkRaw()); _persistedAcks.add(logmsg.getSubId()); } ``` To keep the sample self-contained, this class uses the standard Java regular expressions utility to process messages. In production, we would replace the regular expressions with one of the commonly-used JSON-parsing classes for Java. ## Publishing Updates to the Store While publish operations are very efficient with AMPS, the class takes the standard approach of trying to do minimal work within a message handler. In this case, because the `discard()` methods may be called within a message handler, the AMPSBookmarkStore uses a simple BlockingQueue to deliver the subscription IDs and bookmarks to a worker thread. The worker thread dequeues each request, creates a message, and publishes the message to the SOW. ## Managing Discarded Messages Next, we override the `discard()` method of the MemoryBookmarkStore. In the overrides, we check to see if it's necessary to persist the bookmark to the SOW topic, and then call `discard()` on the `MemoryBookmarkStore`. ```java showLineNumbers @Override public void discard(Message m) throws AMPSException { super.discard(m); checkForPersist(m.getSubIdRaw()); } ``` The `checkForPersist()` method checks the number of messages discarded for the subId. If the number of messages is higher than the configured threshold, the method writes the most recent bookmark to SOW storage. ```java showLineNumbers private void checkForPersist(Field subId) throws AMPSException { Integer count = _discardCounter.get(subId); if (count == null) { count = 0; } ++count; if (count > _threshold) { try { _workQueue.put(new UpdateRecord(subId.copy(), getMostRecent(subId))); count = 0; } catch (InterruptedException e) { throw new AMPSException(e); } } _discardCounter.put(subId, count); } ``` The method checks the current count. If the count is above the configured threshold, the method enqueues an update and resets the counter. Otherwise, the method just increments the counter and returns. Notice that when the method enqueues an update, the method requests that the MemoryBookmarkStore provide the MostRecent value for the update. This ensures that the update contains the current recovery point. The overrides for the persisted methods are equally simple: ```java showLineNumbers @Override public void persisted(Field subId, BookmarkField bookmark) throws AMPSException { super.persisted(subId, bookmark); _persistedAcks.add(subId.toString()); } ``` In this case, we simply note that the subscription is getting persisted acknowledgments, and then call the MemoryBookmarkStore. Adding the subId to the set of persisted acks means that messages that note the most recent bookmark for this subscription will include the information that it gets persisted acknowledgments. The worker that processes the update simply blocks until an update is submitted to the queue, then creates and publishes an update message. ```java showLineNumbers while(true) { UpdateRecord update = _workQueue.take(); String msg = "{\"clientName\":\"" + _trackedName + "\"" + ",\"subId\":\"" + update.subId + "\"" + ",\"bookmark\":\"" + update.bookmark + "\"" + ",\"persisted\":" + "\"" + _persistedAcks.contains(update.subId.toString()) + "\"" + "}"; _internalClient.publish("/ADMIN/bookmark_store", msg); } ``` Finally, we implement two utility methods. The `close()` method iterates all of the subscriptions tracked by the store, and calls updateSubscription for each subscription to store the state of the subscription to the SOW. ```java showLineNumbers public void close() { for (Field subId : _discardCounter.keySet()) { try { _workQueue.put(new UpdateRecord(subId.copy(), getMostRecent(subId))); } catch (AMPSException e) { e.printStackTrace(); // Swallow exception: could also translate to unchecked // or log in the object. } catch (InterruptedException e) { // Unable to update publish store. Recover at last saved bookmark // instead. e.printStackTrace(); break; } } // Wait for the worker thread to drain the queue. while (!(_workQueue.size() == 0 && _workerThread.getState() == Thread.State.WAITING)) { Thread.yield(); } _workerThread.interrupt(); try { _workerThread.join(); } catch (InterruptedException e) { e.printStackTrace(); // trying to close, continue } _internalClient.close(); _internalClient = null; } ``` The `setPersistEvery()` method sets the number of messages to process for a subscription between snapshots of the bookmark into the SOW. ```java showLineNumbers public void setPersistEvery(int messageCount) { _threshold = messageCount; } ``` The rest of the bookmark store interface, and the internal logic for managing the in-memory store is provided by the existing MemoryBookMarkStore. ## Using the Store Using the store from an HAClient is simple. You construct the store with the client to use for persistence and the name of the client to track. For example, the code snippet below creates an AMPSBookmarkStore that uses an existing client: ```java showLineNumbers // control_client is connected to the AMPS instance that hosts the bookmark store HAClient client = new HAClient("Query-Client"); DefaultServerChooser sc = new DefaultServerChooser(); sc.add("tcp://amps-server:9007/amps"); // Server to subscribe to client.setServerChooser(sc); AMPSBookmarkStore store = new AMPSBookmarkStore(control_client, client.getName()); store.setPersistEvery(10); client.setBookmarkStore(store); client.connectAndLogon(); ``` Once the bookmark store is set, you use the client just as you would with any other bookmark store. For example, using the Command interface introduced with the 4.0 clients: ```java showLineNumbers BookmarkMessageHandler handler = new BookmarkMessageHandler(client); client.executeAsync( new Command("subscribe") .setTopic("messages-history") .setSubId(new CommandId("sample-replay-id")) .setBookmark(Client.Bookmarks.MOST_RECENT), handler); ``` Notice that, just as with any other resumable subscription, we set the subscription ID and we ask to start the subscription from the MOST_RECENT bookmark in the bookmark store. The provided sample also uses a bookmark message handler that exits after processing ten messages: this allows you to see how the bookmark store works for resuming after the last processed message. ## Next Steps: High Availability Bookmark Stores Because the topic that stores the bookmarks is just a normal SOW topic in AMPS, you can add high availability to the bookmark store without changing the AMPSBookmarkStore class itself. To do this, you'd make the following changes: 1. Add a transaction log for the `/ADMIN/bookmark_store` topic. 2. Replicate the topic to another AMPS instance. 3. Provide an HAClient as the control client for the `AMPSBookmarkStore`, with the two AMPS instances in the server chooser for the client. ## Wrapping it Up While this post has been a deep dive into using AMPS itself as a BookmarkStore, the techniques in the post apply to any sort of state your application may need. Just to review, the overall technique is: * Restore state from the SOW when the application starts * Periodically save state to the SOW as the application runs * Flush the saved state when the application exits Have a different pattern for maintaining application state in AMPS? Let us know in the comments! --- # Tip: Easy File Sharing File sharing: the first (and last?) great IT problem At 60East, we’re always helping teams analyze complex, interconnected software systems. One thing that’s easy to take for granted is file sharing – something that’s not always easy to do between teams in large enterprises. There are often great reasons for this, but for transmission of log files containing no sensitive information, it can be a real burden. The best way to share files is by using a shared host or file system, but if you have two teams that need to share files that don’t have any shared systems, then there are more efficient ways other than e-mail or file sharing services. Let’s look at two of my favorite alternatives __on Linux__ for sharing files within enterprise teams with no easy way to share files. In this example, a user on system "a.com" wants to share a file named "big.log" with another user on system "b.com". We’d never recommend sharing sensitive information using these techniques in environments where it’s forbidden. Know your restrictions and use these at your own risk. ## Method 1 The easiest way to share a file is for the file owner to open a simple HTTP server while the other fetches the file using a web browser or `wget`. __Open an HTTP Server on port 8080 on a.com, using Python:__ ```sh python -m SimpleHTTPServer 8080 ``` __Fetch the file from b.com (or from any WebBrowser):__ ```sh wget a.com:8080/symbol_store-20150315.tar.gz Connecting to a.com:8080... connected. HTTP request sent, awaiting response... 200 OK Length: 29627512439 (28G) [application/octet-stream] Saving to: “big.log” 1% [ ] 309,681,284 26.3M/s eta 18m 24s ``` __Wait for the file to download to complete. The user on b.com should now have the big.log file!__ ## Method 2 The second method uses _netcat_ (aka `nc`) and is great for file sharing Ninjas that may need to tweak for additional flexibility. The simple form of this method is to open netcat in listening mode on b.com and push the file to the other user from a.com. __On b.com, open the netcat listener on port 8080:__ ```sh nc -l 8080 > big.log ``` __On a.com, push the file over the network to the listening side:__ ```sh cat big.log | nc b.com 8080 ``` __Wait for the file transfer to complete. The user on b.com should now have the big.log file!__ ## Bonus: Add Inline Compression If your file is a big text file, then you could benefit from compressing the data before sending it across. Here’s how to edit the previous command to compress the bytes: __On b.com, open the netcat listener on port 8080:__ ```sh nc -l 8080 | gzip -d > big.log ``` __On a.com, push the file over the network to the listening side:__ ```sh cat big.log | gzip | nc b.com 8080 ``` ## Super Bonus: Add a Progress Bar Transferring large files can be frustrating when there’s no indication of how long it’s going to take. Plus, after the weather chit-chat, the progress indicators give introverted engineers something to talk about during the long pauses of silence when the bits are flying! :-) Let’s add some progress bars to our compressed netcat file transfers! __On b.com, open the netcat listener on port 8080, this is the same:__ ```sh nc -l 8080 | gzip -d > big.log ``` __On a.com, push the file over the network to the listening side (just replace “cat” with the “pv –tpreb” command):__ ```sh pv -tpreb big.log | gzip | nc b.com 8080 386MB 0:00:10 [27.9MB/s] [> ] 1% ETA 0:16:19 ``` ## Transfer At Will With these tricks in your toolbox, we hope you’ll never miss dinner again due to the hours spent trying to send a large file to a colleague sitting in the cubicle across from you. Do you have a great file transfer technique that gets you home on time? Let us know in the comments! --- # Fresh From the Lab: AMPS 4.3 File sharing: the first (and last?) great IT problem At 60East, we've been heads down in the lab, mixing up new ingredients to bring you the future of messaging today. We've just released AMPS 4.3.1.0. This release focuses on extending the futuristic capabilities we introduced in AMPS 4.0, making AMPS even better! Here are just a few of the new ingredients in AMPS 4.3.1.0: * **Composite message types** allow you to combine multiple types of data in a single message type. Among other uses, this feature lets you take advantage of the full flexibility and extensiblity of AMPS content filtering, while letting you control how AMPS parses a message. * **Hash indexing** for topics that maintain a State-of-the-World (SOW) database. You can now explicitly create hash indexes over SOW topics. AMPS hash indexes are a lightweight way to optimize common SOW queries. Hash indexes provide a significant performance boost for a query that uses them, with almost no degradation of publish performance. When a hash index is present, AMPS transparently uses the index for queries wherever possible. * **Explicitly keyed SOW** topics let you create SOW topics with an application-provided SOW key. This gives you the ability to create SOW topics over unparseable message types (such as unparsed `binary`) or to let the application define what makes a record unique, regardless of the data in the message. * **Replication Compression** can significantly reduce the bandwith required for AMPS replication. This is enabled with a simple configuration change on the publishing instance. * **Persistent Journal Indexing** can dramatically speed up recovery times for instances recovering large journal files. Rather than parsing each journal file to build an index of the messages in the file, AMPS now persists the index for the journal, reducing the recovery time. These are just some of the improvements available in this release. AMPS 4.3 is available for download now -- [give it a try](/evaluate)! Stay tuned for blog posts on these new features. To stay informed on what's new with 60East, [subscribe](/newsletter) to our newsletter and notification lists. --- # Composite Message Types: Answers Beyond Ints It's a brave new world out here! “I may not have gone where I intended to go, but I think I have ended up where I needed to be.” ― Douglas Adams, The Long Dark Tea-Time of the Soul In the Hitchhiker’s Guide to the Galaxy, Douglas Adams conveyed to us that the earth was actually a computer system designed to calculate the meaning of life. _SPOILER ALERT: the answer was 42._ Unfortunately, not all our answers come out as a concise short int format. Many of the simulation farms or grids that AMPS is deployed in produce vast amounts of data. This data is often an intermediate step in a workflow of calculations. In fact, we get asked about the movement of large data sets so often that we previously devoted an entire [blog post](/blog/easy-file-sharing/) to the topic. In this article, we are going to highlight how we can be flexible in our treatment of very large messages through the use of the new AMPS composite message types. ### Finding The Big Answers Many people use AMPS to efficiently route messages to their appropriate destinations based on powerful content filtering. With AMPS, it’s straight forward to navigate through complex data in a wide variety of formats including JSON, BSON, FIX, NVFIX, and XML. AMPS also offers an unparsed binary type (or BLOB) that lets us send any data we want, such as a serialized object, but trades off the ability to filter based on the content of the data. Also, by supporting unparsed binary, it not only helps avoid the costs of serialization and deserialization but allows AMPS to work the variety of similar formats found in customer deployments. We have also seen such binary formats used to "hide" content from AMPS, for example, large or deeply-nested XML documents that you don’t want parsed. When a message type is declared as binary, AMPS won’t attempt to parse it. When the messages become very large, people often consider some form of optimization or otherwise transform them into a binary format (i.e. sending a series of one byte chars as sequential bytes on the wire instead of a JSON array). This can make sense if the data can be optimized effectively and alleviates memory and network loads while not incurring too much of a latency hit. Another challenge that it introduces is that we often lose the capability to do content-based filtering or routing. If we already know where they are supposed to go, then we can create the appropriate topic (target) and send them. Unfortunately, the use of explicit topics makes for a more brittle system and incurs the costs of topic management as more and more applications and topics are added to the system. The answer? AMPS composite message types. ![The Answer](/img/blog/Deepthought-opaque.png) With most traditional pub-sub systems the work around for handling binary payloads and rigid topic hierarchies was to utilize custom headers as a vehicle to store meta-data which could be parsed by the routing agents. Now with AMPS composite message types, we enable developers to avoid such a traditional hack and embrace content filtering on regular message parts that are not limited to any particular size or structure. Unlike headers, composite message types can contain arbitrary data, and are fully filterable. Fortunately, if we desire to leverage optimized payloads as well as enabling critical content based filtering, we can employ composite message types. Akin to MIME, the payload can remain untouched while content filtering can be performed on the accessible parts or metadata. In many financial services applications, the actual bulk of the payload are doubles or floating point numbers, and the data that is useful for filtering and routing is metadata and a small subset of critical information. Ideally, we would send the metadata in a format AMPS can filter, while maintaining the bulk of the payload in an efficient binary format. No single message type meets both these needs, but the AMPS composite message type is ideal for this situation. We can store the information to be filtered on in a JSON part in a preprocessing step. We can combine that part with the BLOB payload to form a composite message type. We can treat it as a regular AMPS message type and filter/route it based on the JSON part of the message, all while maintaining the optimized payload until it is needed. Composite message types can be treated like any other message and can be leveraged to create a SOW, conflated topics, or even with delta subscriptions. In terms of setting your application up for using it, you will just need to update your configuration file to declare your composite JSON-Binary message type. One has to just name the part and declare its parts (message types). After that, we just have to bind it to a transport. ```xml showLineNumbers composite-json-binary composite-local json binary composite-json-binary-tcp tcp 9023 composite-json-binary amps ``` ### Searching the Whole or a Part AMPS provides two different ways to create an AMPS composite message, depending on how you want AMPS to parse the message. The `composite-local` option ensures that an XPath identifier can match any of the parts of the message, and that a filter can match a specific part of the message using the ordinal value of the message part. For example, `/0/mymessage=123.4` would test the json message’s field `mymessage`. The alternative option is `composite-global`, which combines all of the parts into a single set of XPaths. This lets you find values without having to know which part of the message contains the value. ### Build a Message The AMPS 4.3.1.0 clients include helper classes for building and parsing composite messages. To build a composite message, we just have to create the message and `append()` each part. Say we had a large set of doubles that represented the results from a scenario calculation. That would be the binary part of the message. We would then take any essential data from that message and, along with any other enrichment information, we would create a JSON message which would be the other part. ```cpp showLineNumbers std::ostringstream json_part; std::vector data; // …skipping the population of variables… // Create the payload for the composite message. AMPS::CompositeMessageBuilder builder; //insert the json part of the message builder.append(json_part.str()); // copy the array of doubles into second message part builder.append(reinterpret_cast(data.data()), data.size() * sizeof(double)); // Create publish the payload on the topic std::string topic("messages"); ampsClient.publish(topic.c_str(), topic.length(), builder.data(), builder.length()); ``` And that is all we have to do to create and publish a composite message. On the subscriber side, we instantiate a `CompositeMessageParser` to access the distinct parts of the composite message. ```cpp AMPS::CompositeMessageParser parser; ``` We then create our subscription and upon receipt, we can parse the message and obtain the distinct parts with `getPart()`. ```cpp showLineNumbers for (auto message : ampsClient.subscribe("messages")) { parser.parse(message); std::string json_part(parser.getPart(0)); AMPS::Field binary = parser.getPart(1); ... } ``` Of course we don’t have to be so explicit, we could first use the parser object to obtain a count of how many parts the message contained. ```cpp showLineNumbers std::cout << "Received message with " << parser.size() << " parts" << std::endl; ``` To make the binary payload usable, we convert it back into a vector. ```cpp showLineNumbers std::vector vec; double *array_start = (double*)binary.data(); double *array_end = array_start + (binary.len() / sizeof(double)); vec.insert(vec.end(), array_start, array_end); ``` In this blog, we looked at how composite messages provides great flexibility on how we can optimize our messages by having them encapsulate different message types. The primary use cases are to allow for payloads that do not need to be parsed by AMPS (i.e. large binaries, custom formats etc) as well as cases that could benefit from optimized or hidden data. With AMPS, having the metadata available in a supported type such as JSON affords us the added luxury of content filtering on that particular part – without having to parse the binary payload. In the future we will analyze more options that could be employed to improve transport optimizations and provide life to those systems still using FIX with Base64 encoded payloads. We can also discuss best practices around Protocol Buffers or Avro and/or leverage modern compression systems such as snappy. In our experience, the best choice of tools and strategies are highly dependent on the use case and characteristics such as system load, tolerance for latency, and burden of maintenance. Let us know how you think Composite Message types may help you and we can work through the ideas with you – just _“Don’t Panic” (Douglas Adams)_. ### Kudos I’d like to give some credit to Cory Isaacson, CTO of Risk Management Solutions, for prioritizing the issue of Very Large Messaging about a decade ago. He was looking at 10-20 Megabyte structured financial products (i.e. CMO, CDO) as well as HL7 and XML were being passed around and made us thoroughly explore distributed shared memory and cache, optimized parsers, and compression. There was no perfect general solution – every use case and customer had different tolerance levels to trading off latency and agility. Today the options and alternatives have improved, send us an [email](mailto:support@crankuptheamps.com) if this is something you want to discuss within the context of AMPS or otherwise. --- # How can a Pure Software Messaging Solution Whip the NIC off a Hardware Messaging Appliance? Fast cars, fast software, and beautiful scenery. Why is it that the latest Bugatti Veyron [doesn't have the fastest time at the Top Gear test track]( http://www.quora.com/Why-are-Formula-1-cars-slower-than-a-Hennessey-Venom-GT-or-a-Bugatti-Veyron )? How could a highly regulated or constrained **2004** Renault R24 Formula One outperform it? The Bugatti outperforms the F1 in many tests; but not on that prestigious Top Gear track due to the design trade-offs their designers made. We also don’t know if the driver of the Bugatti really was in tune with this vehicle. Martin Thompson provides an excellent metaphor from racing to software development in his blog [Mechanical Sympathy](http://mechanical-sympathy.blogspot.com/2011/07/why-mechanical-sympathy.html).
    "The name "Mechanical Sympathy" comes from the great racing driver Jackie Stewart, who was a 3 times world Formula 1 champion. He believed the best drivers had enough understanding of how a machine worked so they could work in harmony with it." -Martin Thompson
    In the software world, "Mechanical Sympathy" comes down to pursuing the principle of building software with an intimate knowledge of the underlying hardware. i.e. If you know your hardware platform, how that platform works, and where it may be advancing, you can optimize your software to realize higher efficacy today and be able to scale along with the advancements. In this post, we will investigate this principle further and answer the question about how AMPS running on best of breed commodity hardware can be 2.5 times faster in message processing and delivery throughput than a specialized hardware appliance. ## Losing the first Heat but Winning in Record Time in the Final: Admittedly, a while back, AMPS was in a similar situation and was outperformed on a test that was like a Top Gear track. The use case was quite a niche one that mostly relied on our state of the world cache and not a lot of our other capabilities. A specialized software vendor made us humble as their product outperformed what we thought was an analogy to a Bugatti, AMPS. We started the race by out-lapping our opponent as we were 30X faster on inserts however after a few more laps we noted we were significantly slower in the querying. The AMPS approach is optimized for frequently-updated and changing data. Most AMPS applications use subscriptions as a "continuous query" that delivers results as the row is inserted. For other applications, AMPS employs a sophisticated parallel divide and conquer approach to querying. This use-case was more of a 'write once, then distribute' model. The test provided an a-priori knowledge of keys and begged for an index based approach. Divide and conquer wasn't good enough in this case. We wanted to win! In a few days, we revamped AMPS' indexing model to add hash indexes, and we were able to outpace the competitor in terms of queries and still maintain the 30X insert advantage. To elaborate, indexing schemes in most database systems rightfully optimize reads at the expense of maintaining indices during writes (updating and possibly rebalancing B-Trees, etc). The result is slow writes/updates but faster reads -- a good tradeoff as most use cases are ‘write once, read many’. By adding hash indexes, AMPS included the ability to perform extremely fast lookup for this scenario while keeping inserts extremely efficient. In the last laps, AMPS was fueled with a combination of statically maintained indexes and on demand indexes which allowed AMPS to zoom past the checkered flag in record time. In summary, AMPS' Hash Index model allows one to very rapidly find data for queries that are fully covered by a hash index, as well as to take advantage of the divide and conquer on-demand indexing traditionally used by AMPS. Hash indexes brought us back to a Bugatti-level as it dramatically improve AMPS performance for query-heavy (NoSQL) scenarios. ## Hardware Messaging Appliances Have a Place Once in a while we see an analogy to a Formula 1 in the guise of a hardware appliance that claims superior throughput, predictable latency, and management simplicity. Their literature tells us there are scenarios where offloading or redirecting workloads to a messaging appliance makes sense, and if you are doing everything you need to do on the NIC, then a hardware messaging appliance is something that could be useful. By avoiding disk paging and CPU interrupts alone, they argue that they can predictably provide low latency with minimal interruption. It's fairly understandable that some people will still think that hardware vendors will always perform faster - it's a dedicated box with minimal contention and can have registers aligned to the actual message types in the use case. The software driving the hardware can be finely tuned and optimized to both the use case and the customized hardware. The vendor can optimize their TCP stack, employ zero-copy processing and hand off results without leaving the NIC - nothing could be better at receiving, processing and pushing out messages... right? The premium costs for dedicated appliances make sense only if they do something better (i.e. faster) than what best of breed commodity hardware/software can do and if they are more simple to manage and you can afford to scale out with more or advanced hardware. That is a lot of “Ifs”... perhaps the Forumula 1 cars should go race on their own special track. What if a software solution proves these assertions wrong? ![Dedicated hardware is fast - until you need to turn the other direction!](/img/blog/formula-1-out-of-control.png "Dedicated hardware is fast - until you need to turn the other direction!") *When your needs change, hardware appliances can make you feel like you’re spinning your wheels.* ## A View to A Trade So let’s take a real world example: a View Server system where front office equity traders are constantly changing the data they want to see. With even a dozen traders, let alone 100, this can be taxing on the server technology during peak loads when the server has to process incoming market streams as well as quickly serving data requests from the traders while potentially having to enrich or validate the structured and unstructured data. The hardware appliance was able to drive throughput to 1.6 million 1KB messages per second and the AMPS software (running on commodity hardware) was able to realize 4.5 million messages per second – only being bound by the memory, network and CPU. If we upgraded any part of the hardware, we would have hit better numbers – and in fact, when we switched from a 10GB network to 40GB we realized 6.5 million messages per second. To read more about this, look at our [‘Shock Absorber’ white paper](/blog/ultimate-shock-absorber/) or if you want particular fan-out details and CPU/Network saturation #s, please get in touch with [us](mailto:support@crankuptheamps.com). If you need to see it in action, even our evaluation offering on a VM will demonstrate its inherent high throughput capacities (though AMPS works even better on real hardware). ## Defending COTS is Hard Work! AMPS is implemented in regular programming languages and works best on best of breed Commercial Off The Shelf (COTS). It will even scale down to work on VM which is great for Dev-Ops testing and development. However when embracing COTS, one has to not assume your software is really at the whim of the compiler optimization, tuned VM, OS and hardware capacity. Much of the performance gain on COTS comes from understanding each of the levels of the system and spending much effort in measuring and fine-tuning at every stage of processing. ## Measuring Success: Many of our successful real world customers are also driven by measurements and give us throughput targets as well as latency tolerance goals that are indicated by minimum, median and maximum times. In our deployments and in our proof of concepts, we actually work towards predicting how well we would perform. The challenging part is calculating the processing step but we have been doing this long enough that we understand the dynamics and can intelligently inform our estimates for AMPS’ maximum throughput from incorporating the type and size of the memory scheme, CPU type and networking capacity etc. By ensuring highly concurrent processing, intelligent memory usage strategies, and a lot of common sense, we hit our predicted rates. When we hear that AMPS provided 2.5-4X more throughput in different cases... we aren’t ecstatic. Instead we try to reassess if the 2.5 X more throughput was a theoretical bound or if we were doing something wrong that prevented us from hitting that bound. It is due to this philosophy that AMPS has become deployed in some of the largest and successful high volume trading systems in the US, and AMPS scales with best of breed COTS. Given AMPS' effective use of resources, you can do much with one server and this reduces the need to scale AMPS out. Its so well behaved that we are often hosted in a multi-tenant server. If you upgrade CPUs, memory or network, AMPS scales to take advantage of the new capacity. It can scale down to run on a VM for development testing ops and up to the 400 CPUs as we need to set aggressive goals for ourselves as that has become what is expected of us by both clients and ourselves. ## The Secret Sauce In terms of the secret sauce, its simple, don’t follow all of Google’s C++ standards :), but we do implement amazingly fast XML, FIX and JSON etc parsers, exploit concurrency with lock free data structures and be built for the enterprise. The latter point, ‘the enterprise’ is vital as AMPS has a simple model for deployment, conjugation, and updates. It also has enterprise level high availability, monitoring, and admin tools. Due to the low cost point of AMPS, we have seen deployments in the thousands with minimal dev ops support requirements. Still not convinced? Our many-trick pony also has a rich set of our multi-language API data selection capabilities that ensures it only sends the data that needs to be sent. This reduces risk of broadcast storms or network contention and minimizes CPU waste due to over-subscriptions. Developers can leverage a state of the world data cache of all recent values, complex aggregations and calculations, conflated messages with data updates at an interval, out of focus notifications, and it comes with the greatest invention of all time - a transaction log which can be leveraged for message replay, failover, and to resume subscriptions at the exact point that a subscriber restarted. All of these capabilities are not implemented with the idea that we are going to be trading off throughput performance – it’s the opposite. They increase the holistic system performance and resilience. We are fortunate to say that we win a lot of races but we know we have to keep on being better and fully embrace the “Mechanical Sympathy” principle. The hardware messaging appliance did a great job serving 1.6 million messages per second and the simple yet finely tuned software vanquished all doubt by realizing 4.5 million 1KB messages per second in its ‘slowest case’…When we added the 40GB network, we hit over 6.5 million messages per second...A little COTS can go a long way! --- # The NBA of Data Science We love basketball: especially where data is involved!. I’ve been pondering for a while on how to showcase some replay functionality in our AMPS product in a way that’s general enough that everyone understands the concept, yet provides a solution metaphor that easily translates into other domains. Ideally, the data would be from some real-world system, where time-series, ordering, and content filtering could be useful (again, leveraging the features of the product I’m trying to explain.) One thing is for certain: whether you’re building a demo for your product or just trying to practice skills in big data, __YOU NEED DATA__. While Fisher’s Iris data set from the 1930’s is great for text book explanations of clustering, k-means, and some machine learning concepts, it doesn’t really exercise the cutting edge technologies that are designed for today’s large scale problem domains. Earlier this month, I came across one of the most fascinating datasets I’ve seen to date: The NBA Motion Tracking database. Since the 2013-2014 season, every NBA court has 6 cameras elevated above the playing field that samples the 10 players and ball (in 3 __DIMENSIONS__!) at 25 times per second. That’s HUGE! You have a list of distinct players, teams, shot locations, all packaged into a beautiful time series accessible data set. All game events combined are around 6 BILLION events from 2013 through 2015, including the standard games, playoffs, and all-star games. The ball metrics include the x, y, and radius, which tracks the ball in all 3 dimensions throughout the game. Without a doubt, knowing that the NBA is tracking all of this data and making it accessible[1] has made __BASKETBALL MY FAVORITE SPORT__ (sorry, Soccer!) Why would the NBA be doing this? Just look at the stats on [http://stats.nba.com](http://stats.nba.com), which track certain player performance metrics: speed, shot percentages, court coverage, etc. With this data, team owners could even do their own data science projects determining the rate at which certain players fatigue, becoming less effective. Or, imagine A/B testing combinations of players on your team to find the best lineup for the playoff games. For my purpose, I just downloaded a subset of the data and assembled a visualization where I used AMPS for high performance replay of the messages to my browser using the AMPS Javascript client API and then ran visualizations through D3.js. Given that the data is tracking physical objects over time, it makes for a fun visualization that requires little explanation. If you want to check it out, it’s here: [http://replay.demo.crankuptheamps.com](http://replay.demo.crankuptheamps.com). [follow the bouncing ball... if you can keep up with AMPS!](http://replay.demo.crankuptheamps.com) ![Demo Screenshot](/img/blog/Demo-Screenshot.png "Demo Screenshot") __Bottom Line__: If you’re searching for cool data to explore data science and/or visualization techniques, you should totally swing on over to [http://stats.nba.com](http://stats.nba.com) and take a gander – it’s so much fun. Don’t let the lack of documentation discourage you, there are some great resources available as people are making use of this dataset. Below are some resources to get you started[2] if you’re new to this. Plug: If you’re interested in the fastest message replay engine (which happens to support pub/sub, message queueing, SQL queries, historical queries, content filtering, aggregation and conflation… and so much MORE), then please check out AMPS at [http://www.crankuptheamps.com](/). Thanks! [1] “accessible” is used loosely here, since I couldn’t find any public API’s or documentation on what the data contained. Luckily, it’s large JSON objects, and they’re used from most of the pages on stats.nba.com, making it easy to understand what the data contains. [2] How to scrape the NBA data: [http://www.gregreda.com/2015/02/15/web-scraping-finding-the-api/](http://www.gregreda.com/2015/02/15/web-scraping-finding-the-api/) [3] For info on how to build (and get inspired by others!) visualizations, check out the [d3js.org](http://d3js.org) gallery. --- # Comparison: AMPS and RabbitMQ I'm late! The Advanced Message Processing System (AMPS) from 60East Technologies is used in production for thousands of enterprise messaging applications. These applications use AMPS because they have the most demanding throughput and latency requirements for publish/subscribe messaging. These applications also take advantage of the AMPS durable message storage, historical replay and audit, global replication and high availability, sophisticated aggregation and analytics, and more. The 5.0 release of AMPS adds durable message queues, built on the proven AMPS engine for the highest levels of reliability and performance. To illustrate the performance of AMPS queues, our engineers ran a head-to-head comparison against RabbitMQ, the most popular message queueing product in use today. This document captures the performance testing results and compares features and functionality. While AMPS supports multi-messaging paradigms and provides extensive capabilities beyond message queueing, this paper focuses on the features of AMPS that are most relevant to solutions benefiting from message queueing. A majority of the AMPS user base deploys AMPS into enterprises that cannot tolerate message loss or low performance. AMPS Queues are durable by default and are able to achieve 40x the throughput of RabbitMQ with better durability and message delivery guarantees.
    AMPS Queues are durable by default and are able to achieve 40x the throughput of RabbitMQ with better durability and message delivery guarantees.
    ## Performance Comparison In this comparison, durable queues were used in both AMPS and RabbitMQ. The AMPS test used the AMPS C++ client library, and the RabbitMQ test used the alanxz/rabbitmq-c library. For both products, the consumers were set to acknowledge consumption in batches of 80 messages. For RabbitMQ, 60East set the prefetch count for each subscriber to 100 messages in an attempt to achieve higher performance and reduce the time waiting for new messages to arrive. For AMPS, the maximum backlog of the subscription was set to 100 messages and the proportional distribution method was used in order to optimize for subscriber efficiency. This approach is the most compute-intensive distribution method for the server. Test results were run in the 60East Engineering Lab. Both tests were run on the same system, a Supermicro SYS 1028U TNR4T+ with 2 Intel Xeon E5-2690 v3 processors (@ 12 cores each) and 128GB of memory. All tests were run over the loopback interface, with both publishers and subscribers on the same system. Because AMPS is able to completely saturate the storage on most systems, our engineers ran the test persisting the queues to two different storage devices. The first test used a mid-range, enterprise-class, 7200RPM spinning drive for the queue persistence. ![HDD Publish Performance](/img/rabbit-amps-hdd-write.png) ![HDD Consume Performance](/img/rabbit-amps-hdd-read.png) For all message sizes, AMPS consistently outperforms RabbitMQ by factors ranging from 2X - 15X better for both publishers and subscribers. More interesting than that, though, is that while AMPS performs better with smaller messages -- as would be expected, since the storage device can handle more messages in the same amount of I/O bandwidth -- RabbitMQ provides consistent performance regardless of the message size, which indicates that the performance is a result of the RabbitMQ implementation rather than I/O constraints. Most production installations of AMPS are for applications that require the highest levels of throughput and lowest levels of latency. To test performance under these conditions, our engineers moved the persistence to an Intel DC P3700 NVMe PCIe solid state storage device. ![SSD Publish Performance](/img/rabbit-amps-ssd-write.png) ![SSD Consume Performance](/img/rabbit-amps-ssd-read.png) As in the hard disk scenario, AMPS performed significantly better, with results 24x - 40x better for publishers and 16x - 30x better for subscribers. The most surprising results were that RabbitMQ seemed to gain little or no performance benefit from higher-bandwidth I/O. Notice that, for all tests, RabbitMQ provides consistent performance, regardless of the amount of traffic being delivered to the drive. AMPS, in contrast, fully uses the I/O capacity of the drive, and scales with the usage of that capacity. There are two notable differences in performance. The first is the difference in magnitude. AMPS was faster in testing across the board. The second difference is the difference in device scalability. AMPS performance dramatically improved with a faster device, while RabbitMQ's low performance remained constant, providing no material benefit from the upgraded storage device.
    RabbitMQ's low performance remained constant, providing no material benefit from the upgraded storage device.
    This behavior is a result of the difference in persistence guarantees between the products. AMPS fully persists messages before providing them to subscribers, which means that AMPS performance scales proportional to the bandwidth of the storage device. In contrast, RabbitMQ uses a strategy that batches calls to fsync[1] -- which means that that RabbitMQ does not depend on I/O throughput and has constant performance regardless of the amount of bandwidth made available. This explains the difference in scalability between RabbitMQ and AMPS. But that's not the most striking result in the research. Why are the differences in performance so large? ## What's Going On? The difference is not a result of the test or the configuration. The difference is a result of how AMPS is engineered. AMPS was designed to achieve peak performance and scale on modern multi-core hardware. The software architecture consists of a super-pipelined processing engine that allows maximum concurrency to leverage every ounce of CPU to complete tasks. The core engine has been thoroughly NUMA optimized to ensure low jitter and effective use of memory bandwidth. Finally, the entire system has been coded with exacting attention to detail: every data structure, every cache line, every thread, every algorithm, every point where threads interact has been carefully considered to produce a finely tuned processing engine. The result is a system built for "forward scaling" and as new processors, memory technologies, storage devices, and networking devices come to market, the performance delta between RabbitMQ and AMPS will continue to increase. The differences don't stop at performance. The feature rich queues are a story unto themselves. ## Message Queues Reinvented ![Through the looking glass, to a better world of queueing.](/img/curiouser.png "Through the looking glass, to a better world of queueing.") AMPS takes a completely different approach to message queuing than RabbitMQ. With AMPS queues, all messages are recorded into the AMPS transaction log, a durable, high-performance, fully-queryable sequential message store engineered to support applications that require sustained throughput of millions of messages a second. However, a queue is only one of the ways the AMPS engine can distribute a message. In effect, each queue is an independent view of the messages within the AMPS transaction log. As messages are delivered and consumed, AMPS simply tracks the state of that message in the view. The producer of a message needs no knowledge of either the distribution model or the consumer of the message. New queues, new consumers, and new applications can easily be added with no changes to the producer. In addition to blazing-fast performance, AMPS provides other benefits: __Ultimate Routing Flexibility__ AMPS provides the ultimate flexibility in decoupling producers from consumers. The full power of AMPS topic-matching (including regular expressions in the topic) and content-filtering (a combination of XPath and SQL-92 which includes regular expressions, calculated expressions, and user-defined functions) is available for message routing. With AMPS queues, applications are no longer limited to topics, routing keys, or matching on simple key/value pairs to determine delivery. AMPS completely decouples publishers from the details of how the queues are arranged or the ways consumers will use messages. In fact, AMPS can populate queues from its transaction log, so messages can later appear in queues that did not exist at the time the message was published providing ultimate routing flexibility. __Multiple Distribution Models__ AMPS fully supports multiple distribution models, even for the same message. From a single publish, AMPS can provide the message to any number of pub/sub subscriptions, archive it for audit or back testing, aggregate it into a view, replicate it to a disaster recovery site and enqueue the message for distribution to workers in queues across multiple regions. The publisher does not need to know how AMPS or any of the consumers will use the message. __Aggregation and Complex Event Processing__ AMPS is content-aware and includes a sophisticated aggregation and event processing engine. AMPS allows the user to create a real-time aggregated view of the contents of the queue as messages are added to and consumed from the queue. With AMPS, the user's view of the messages in the queue isn't limited to queue depth or consumption rate. Aggregation provides true insight: for example, a monitoring application can track the total value of orders in the queue automatically, in real-time, and bring your key business metrics and risk into focus. __High-Performance Delivery and Fairness Models__ Message distribution in AMPS is optimized for high performance and processing efficiency. Traditional queues require consumers to poll the queue periodically, which adds latency and network overhead. AMPS works on a subscription model that provides messages to a consumer as soon as the message is available. Consumers can declare a _backlog_, which sets the maximum number of unacknowledged messages provided to a consumer at a given time. In addition, consumers can concurrently acknowledge and process messages. This _smart pipelining_ allows the consumer to run at full capacity without blocking to wait for the next message from the queue. Delivery fairness models allow the user to tune each individual queue for lowest overall latency, equal balance across processors, or most efficient use of client resources. Fairness is applied across all consumers for each message, and AMPS manages the consumer backlog according to the queue fairness model. Unlike systems that allow a consumer that requests a large batch size to starve other processors, AMPS distributes each message according to the fairness model. This helps ensure that work reaches each consumer, and prevents a single "greedy" consumer from consuming all of the available messages, thus slowing down the overall message processing system. Many existing queuing products present an interface that is similar to AMPS by polling in a background thread and allowing a client to retrieve multiple messages at the same time. AMPS provides the programming advantages of that model, while solving the underlying problems in a unique way that provides dramatically better performance. __Distributed Queuing and High Availability__ The AMPS high-availability features fully support distributed queueing using AMPS replication. Replication in AMPS is fault-tolerant and resilient even over WAN connections or if one of the instances is restarted. AMPS uses a unique, patent-pending method to manage message distribution and distributed queueing without partitioning. AMPS instances automatically use remote instances to absorb processing load when processing on a local instance is unable to keep up with the rate at which messages are enqueued. If a replicated instance goes offline and comes back online, AMPS replication automatically synchronizes messages between that instance and its replication peers. With AMPS, there's no need to trade-off between highly-scalable and highly-available. AMPS provides both in the same deployment. __Content-Based Entitlement Control__ The AMPS entitlement system is built from the ground up to be content-aware, for precise control over entitlements. The entitlement system provides the ability to grant permissions to both publishers and subscribers based both on the topic and the content of the message, using the full power of AMPS expressions (including custom functions) to specify the content to which a producer or consumer is entitled. ## Beyond Queueing AMPS is built from the ground up to be a complete platform for data-intensive applications that demand the highest levels of performance and throughput. In addition to message queues, AMPS provides other unique capabilities which make it easy to build high-performance applications: __State of the World Database__ AMPS state of the world databases provide quick access to the current value of each distinct message on a topic. For pub/sub systems, this capability provides a way for a subscriber to quickly retrieve current values. The state of the world database provides full query capability, and also offers optional key/value index for retrieval times that meet or exceed popular NoSQL key/value store databases. AMPS also provides the ability to atomically retrieve current state and enter a subscription for updates, ensuring no duplication or message loss. __Message Recording and Replay__ As mentioned earlier, the AMPS transaction log is fully queryable. Applications can start a subscription from any point in the transaction log. AMPS replays messages from the log, in the original order. Once replay is complete, the subscriber receives any new messages that arrive. AMPS allows subscribers to pause and resume replay, as well as set a maximum rate for the replay. AMPS provides even more advanced features such as Out of Focus notifications for detecting when a message is no longer relevant, incremental (delta) updates to messages, and more. If you're ready to explore the diverse features of the fastest messaging product, please try an evaluation of AMPS, available from [http://www.crankuptheamps.com/evaluate](/evaluate). 60East provides the full details of the test and the code we used at the github repository at [https://github.com/60East/comparison-rabbitmq](https://github.com/60East/comparison-rabbitmq). Just to keep things fun, we'll send a free Version 5.0 T-shirt to the first 50 people that download AMPS 5.0 and email the full version number to support@crankuptheamps.com with their mailing address and shirt size. [1] As described at [https://www.rabbitmq.com/confirms.html](https://www.rabbitmq.com/confirms.html), "The RabbitMQ message store persists messages to disk in batches after an interval (a few hundred milliseconds) to minimise the number of fsync(2) calls, or when a queue is idle. This means that under a constant load, latency for basic.ack can reach a few hundred milliseconds." Further, applications must be aware of this. After a restart or failover, when consuming messages from a durable queue "... the client could reasonably assume that the message will be delivered again. This is not the case: the restart has caused the broker to lose the message. In order to guarantee persistence, a client should use confirms." --- # AMPS 5.0: Finely-Tuned Messaging AMPS 5.0 is now available - image by mitch huang AMPS 5.0 is now available. This version of AMPS builds on the technology in previous releases to refine existing features and bring all new capability to AMPS. With this release, AMPS provides extremely [high performance](/blog/rabbitmq-comparison-to-amps/) __persistent message queues__. The message queues include a variety of fairness models, and include an innovative approach to message delivery that eliminates polling and keeps processors running at full capacity while there is work to do. Message queues work seamlessly with the AMPS entitlement system. Replication is queue-aware, and AMPS includes a patent-pending method for distributed queueing while maintaining the highest levels of performance and strict delivery guarantees. AMPS 5.0 also includes: * __Replication Validation__. To help troubleshoot replication configuration, AMPS now confirms that replication connections reach the expected instances and that those instances have the expected configuration. * __Google Protocol Buffers__. AMPS now provides full support for Google Protocol Buffers, including content filtering and delta messaging. * __Rate Control on Bookmark Replay Subscriptions__. When replaying messages from the transaction log, subscribers can now specify a maximum rate for bookmark replay. * __Pause and Resume on Bookmark Subscriptions__. Subscribers can now pause subscriptions and resume them at a later time. * __Improved Slow Client Protection__. Slow client protection now provides per-instance, per-transport, and per-client limits. Self-tuning defaults make configuration simpler and make it less likely for a large query to trigger slow client protection. * __Usability and Performance Improvements__. AMPS 5.0 includes many other improvements that both increase performance and make AMPS easier than ever to configure and use. AMPS 5.0 is available now. Download an [evaluation copy](/evaluate) today, and let us know how the new features work for your applications! --- # bFAT: Messaging with Extra Cheese bFat is like a thick, juicy hamburger for your messaging apps. image by Jeffrey Bary -- CC BY 2.0 Here at 60East, we're dedicated to messaging. All messaging. All the time. Day in and day out. And that's why we're thrilled to bring you the newest, best, hottest, juiciest new message format. We call it __BFat__. Check it out at [bfat.io](http://bfat.io). Would you like fries with that? --- # Keeping State in AMPS, Rebooted keep track of where your subscription is at with this simple bookmark store -- image by Simon Cocks -- CC BY 2.0 In this post, we will revisit the topic of extending the AMPS client to provide a bookmark store for stateless clients. This post is in response to requests for a simple stateless store that does not provide all of the functionality of the local stores, but instead just makes it possible for an application with very simple needs to pick up a subscription after failover. The implementation that we will discuss here is fairly limited, but should provide a starting point for less restrictive implementations. Before we begin, I would like to encourage you to read [No Filesystem? No Problem! Keeping State in AMPS](/blog/no_filesystem_no_problem_keeping_state_in_amps/) in order to gain a deeper understanding of bookmark stores. **Update: This post describes an approach that was used with older versions of the AMPS clients. Current client versions include a recovery point adapter interface that can be used to store recovery points. Current clients also include an out-of-the-box recovery point adapter that can be used with a SOW topic. For current client versions, 60East recommends using a recovery point adapter, as described at [Bookmark State Without a Filesystem: Ultimate Director's Cut](/blog/sow-recovery-point-adapter), rather than the approach in this blog.** ## Maintaining a Bookmark Store in AMPS This implementation of the bookmark store will do two things: 1. Load state from AMPS during subscription or re-subscription. 2. Update the bookmark store as messages are processed by the client. These two operations maintain the point at which a particular subscription should resume on recovery. As with the bookmark store in the previous post, we can avoid state on the client by updating a State-of-the-World(SOW) topic with the bookmark of a message when processed by a subscriber. It is worth noting that the instance that contains the store SOW topic does not need to be the instance that the subscription is placed on. ## Restrictions of this Implementation As mentioned before, the implementation discussed in this post will be quite limited. The following restrictions will apply to this store: * Bookmark Live subscriptions are not supported. * Messages are required to be discarded in the exact order that they are received. * Replicated topics should not be used with this store. * You must use this store with an HAClient. * The client provided to the bookmark store MUST NOT be the client placing the subscription. ## Bookmark Store Messages The messages that will be sent to the SOW bookmark store are of the following form: ```json showLineNumbers {"clientName":"trackedClient", "subId":"1", "bookmark":"13948331409633568391|1465596069899000013|"} ``` For this example we are using JSON, but any message type can be used. For a simple bookmark store, we only need these 3 pieces of information because AMPS enforces the following 3 rules: 1. The client name must be unique, as is the case anytime you are using a transaction log. 2. A subId must be unique to each client. The same subId can be used with different clients. If no subId is provided to AMPS, one is automatically assigned. 3. The bookmark corresponds to a unique message in the transaction log. The bookmark value that we record in the bookmark SOW represents the last message processed by the subscriber. A more complex bookmark store may require message fields to indicate if a message has been discarded, or if a persisted ack has been sent. Since we have designed this bookmark store for a very limited use case, we don't need to worry about that here. Now that we know what our messages will look like, it's time to configure AMPS! ## Configuring AMPS The SOW needs to be configured so that we only have one record per `clientName` and `subId` pair. We do this by making these our `Key` values. ```xml showLineNumbers /sample/bookmarkStore ./data/sow/bookmark.sow json /clientName /subId ``` Since we are using a JSON message type, the server must have a transport configured that is able to accept JSON messages. Note that just because we are using JSON messages for the bookmark store does not mean that the message type must be JSON for the bookmark subscription. ## Working with the Bookmark Store Interface Unlike the previous blog article, we will not be calling method implementations used in other stores. Instead, we will define each method in the `sow_bookmark_store` class. The following methods will be implemented: * `set_server_version(version)` Internally used to set the server version so that the store knows how to deal with persisted acks and calls to `get_most_recent(subid)`. * Though this method is required, we will not be using it because this implementation requires that messages be processed in-order. As such, acks will not be processed. * `get_most_recent(subid)` returns the most recent bookmark from the store. This bookmark should be used for (re-)subscriptions. * `is_discarded(message)` is called for each arriving message to determine if the application has already seen this bookmark. If it has, then the message should not be reprocessed. * Since we are requiring that messages are processed in-order, and this store does not provide any duplicate detection, this will always return False. * `log(message)` is used to log a bookmark into the store and return the bookmark sequence number. The bookmark sequence number is the internal location where the store recorded the bookmark for this message. * Since we will only ever have one record per clientName and subId pair, the sequence number does not matter. Thus, we will always return `1`. * `persisted(subid, bookmark)` marks all bookmarks up to the provided one as replicated to all replication destinations for the give subscription. * This is only used for Bookmark Live subscriptions and Replication. Since our sample bookmark store will support neither, we can leave this unimplemented. * `discard_message(message)` marks a message as seen by the subscriber. * `discard(subid, seqnumber)` is deprecated, so we will not be implementing it. ## Defining the sow_bookmark_store Class Now we will be implementing the class for the SOW bookmark store. The bulk of the work will be done when we initialize the class. The __init__ method will take 3 arguments in the following order: 1. `bookmark_client`: This is the client that will become our internal client for the bookmark store. It must be connect and logged on. 2. `topic`: The SOW topic defined in the config that will function as our bookmark store. 3. `tracked_client_name`: The name of the client whose bookmarks this store manages. Again, the `bookmark_client` does not need to be connected to to same AMPS server as the subscriptions being tracked. The client corresponding to the `tracked_client_name` must be an HAClient. ```python showLineNumbers class sow_bookmark_store(object): def __init__(self, bookmark_client, topic, tracked_client_name): """ Class for creating and managing a SOW bookmark store :param bookmark_client: The client that will become the bookmark store internal client :type bookmark_client: AMPS.HAClient :param topic: The SOW topic defined in the config that will be used for the bookmark store. :type topic: string :param tracked_client_name: The name of the client whos bookmarks we will be storing. :type tracked_client_name: string :raises AMPS.AMPSException: if the internal client fails to query the SOW. """ self._internalClient = bookmark_client self._trackedName = tracked_client_name self._topic = topic self._mostRecentBookmark = {} try: for message in self._internalClient.sow(self._topic, "/clientName = '%s'" % self._trackedName): if message.get_command() != 'sow': continue data = message.get_data() bookmark_data = json.loads(data) if 'bookmark' in bookmark_data and 'subId' in bookmark_data: self._mostRecentBookmark[bookmark_data['subId']] = bookmark_data['bookmark'] except AMPS.AMPSException as aex: raise AMPS.AMPSException("Error reading bookmark store", aex) ``` The `__init__` method is responsible for getting the most recent bookmark for all subIds corresponding to the tracked_client_name. This operation is performed in `__init__` as opposed to `get_most_recent(subid)` as a performance enhancement. Instead of issuing a SOW query for each `subId`, we can issue one SOW query and create a dictionary from the results. Doing the work in `__init__` also allows us to throw an exception if the `_internalClient` is not able to reach the server. ## Subscribing and Recovering Upon subscribing or recovering, `get_most_recent(subid)` will be called. This method is responsible for returning the last bookmark processed for the corresponding subscription identifier. ```python showLineNumbers def get_most_recent(self, subid): """ Returns the most recent bookmark from the store that ought to be used for (re-)subscriptions. :param subid: The id of the subscription to check. :type subid: string :returns: mostRecentBookmark[subid] or '0' """ # if we have a most recent value for that subId, then we'll return it # if not, we return EPOCH if subid in self._mostRecentBookmark: return self._mostRecentBookmark[subid] else: return '0' ``` This method will simply check the `_mostRecentBookmark` dictionary for the subid. If we find it, we will return the bookmark stored in the dictionary. If we do not find that key, then we assume that this is a brand new subscription. As such, we return `EPOCH`. ## Publishing to the Bookmark Store Since one of the requirements is that the message will be processed in the order that they are received, we can tell the bookmark store that a message was processed by the subscriber any time `discard(message)` is called. To mark a message as processed, all we need to do is publish the bookmark of that message to the store using the message format mentioned above. ```python showLineNumbers def discard_message(self, message): """ Mark a message as seen by the application. :param message: The message to mark as seen. :type message: AMPS.Message :raises AMPS.AMPSException: if the internal client cannot publish to the server. """ subid = message.get_sub_id() bookmark = message.get_bookmark() if bookmark is None or subid is None: return msg = '{"clientName": "%s", "subId": "%s", "bookmark": "%s"}' % (self._trackedName, subid, bookmark) try: self._internalClient.publish(self._topic, msg) self._mostRecentBookmark[subid] = bookmark except AMPS.AMPSException as aex: raise AMPS.AMPSException("Error updating bookmark store", aex) ``` The subscription identifier and bookmark for the message being passed to this method can be retrieved by calling `get_sub_id()` and `get_bookmark()`, respectively, on the message object. You will notice that this method is called `discard_message(message)` not `discard(message)`, but the client will still call `discard(message)`. Before publishing, we check that subid and bookmark are set, this is to prevent messages from entering our bookmark store that should not. For example, a subscriber calling discard on every message that it sees could update the bookmark store with a message that does not contain a bookmark: if we were to save that message to the store, we might store an empty or invalid bookmark and recover from the wrong point. ## The log Method For bookmark stores that will support out-of-order message processing, `log(message)` is responsible for assigning a sequence number to a bookmark, then publishing this information to the bookmark store. The method then returns the sequence number that it assigned to the message. ```python showLineNumbers def log(self, message): """ Log a bookmark to the store. :param message: The message to log in the store :type message: AMPS.Message :returns: The corresponding bookmark sequence number for this bookmark. """ # since we only ever have one SOW record per _trackedName and subId pair, this can # always return '1' return '1' ``` In this bookmark store, we are requiring that messages are processed in-order. Since this is the case, we need not need to assign a unique sequence number to each message. Instead, we can return '1' since there will be at most one message for each `_trackedName` and subscription identifier pair. ## Checking if a Message is Discarded In a bookmark store that supports out-of-order message processing, `is_discarded(message)` will return a boolean value indicating if a message has been discarded or not. During replay, AMPS checks if a message is discarded before delivering it. This is to prevent messages that have already be processed by the application from being delivered again. ```python showLineNumbers def is_discarded(self, message): """ Called for each arriving message to determine if the application has already seen this bookmark and should not be reprocessed. Returns 'true' if the bookmark should not be re-processed, false otherwise. :param message: The message to check :type message: AMPS.Message :returns: True or False """ # since messages are being processed in order, we never see a discarded message. return False ``` As mentioned before, the bookmark store that we are building is designed to process messages in the order that they are received. As such, we will never have a situation where a message that is discarded will be sent to a subscriber. This being the case, we can always return False. This concludes the methods that will need to be implemented. However there are 2 more methods that will need to exist. ## Unimplemented Methods The first of these methods is `set_server_version(version)`. This method is used to tell AMPS how to handle persisted acks. Based on the in-order processing of messages, we will not need to concern ourselves with acking. For this reason we can leave this method unimplemented. ```python showLineNumbers def set_server_version(self, version): """ Internally used to set the server version so the store knows how to deal with persisted acks and calls to get_most_recent(). :param version: The version of the server being used. :type version: int """ pass ``` The next method is `persisted(subid, bookmark)`. This method is used to mark all bookmarks prior to the provided one as persisted. Persisted acks are necessary for Bookmark Live subscriptions, but our implementation will not support this functionality. With that in mind, we do not need to concern ourselves with implementing this method. ```python showLineNumbers def persisted(self, subid, bookmark): """ Mark all bookmarks up to the provided one as replicated to all replication destinations for the given subscription. :param subid: The subscription Id to which to bookmark applies :type subid: string :param bookmark: The most recent bookmark replicated everywhere. :type bookmark: string """ # Bookmark Live and Replication are not supported, so this does nothing. pass ``` It is also worth noting here that there was previously an option to discard based on subscription identifier and sequence number. This method has been deprecated and should not be implemented. ## Using the Store To use the bookmark store we will need to follow these simple steps: 1. Create a client for the bookmark store to use. This client must be connected and logged on to the instance that contains the SOW topic for the bookmark store. This need not be the instance that contains the topic the application will subscribe to. 2. Construct an HAClient for your application to use. 3. Set the bookmark store for the HAClient. 4. Call discard(message) on each message when your application is done with it. ```python showLineNumbers class handler: def __init__(self, client): self._client = client def __call__(self, message): print message.get_data() self._client.discard(message) bkmrkchooser = AMPS.DefaultServerChooser() bkmrkchooser.add("tcp://localhost:9007/amps/json") bkmrkclient = AMPS.HAClient("bkmrk") bkmrkclient.set_server_chooser(bkmrkchooser) bkmrkclient.connect_and_logon() chooser = AMPS.DefaultServerChooser() chooser.add("tcp://localhost:9007/amps/json") haclient = AMPS.HAClient("haclient") haclient.set_bookmark_store(sow_bookmark_store(bkmrkclient, "/sample/bookmarkStore", "haclient")) haclient.set_server_chooser(chooser) haclient.connect_and_logon() haclient.bookmark_subscribe(handler(haclient), "orders", "recent") ``` Every time `haclient` processes a message via the `handler` class, `self._client.discard(message)` will be called. This will keep the bookmark store up-to-date with the messages processed by the subscriber. ## Closing Thoughts Most bookmark stores protect against duplicate message delivery. The store that we created in this post does not, and it can cause duplicate messages to be sent to your application on recovery. One way this can happen is when the AMPS instance containing the bookmark store becomes unavailable. This would result in the store implementation being unable to update the SOW that contains the bookmarks. If, at this point, the subscriber were to restart, the store would be one message behind. If more than one message has been processed before the subscriber restarts, the store would be further behind. With that in mind, if your application can tolerate duplicate messages, then this simple implementation should work for you! ## Get the code The bookmark store implementation, a configuration file that includes the SOW configuration, and a simple sample program can be downloaded [here](https://github.com/60East/amps-gems/tree/master/client/python/SOW-backed-bookmark-store). --- # Try AMPS NOW With Cloud Evaluation Beta Power of the CLOUD The Advanced Message Processing System (__AMPS__) from 60East Technologies is a state-of-the-art technology that powers up many of the Fortune 500 companies. Developers who tried __AMPS__ love it and use it in their products. But is there an easy way to test features of __AMPS__ without having access to a linux machine and installing __AMPS__ Server? There is NOW! We introduce a new quick and convenient way to evaluate __AMPS__: [Cloud Evaluation Dashboard](/evaluate). We’ll take care of the Linux server hosting let you start trying __AMPS__ in your products right away.
    [Cloud Evaluation Dashboard](/evaluate)
    Once you have signed up for the [Cloud Evaluation Dashboard](/evaluate) account and gotten an email from us containing the link to your dashboard, you are just 4 steps away from being amazed by AMPS features and its blazing-fast performance. ### Step 1: Download Client files Our first step is the Downloads page. Here you can select and download AMPS client libraries to use in your code: ![Downloads Page](/img/blog/cloud-eval-downloads.png "Downloads Page") We offer evaluation kits for the AMPS client libraries for the most popular programming languages: * C# * Java * C++ * Python and operating systems: * Windows * Linux Don't see your favorite combination of language and operating system in the prebuilt evaluation kits? Visit our [Developer Center](/develop) for the full selection of clients! You have your client files downloaded? Don't worry about unzipping and setting them up yet. Let's proceed to the next step! ### Step 2: Turn ON your __AMPS__ Instance Your personal cloud instance of AMPS Server is waiting for you! Hit the Turn ON button and let us do all the installation and configuration of the AMPS Server for you. It can take a few minutes for the virtual machine to be provisioned and start running, so be patient. You're almost ready to crank it up! You can even browse the dashboard pages while it's starting. Within a few minutes the instance will be up and running: ![Instance Controls](/img/blog/cloud-eval-instance-controls.png "Instance Controls") ![Instance: Time Left](/img/blog/cloud-eval-instance-timeleft.png "Instance: Time Left") This page also serves as the control center of your cloud experience. From here you can start, stop, or reset your instance, and see different statistics about performance and hardware. This is your personal instance of AMPS in the cloud -- this instance is not shared with others. ![Instance: Stats](/img/blog/cloud-eval-instance-stats.png "Instance: Stats") ![Instance: Hardware](/img/blog/cloud-eval-instance-hardware.png "Instance: Hardware") The instance is ready? Let's move to the next step! ### Step 3: Configure your Client Libraries Once you have the AMPS Server instance up and running, you will have all the pieces required to set up the client libraries downloaded at the Step 1. Navigate to the Configure page in order to get personal instructions on how to setup AMPS client libraries according to your preferred language. The IP of your cloud instance is already in the instructions, so there's no need to go back and forth and copy-and-paste it: ![Configure Client](/img/blog/cloud-eval-instance-configure.png "Configure Client") However, since the instance changes its IP address every time it starts/stops, you need to update IP address in the client settings. The Configure page will always contain up-to-date settings. ![Configure Client: Repeat](/img/blog/cloud-eval-instance-configure-repeat.png "Configure Client: Repeat") At this point you are ready to start playing with AMPS! Proceed to the next step! ### Step 4: Get a Taste of __AMPS__ features We've prepared simple and clear examples of a few of the main features of __AMPS__: ![Try AMPS: Functionality](/img/blog/cloud-eval-instance-functionality.png "Try AMPS: Functionality") On these pages you'll find quick introduction to a feature and code examples that are ready to be copied and pasted in the code: ![Try AMPS: Code Example](/img/blog/cloud-eval-instance-code-block.png "Try AMPS: Code Example") Code examples are available in all supported languages and are tailored to work with your __AMPS__ instance and update their content automatically. ### Further Steps After you've gotten a sample of AMPS with the Cloud AMPS Server, the next step is to try AMPS on your own hardware. The table below demonstrates main differences between the Cloud and the Local Evaluation modes: | Cloud Instance | Local Instance | | --- | --- | | Pros: Easy to start Fully configured Runs on our servers Works well with the code examples | Pros: No limits of use within the evaluation period (14 days) Instance location controlled by you Runs on your hardware Can have any configuration, including replication | | Cons: Sessions are limited to 3 hours IP address is dynamic Can't change AMPS configuration Hardware is limited Network latency to the cloud | Cons: Linux server is required Writing an AMPS configuration file requires additional knowledge/support Hardware and system maintenance | Want to know more about how __AMPS__ can help you build great applications? Still having trouble getting __AMPS__ to play your tune? For help or questions, send us a note at [support@crankuptheamps.com](mailto:support@crankuptheamps.com). --- # AMPS 5.2: More Power AMPS 5.2 is now available AMPS 5.2 is now available. This release of AMPS includes new features designed to help manage extremely complex, high-volume data flows that require data transformation in the AMPS server — while maintaining the performance and ease of use that AMPS is known for. The release also includes a set of features designed to make AMPS easier to configure and administer, and a wide variety of other usability and performance improvements. The new functionality in AMPS 5.2 includes: * __Inline Message Enrichment__. AMPS can now modify and enrich messages as they are published, without requiring that a separate view be configured. When a message is enriched, AMPS persists the enriched version of the message into the transaction log and topic SOW file. * __Aggregated Subscriptions__. A subscriber can now request that aggregation and analytics occur for an individual subscription, without requiring that a view be defined in the AMPS configuration file. * __Conflated Subscriptions__. A subscriber can now request that conflation occur for an individual subscription, without requiring that a conflated topic be defined in the AMPS configuration file. * __Monitoring Interface__. AMPS includes an enhanced monitoring interface, the _Galvanometer_, that displays information about a set of instances in an easy-to-understand graphical format. * __SSL Support__. The AMPS server now fully supports SSL/TLS connections. * __Order Chaining__. AMPS now includes the ability to update SOW records with chained, or hierarchical, SOW keys. This is most commonly used with FIX order chaining, but can also be used any time a hierarchical document structure should update a single record in the SOW. * __Expanded Set of Functions__. This release of AMPS includes more functions for working with data. Expanded functions include string construction functions such as `CONCAT()`, numeric functions such as `ROUND()`, and aggregation functions such as `STDDEV_POP()` and `STDDEV_SAMP()`. * __Configuration File Simplification__. AMPS now supports the ability to easily compose configuration files from a library of common definitions with the new `Include` configuration file directive. * __File System Threshold Actions__. The AMPS action set now includes the ability to conditionally run actions based on the capacity of the file system, to provide finer-grained control for instances with limited filesystem capacity. * __Local Queues__. AMPS 5.2 adds the ability to explicitly declare that a queue is maintained only on the local instance, even in cases where the data in the underlying topics for the queue is replicated. * __Performance Improvements__. AMPS 5.2 includes many other improvements that both increase performance and make AMPS easier than ever to configure and use. AMPS 5.2 is available now. Download it, or start an [evaluation](/evaluate) today, and let us know how the new features work for your applications! --- # Get more AMPS with Galvanometer Galvanometer: Get MORE AMPS! Every system needs control, and **AMPS** is no exception. We already have a pretty powerful and flexible **Admin** module that provides various information about the **AMPS** instance and the host system, in several formats such as **XML**, **JSON**, **CSV**, and plain text. It is very convenient for applications, scripts, services... but what about _humans_? We, the people, prefer information processed and visualized. Among many other cool features that **AMPS 5.2** introduced, this one is literally very easy to notice. Ladies and gentlemen, allow me to introduce the new admin interface: **Galvanometer**! ### What is Galvanometer? **Galvanometer** is the new graphical **AMPS** admin interface. It is included and enabled by default in all **AMPS** instances starting from **5.2**. It doesn't require any additional installation steps, simply go to the admin address and it's already there! Don't worry, the classic admin module is there as well, and will always be. Some of the features of **Galvanometer**: - Live Instance stats, such as Messaging, Clients, Lifetime, and so forth; - Live Host stats, such as Memory, CPU utilization, Networking, Storage; - Time Machine; - Built-in SOW/Subscribe functionality (special thanks to our new **JavaScript** client); - Details about Views, Topics, SOW; - Replication monitoring; - Transaction Log monitoring; - Graph Builder. Are you excited to take a look? Let's do it right now! ### Know your Instance **Instance** is the main page of **Galvanometer**. It visualizes and updates in real time the most important information about a running **AMPS** instance: ![Galvanometer Instance Page](/img/blog/instance.png "Galvanometer Instance Page") On this page you can find: - The **Lifetime** widget, a bar that represents the life span of the instance, including shutdown periods, occurred minidumps, etc; - The **Messaging** widget that visualizes the messaging stream of the instance. In real time you can see how many messages are being processed by **AMPS**; - The **Clients** table with the list of connected clients and their status and activity. If you click on a listed client you will be able to see its contribution to the total messaging stream on the **Messaging** widget. Double click or a click on the `Details` link will provide full information about the client, including its subscriptions, messaging, etc: ![Galvanometer: Client Details](/img/blog/client_details.png "Galvanometer: Client Details") ### Host: Hardware Monitoring It is important to know not only how and what the **AMPS** instance is doing, but its **Host** as well. **Galvanometer** has a dedicated page to monitor the host, including it's CPU cores utilization, memory, storage and networking: ![Galvanometer: Host Page](/img/blog/host.png "Galvanometer: Host Page") All these widgets refresh automatically in real time. ### Great Scott! Time Machine! Ever wonder how many messages **AMPS** was processing 30 minutes ago? Do you want to see how the CPU was utilized during the holiday season? All it takes is to travel back in time and see with your own eyes... _But it is impossible without a time machine_, you might say. Not for us! Our engineers hacked time itself and now we proudly present a working **Time Machine** that is shipped with every **Galvanometer**: ![Galvanometer: Time Machine Widget](/img/blog/time_machine_widget.png "Galvanometer: Time Machine Widget") Select the date and time you want to travel to and hit `Go!`. Time travel won't take long and soon you'll see this the same page, but in the past. It's even paused so you can take your time and look around. Once you're adjusted to the past, hit the `Play` button again to resume the time flow. Be careful: you're still in past! ![Galvanometer: Time Machine Mode](/img/blog/time_machine.png "Galvanometer: Time Machine Mode") Currently, our time machine only works with **AMPS** and **Galvanometer**, stay tuned for updates! ### SQL: SOW and Subscribe without leaving Galvanometer Wouldn't it be nice to be able to subscribe to regular topics and query SOW topics and see the results right away without leaving **Galvanometer**? We thought so too, and that's why every **Galvanometer** contains a built-in **AMPS** client, allowing users to get real time information about the data flow: ![Galvanometer: Host Page](/img/blog/sql.png "Galvanometer: Host Page") This uses our new **JavaScript Client**: stay tuned for more details in a future blog post! ### Replication: See the Big Picture **Replication** allows to build a distributed yet synchronized system of AMPS instances. Now it is very easy to visualize your replication fabric and get information about data flow between replicated instances! **Galvanometer** provides three ways of representing replication: - Chorded replication Graph; - Replication Matrix Table; - Force-directed replication Graph. ![Chorded Graph](/img/blog/replication_graph.png "Galvanometer: Chorded Graph") *Chorded Graph* ![Replication Matrix](/img/blog/replication_matrix.png "Galvanometer: Matrix") *Replication Matrix* ![Galvanometer: Force-directed Graph](/img/blog/force_directed_graph.png "Galvanometer: Force-directed Graph") Force-directed Graph ### Fine-grained Stats with the Graph Builder **Graph Builder** is a nice tool that allows you to watch and monitor a particular metric through time. Let's say you want to see how **AMPS** was consuming memory for last twenty minutes, or during peak hours last Thursday. It's never been easier - just search for a metric, pick one from the dropdown menu, select a time range, and voilà: ![Galvanometer: Graph Builder](/img/blog/graph_builder.png "Galvanometer: Graph Builder") The generated graph can be saved as an image or a **PDF** file. ### Try it today! **Galvanometer** is available and enabled by default in the newly released **AMPS 5.2**. It has a metric ton of cool features — give it a try! Please share your feedback and comments with us. Let us know how the **Galvanometer** is making your life easier :) ![AMPS](/img/blog/cloud-eval-instance-amps.png "AMPS") --- # Introducing the AMPS JavaScript Client minified JavaScript client code **AMPS** is a very robust system due to its amazing performance, flexibility, and reliability. 60East already provides client libraries for **C/C++**, **C#**, **Java**, and **Python**, and today we introduce the first version of our official **JavaScript** client that will power up both modern front end web applications and **Node.js**-backed back end applications. Features available in this version of the **JavaScript** client: - Minimal external dependencies (**WebSocket** and **Promise**) - A fully asynchronous **Promise**-based interface - Convenience methods for common commands, such as **publish**, **subscribe**, **sow**, **deltaSubscribe** - A **heartbeat** mode for quickly detecting connection failures - A **Command** interface for fine-grained control over commands and their options - Support of enterprise authentication systems, such as **Kerberos**, **NTLM**, and **Basic Auth** - Support of **Authenticators** for custom authentication scenarios Let's take a closer look and see it in action! ### Easy to Get, Easy to Use The client is designed and built to have as few dependencies as possible. It is compatible with both Browser and Node.js environments. For example, in order to use it in the browser environment, here's all it takes to get a working **AMPS** client in your web application: ```html showLineNumbers <-- Optional import support for obsolete browsers --> ``` The distribution already contains **ES6-Promise** polyfill to support obsolete browsers. ### Born Asynchronous to Live Free (From Blocking) The client adopts the **JavaScript** style and design approach in order to deliver the most intuitive, fully asynchronous interface for JavaScript developers. The client is very easy to use due to heavy use of the **Promise** feature. **Promises** are objects that encapsulate values that may be available now, in the future, or never. In the asynchronous world of JavaScript this is a convenient way of structuring and organizing actions that may take an uncertain amount of time to execute. To demonstrate the idea of the **Promise**, check out how the client connects to the **AMPS** server: ```javascript showLineNumbers let client = new amps.Client() client .connect('wss://fortune500company.com:9100/amps/json') .then(() => client.publish('test-topic', {id: 7})) .catch(err => console.error('Connection Error: ', err)) ```
    In code examples, we use the new JavaScript ES6 syntax
    Even though the above code is executed in the non-blocking asynchronous mode, the syntax is very compact and resembles the traditional synchronous style, thanks to **Promises**! ### Get Used to Convenience The client includes a full set of convenience methods that make common tasks easier to do. Here's how we can subscribe to topics, and then publish to them using `subscribe()` and `publish()`: ```javascript showLineNumbers let onMessage = (message) => console.log(message.data) let client = new amps.Client() client .connect('wss://fortune500company.com:9100/amps/json') // connected, subscribe for the first topic .then(() => client.subscribe(onMessage, 'orders', '/qty > 0')) // second subscription .then(() => client.subscribe(onMessage, 'reservations')) // third subscription .then(() => client.subscribe(onMessage, 'notifications')) // now we can publish messages to these topics .then(() => { client.publish('notifications', {note: 'Ordered Tesla 3'}) client.publish('orders', {order: 'Tesla 3', qty: 10}) client.publish('reservations', {res: 'Tesla 3', qty: 10}) }) // if any subscription failed, the chain will end up here .catch(console.error) ``` In the example above we subscribe to three different topics, and make the subscriptions sequentially. That is, we're waiting for the first subscription to be processed, then the second and finally the third. However, we can make our application a bit faster by parallelizing this process: ```javascript showLineNumbers client .connect('wss://fortune500company.com:9100/amps/json') // connected, subscribe to topics in parallel .then(() => Promise.all([ client.subscribe(onMessage, 'orders', '/qty > 0'), client.subscribe(onMessage, 'reservations'), client.subscribe(onMessage, 'notifications') ])) // subscribed to all topics at this point .then(subscriptionIds => { console.log('ids of subscriptions: ', subscriptionIds) client.publish('notifications', {note: 'Ordered Tesla 3'}) client.publish('orders', {order: 'Tesla 3', qty: 10}) client.publish('reservations', {res: 'Tesla 3', qty: 10}) }) ``` All convenience methods return a **Promise** object, which allows us to chain and parallelize them. The `publish()` method is an exception, since `publish()` does not wait for confirmation of processing from the server by default. (It is possible to confirm the publish by making a custom **Command** object.) ### I Measure the Moment in the Heartbeats I Skip The heartbeat feature is a quick and reliable way of detecting connection failures. It's simple to set up, and convenient to use: ```javascript showLineNumbers let onError = err => console.log('Heartbeat: ', err) let client = new amps.Client() .heartbeat(5) // 5 seconds between beats .errorHandler(onError) ``` In the above example the **AMPS** server will publish periodic heartbeat messages every 5 seconds to the client and will expect the client to respond with a heartbeat message. If the client does not provide a heartbeat within the time specified, the server logs an error and disconnects the connection. The client will report the heartbeat error to its dedicated error handler. The `heartbeat()` command can be used again in order to refresh the timer and/or change the heartbeat period. ### Command Interface: You Have Full Control The client includes a low-level interface for constructing **AMPS** commands, the **Command** interface. Compared to the convenience methods, Commands have the full range of options and controls to set: - command name - message handler - options, such as `ackType`, `orderBy`, `bookmark` and many more - flags (a comma separated list of values available to a command) Here's the example of a publish command that confirms the processing of a published message by the server: ```javascript showLineNumbers let client = new amps.Client() client .connect('wss://fortune500company.com:9100/amps/json') // connected, let's publish a message with confirmation .then(() => { let publishCommand = new amps.Command('publish') .topic('orders') .data({order: 'Tesla 3', qty: 10}) .ackType('persisted') return client.execute( publishCommand, ack => { console.log('message persisted: ', ack) client.disconnect() }) }) // connection or command execution error .catch(console.error) ``` ### Try it Today! The new JavaScript client is available starting today for all our customers. Get the latest version from the Downloads page. The API reference and the quick-start guide are available [here](https://devnull.crankuptheamps.com/documentation/api/js/5.3.4.0/api_reference) ### It's Not What JavaScript Can Do for AMPS, It's What AMPS Can Do for JavaScript Behind the scenes, AMPS offers rich features to create the most flexible, scalable and responsive view servers. By combining database, messaging and aggregation technology into one integrated engine, one can reduce data movement and exploit concurrency. More importantly, this design allows us to offer extremely valuable capabilities. For example, imagine being able to populate your GUIs with a query to the AMPS State of the World database and subscribing to the real time message flow in one single atomic step! Here are just some of the other ways that AMPS amplifies your JavaScript systems: - The power of **Content Filtering**. AMPS allows you to flexibly reduce the message flow to clients based on the content of each message - mitigating network congestion when scaling up to thousands of users or more and reducing the processing burden on each client at the same time; - With **Delta Messaging**, your applications can publish or subscribe to only the parts of a message with changed data; - With **Out of Focus Notification**, applications receive notification when a previously-received message is no longer relevant to a subscription; - With **Aggregations with Joins**, AMPS offers real time aggregation of streams of data, including the ability to join and aggregate streams of different message types; - A powerful **Replay Engine** for back testing and rapid recovery. Please share your feedback and comments with us. We also want to express our gratitude for those customers who worked with us while the JavaScript client was in beta testing stages. _Who knew that there were that many browsers to test on???_ :) > Want to see the JavaScript client in action *now*, right from your browser? Check out the [Basketball Replay Demo](http://replay.demo.crankuptheamps.com) and the [related blog post](/blog/nba-of-data-science) --- # Time-Based Triggers robot maid Imagine, if you will, the following scenario. You have an AMPS SOW topic representing the state of orders in your business. You have a well tuned AMPS configuration, you have optimized your client applications and your network is at optimal capacity. You are processing millions of messages and everybody seems happy. But you have discovered a nefarious problem. A consuming client is behind and there are orders that are not getting processed! Orders come into AMPS, but never leave the "Pending" state. Seconds go by and they still don't update. They are stuck, like that last bit of jam that won't come out of the jar. You don't know which client is the culprit and you can't keep track of which messages are being left behind! What do you do? Hire a nanny to sit and watch your orders? Of course not! You build a robot nanny! AMPS to the rescue! ![Robot Nanny](/img/blog/robot_maid_full.png "AMPS to the rescue!") ## Action On Message Condition Timeout Let me introduce you to one of the new, simple but powerful, actions included in AMPS 5.2. *Action on Message Condition Timeout* allows AMPS to run an action when a message in a SOW topic meets a specific condition for longer than a specified period of time. This module uses the Out-of-Focus notification (OOF) mechanism to do its magic. When a message matches the specified topic and filter, the module begins tracking that message. If no OOF notification is received for that message within the specified timeout, the action runs for that message. ### Back to our story, how do you save the day? You use this action to configure amps to send an alert for messages in your orders SOW topic with a state of "Pending" and a duration of 5 seconds. *Any message that still has a state of "Pending" after 5 seconds will trigger the action.* Configure the action to publish a message to an "Alerts" topic, set up a "nanny-bot" to listen on the Alerts topic, and then boom! Send an email, buzz your pager, turn on your toaster, or fire the death ray! You name it! All at AMPS speed! ## Working Example Here are the steps to a basic working example that you can run on your own AMPS instance. *Note: This example assumes that you are running the AMPS server on your local machine using the loopback address. If you are running your AMPS server on a different machine or network configuration, the example may not work as written* 1\. Copy the following AMPS server configuration and save it to a file called `sample-on-message-condition-timeout-config.xml` inside of your amps installation directory. *Note: This is a bare bones demo configuration. Use for anything else at your own risk!* ```xml showLineNumbers <-- Name of the AMPS instance --> AMPS-On-Message-Condition-Timeout-Demo <-- Configure the administrative HTTP server on port 8085 This HTTP server provides admin functions and statistics for the instance --> localhost:8085 <-- Configure a transport that accepts any known message type over TCP port 9007 using the amps protocol. --> any-tcp tcp 9007 amps stdout error 00-0015 Orders json /OrderID transient Alerts json /OrderID transient amps-action-on-message-condition-timeout json Orders /status = 'PENDING' 5s amps-action-do-publish-message json Alerts {{ "{{AMPS_DATA" }}}} ``` 2\. Open a terminal and navigate to your amps installation directory. 3\. Start amps using the configuration file that you just created: ```bash bin/ampServer sample-on-message-condition-timeout-config.xml ``` 4\. Open a second terminal and navigate to your amps installation directory. 5\. Run the following command to listen on the **Alerts** topic using *spark*, the AMPS reference client: ```bash bin/spark subscribe -topic Alerts -server 127.0.0.1:9007/amps/json ``` 6\. Open a third terminal and navigate to your amps installation directory (last one, I promise). 7\. Run the following command to send a message to the **Orders** topic using *spark*, the simple AMPS command-line client: ```bash echo '{"id":1,"status":"PENDING","message":"Take over the world!"}' | bin/spark publish -topic Orders -server 127.0.0.1:9007/amps/json ``` 8\. Bring your second terminal into focus. 9\. Wait 5 seconds; the longest time you have ever had to wait for anything AMPS related. Voila! Your message has appeared! ```bash % bin/spark subscribe -topic Alerts -server 127.0.0.1:9007/amps/json {"id":1,"status":"PENDING","message":"Take over the world!"} ``` I will leave the "firing of lasers" portion of the demo as an exercise to the user ;-) --- # Identifiers, Changes and Chains red hot chain In some applications, unique identifiers for objects change through the object's lifetime. It can be difficult to decide how to model this in systems where an identifier is necessary, such as topics in the AMPS State-of-the-World (SOW).
    A prime example of this problem is in FIX Order ID Chaining. The FIX specification allows systems to declare that a previous Order is canceled and that a new Order, with a new ID, replaces it. A system does this by using the `/41` or `/OrigClOrdID` field of the message. Applications frequently want to model this replacement order as the same Order object as the previous message. In this case, though, there is no shared identifier in a common field that exists across the "chain" of orders, so there's no common field that can be used as a Key: ![fig 1](/img/blog/orderchain_1.png) In the diagram, three messages representing one Order are seen, but neither the `/11` or `/41` fields are suitable for a unique ID. For any given combination of `/41` or `/11` present in an incoming message, previously seen values for those identifiers must be consulted to determine if the incoming message is an update to a previous record, or if the message constitutes a new Order. ## Introducing AMPS ID Chaining AMPS 5.2 provides new functionality which supports correctly identifying messages that follow this chaining pattern. The functionality is called the ID Chaining module, and is configured in your SOW topic configuration. Here's an example: ```xml showLineNumbers libamps-id-chaining-key-generator libamps_id_chaining_key_generator.so ... Orders ./sow/%n.sow nvfix libamps-id-chaining-key-generator /11 /41 ./sow/order.chaining.data ``` Let's dissect this configuration file a bit. The first section, `Modules`, loads an optional module that ships with AMPS called `libamps_id_chaining_key_generator.so`. This module is now available for use as we declare SOW topics. Later in the configuration, we declare a SOW topic called `Orders` of message type `nvfix`. Unlike most SOW topics, we never specify a ``; instead, we specify a `KeyGenerator` element referring to the `libamps-id-chaining-key-generator` module we loaded above. This results in AMPS invoking the ID Chaining module to create a SOW key for each incoming record. We pass `Options` to the ID Chaining module to configure it for our data. The `Primary` and `Secondary` options are used to indicate the message fields which serve as the current/primary ID, and the previous or secondary ID. If you're configuring the module for use with standard FIX data, these will most typically be set to `/11` and `/41`, but they may be set to any fields you'd like. The `FileName` option specifies a file you want the module to store its state in. The module will read this file on startup and keep it updated with the data it needs to preserve the linkage between every valid identifier for each chain. ## See It In Action With this configuration, let's publish some sample data and see what happens. Here's our sample data set: ```bash showLineNumbers $ cat -v orders.nvfix 11=A-1111^A55=MSFT^A44=60.10^A38=100^A 11=A-1121^A41=A-1111^A44=60.10^A38=50^A 11=A-1131^A41=A-1121^A44=60.10^A38=75^A 11=B-1111^A55=IBM^A44=120.01^A38=100^A 11=B-1121^A41=B-1111^A44=120.01^A38=50^A 11=B-1131^A41=B-1111^A44=120.01^A38=75^A ``` Notice we have two order chains present; one identical to the example in the first diagram (the `MSFT` order chain). The second order chain (`IBM`) has a significant difference: the third publish refers to the first ID used in that chain, `B-1111`. Let's publish this data and see how AMPS resolves these ID chains: ```bash showLineNumbers $ ~/spark publish -delta -server localhost:9007 -topic Orders -type nvfix -file orders.nvfix total messages published: 6 (3000.00/s) $ ~/spark sow -server localhost:9007 -type nvfix -topic Orders | cat -v 11=A-1131^A41=A-1121^A44=60.10^A38=75^A55=MSFT^A 11=B-1131^A41=B-1111^A44=120.01^A38=75^A55=IBM^A Total messages received: 2 (Infinity/s) ``` As expected, AMPS ID Chaining resolved these 6 messages into two distinct Orders. Note that even though we used an older ID (`B-1111`) in the 3rd publish on the `IBM` chain, AMPS was still able to resolve this publish to the correct chain. This is because AMPS ID Chaining tracks every distinct ID ever used in the chain, not just the most recent. Doing so frees systems from being concerned that an older order ID might still be used by upstream systems. ## Failure Detection AMPS ID chaining requires that two distinct chains are never resolved together by a future message. For example, this order of publishes cannot resolve to a single message, because the 3rd publish attempts to resolve two existing order ID chains into one: ![fig 2](/img/blog/orderchain_2.png) If publishers cannot be prevented from publishing data which creates this scenario, the ID Chaining module includes a `Validation` configuration option which detects ID Chaining sequencing errors. This option requires extra space and processing time, but can be very helpful in tracking down publisher errors. To enable this option, update the configuration with an entry like the following: ```xml showLineNumbers libamps-id-chaining-key-generator /11 /41 order.chaining.data true ``` When a publisher publishes data with a sequencing error that AMPS detects, errors are emitted to the AMPS log and the message is rejected: ```bash showLineNumbers 2017-03-07T15:01:53.5953200-08:00 [32] warning: 29-0104 Sequencing error: An attempt to map id [A-1121] to SOW key 15073404310751987725 failed, because it was already mapped to SOW key 12015654067891347767. This indicates a sequencing error in upstream publishers. 2017-03-07T15:01:53.5953220-08:00 [32] error: 02-0040 client[my-publisher] published a message which could not be processed by the SOW KeyGenerator: topic = 'Orders' client seq = 0 data = [11=A-1121^A41=A-1111^A44=60.10^A38=50^A] ``` ## Conclusion Systems that process FIX orders are often faced with the need to track "chains" of order IDs that change through time, but this problem isn't confined to FIX orders. Many systems are faced with challenges of identifiers that are not constant. AMPS ID Chaining provides an important tool when working with data that lacks a consistent unique ID per object. For more information on using this feature, consult the [User Guide](/docs/amps-user-guide) --- # Same Data, Unique View: Aggregated Subscriptions unique fish AMPS 5.2 introduces a powerful new capability for subscribers to create custom aggregations and projections to AMPS SOW topics -- with no configuration necessary! We call this functionality _Aggregated subscriptions_. Aggregated subscriptions are like private views for an individual subscription. You no longer have to reconfigure and restart AMPS to test a different calculation, or add a full view for a subscriber that needs different data -- but only for a few days at the close of the month. When a subscriber has unique needs, aggregated subscriptions can give that subscriber a unique view. Aggregated Subscriptions can be used with any command that queries a State of the World topic (for those of you familiar with AMPS, this includes the `sow`, `sow_and_subscribe`, and `sow_and_delta_subscribe` commands.) To use Aggregated Subscriptions, configure one or more SOW topics on your AMPS instance, for example: ```xml showLineNumbers Orders json /order_key ... ``` No additional configuration is required to support aggregated subscriptions; any topic in the SOW may be used with these options. Aggregated Subscriptions specify a set of `grouping` fields and a set of `projection` fields when placing the subscription or issuing the SOW query. These serve the same purpose as the `Grouping` and `Projection` elements in the AMPS configuration when defining a `View`. However, instead of specifying these fields in a server configuration file, you provide these options through the AMPS Client you use, in the `options` field of the command. The AMPS command-line tool `spark` supports providing an options field via the `-opts` argument, so we can use `spark` to quickly test new aggregations. ## Examples Suppose our Orders topic above has been seeded with a few sample messages: ```javascript showLineNumbers {"order_key":1, "symbol":"MSFT", "price":62.30, "qty":100} {"order_key":2, "symbol":"MSFT", "price":62.28, "qty":150} {"order_key":3, "symbol":"IBM", "price":180.20, "qty":16} {"order_key":4, "symbol":"FIZZ", "price":61.77, "qty":4000} {"order_key":5, "symbol":"YUM", "price":64.07, "qty":123} ``` We can use the `sow` command with `projection` and `grouping` options to ask for custom aggregations to be built and returned. Suppose we'd like to know the average order price for each symbol, for example. We can use the command line utility `spark` to easily execute this query: ```bash showLineNumbers $ ~/spark sow -server localhost:9007 -topic Orders -opts "projection=[/symbol,avg(/price) as /avg_price],grouping=[/symbol]" {"symbol":"FIZZ","avg_price":61.77} {"symbol":"YUM","avg_price":64.07} {"symbol":"MSFT","avg_price":62.29} {"symbol":"IBM","avg_price":180.2} ``` Note the syntax of the `projection` and `grouping` options in the `-opts` argument. Both options take a list of fields. For the `grouping` option, this is a list of one or more fields you'd like to group your results by. The list of fields in `projection` is more flexible, and allows you to simply project a field through (e.g. `/symbol`), or use AMPS SQL-like syntax to compute a value you'd like projected (e.g. `avg(/price) as /avg_price`). ## Customizing Output The `projection` syntax allows us to do arbitrary computation and to call User Defined Functions as well. Imagine we'd like to compute and return the average order total by symbol, for example: ```bash showLineNumbers $ ~/spark sow -server localhost:9007 -topic Orders -opts "projection=[lower(/symbol) as /symbol,avg(/price*/qty) as /avg_total],grouping=[/symbol]" -orderby "/avg_total desc" {"symbol":"fizz","avg_total":247080.0} {"symbol":"yum","avg_total":7880.61} {"symbol":"msft","avg_total":7786.0} {"symbol":"ibm","avg_total":2883.2} ``` In this example we use an AMPS built-in function `lower` to convert the symbol names to lowercase; we also average on the order's price multiplied by the order's quantity, and sort the results on this new `/avg_total` field using `-orderby`. ## Subscriptions In addition to a one-time query, aggregated subscriptions can be placed which allows your application to see ongoing updates to the results of the aggregation as changes to underlying data arrive. For fast-moving underlying data, this may be combined with subscription conflation to reduce update frequency. As an example, imagine we place this subscription in one session: ```bash showLineNumbers $ ~/spark sow_and_subscribe -server localhost:9007 -topic Orders -opts "projection=[lower(/symbol) as /symbol,avg(/price*/qty) as /avg_total],grouping=[/symbol],conflation=5s" -orderby "/avg_total desc" {"symbol":"fizz","avg_total":247080.0} {"symbol":"yum","avg_total":7880.61} {"symbol":"msft","avg_total":7786.0} {"symbol":"ibm","avg_total":2883.2} ``` `spark` keeps running, listening for more data. Our use of `conflation=5s` means AMPS will conflate messages it might send us on a 5 second interval. In another window, we quickly publish 4 new Orders for `YUM`: ```javascript showLineNumbers {"order_key":10,"symbol":"YUM","price":70,"qty":10000} {"order_key":11,"symbol":"YUM","price":70,"qty":8000} {"order_key":12,"symbol":"YUM","price":70,"qty":9000} {"order_key":13,"symbol":"YUM","price":70,"qty":7000} ``` Because we've specified `conflation=5s`, we see just one additional message published to our subscriber, a few seconds later: ```javascript {"symbol":"yum","avg_total":477576.122} ``` In addition to conflation, aggregated subscriptions may be combined with both content filters and with delta subscriptions to even further reduce the amount of data your subscriber must process. Aggregated subscriptions are unique to each client, even when they contain the same projection fields and grouping clause, so note that additional system resources are used for each client that requests an ongoing aggregation subscription. The resources used by a client's aggregations count against the configurable byte limits for a client; a client may be disconnected by the server if this exceeds the configured limit. Conclusion === AMPS 5.2's Aggregated Subscriptions make AMPS much more flexible and allows you to build more responsive, customizable applications. This feature allows you to build richer experiences for your users, and to make changes to the aggregations you offer without reconfiguring AMPS. For more information on these new capabilities, consult the User Guide. --- # 60East Launches Media Division with the World’s Most Advanced ASCII Movie Player! ASCII Buster Keaton 60East Technologies is thrilled to launch the world’s most advanced online ASCII movie service, [http://asciiflix.com](http://asciiflix.com). The first project of the new 60East Media Division, this new service leverages the precision and parallelism of 60East’s Advanced Message Processing System (AMPS), using the transaction log replay and State-of-the-World functionality to deliver the most advanced media experience ever seen by humankind. "We’ve been following the revival of vinyl’s popularity among those with the most discerning hearing and waiting for the perfect opportunity to bring the same sort of high-fidelity, old-school precision to fans of movies.", says Brand Hunt, one of the company’s founders. He continues, "With [asciiflix.com](http://asciiflix.com) we’re really pushing the envelope of what’s capable over the web. YouTube is working on 8K video and it’s spectacular, but it can’t tickle the retinas of hipsters like 8 color, 7-bit ASCII can." One anonymous analyst said, "With [asciiflix.com](http://asciiflix.com)’s focus on CC0 and silent movies, they could be a unicorn within 18 months if they can keep the network distribution costs down. No one else is doing anything like this – it’s completely different." 60East media division also announced plans for two ground-breaking technologies on top of the service next month: * __NO-Ctm Advertising__ Ads are spliced into the movies where it’s difficult for the viewer to notice they’re watching advertising. Based on Dr. Julia Eyelidminder’s work at the University of Chicago Starbucks, 99% of the viewers are completely unaware that they’ve even seen an advertisement. This is a stark contrast to other ad delivery systems where it’s immediately obvious to a viewer that they are watching an ad. * __VT100 Support__ Proprietary dithering algorithms, based on the work of Floyd and Steinberg, and offered through the real-time aggregation system of AMPS, the service can tap into the device renaissance and extend the use into devices from the 1970’s, while still being able to please viewers on modern mobile devices. (VT52 support is planned for late next year.) ![Detail of ASCII Buster Keaton](/img/blog/ascii_buster_detail.png) The technology behind ASCIIFlix is nothing short of groundbreaking. Each frame of the video is stored as an individual message within an AMPS server. Viewer applications simply replay the message stream to watch the video. Full support for bookmark replays means that a viewer can pause video at any point, and even lets an application save favorite scenes for later reference. Delivering the video service on top of AMPS allows unheard of scale and precision of frame delivery with minimal buffering on the client side. This is ideal for viewing ASCII videos on small, underpowered IoT devices or decade old terminals. As of launch time, the service offers viewers classic movies such as __The General__, starring Buster Keaton, or delightful short films such as __Peach Bird__ and __Man Typing on a Keyboard__. Enjoy now, while it’s still free! Just go to [http://asciiflix.com](http://asciiflix.com) and sit back (way back… further… no, really, it looks better if you’re not so close) and enjoy the movies! --- # Preprocessing and Enrichment medieval writing desk In AMPS 5.2, we've introduced a new set of capabilities for modifying messages as they are published to AMPS: Message Preprocessing and Message Enrichment. Both features are configured in your AMPS configuration file, on the individual SOW Topics where you would like to use them. These new capabilities can streamline applications that need complex message flows, producing higher performance and easier administration. Here is a brief example of configuring Preprocessing and Enrichment on a SOW topic `Orders`, to both add a new field and validate existing fields: ```xml showLineNumbers Orders json CONCAT(/customer_id,"-",/order_id) as /order_key /order_key IF(/qty < 0, /qty OF PREVIOUS, /qty) as /qty IF(/price < 0, /price OF PREVIOUS, /price) as /price ``` With this configuration, a message published to AMPS that looks like this: ```javascript {"customer_id":"A-111", "order_id":1000, "qty":-1, "price":100} ``` Will be transformed into the following before it is stored in the AMPS SOW, written to the transaction log, or delivered to subscribers: ```javascript {"customer_id":"A-111", "order_id":1000, "order_key":"A-111-1000", "qty":null, "price":100} ``` Note the new `Enrichment` and `Preprocessing` elements in this SOW topic definition. If you have used AMPS Views before, the syntax of each of these features may seem familiar: you define one or more `Field` elements based on AMPS expressions using a SQL-like syntax to define the content of each field. Unlike View projections, enrichments and preprocessing are evaluated as you publish a message into the SOW. The message received by AMPS is amended or changed based on the preprocessing and enrichment rules defined in your configuration file. The altered message is the one stored in the SOW and sent to subscribers. Preprocessing versus Enrichment ------------------------------- Preprocessing and Enrichment are very similar but run at different stages of message processing, allowing you to accomplish unique things with each. Here's an abstract outline of when these steps occur: ![here](/img/blog/preprocessing_1.png) Preprocessing occurs _before_ we evaluate the message's SOW key. This means you can use preprocessing to clean or trim the message's key fields before we use them. In the above example, we compute a brand-new field `/customer_key` which is then used as the SOW key of the message. Enrichment runs later in message processing, after the SOW key has been located and we can load the existing message in the topic for that key, if any. If the publish was a `delta_publish` we merge the message into the existing one as expected, but the previous values for that record are available to enrichment fields via a new `OF PREVIOUS` syntax. `OF PREVIOUS` allows Enrichment rules to use the _previous_ values in a message as part of enriching a new message. In the above example, we use `OF PREVIOUS` to ensure `/qty` and `/previous` are not updated to values < 0, and default them to whatever they were previously. The rule validates that the newly supplied `/qty` value is not less than 0. If it is, the IF clause evaluates to the previous value of `/qty` for that record, and the `/qty` field in the updated message has the value of `/qty` from the previous record. Enrichments are a powerful tool for implementing data quality rules that cannot be implemented at the publisher, since the publisher may not have access to the entire state of the record. Uses ---- Preprocessing and Enrichment are useful for computing new fields and validating or changing existing ones. Here's a few ideas of how you might be able to use them: - Use Preprocessing rules to construct unique SOW keys when publishers do not provide suitable values. - Enforce business logic/business rules for important fields at the server with Preprocessing and/or Enrichment. - Allow or reject different types of updates to a record based on that record's other fields (for example, a `state` field on an order might be used to determine whether the `price` can still be modified.) More ------------------ Additional options exist on Enrichment and Preprocessing for (optionally) removing fields from a message if desired, and for controlling the order of execution of individual Fields. For more on these options and on everything else you can do with Preprocessing and Enrichment, visit the [Enrichment Chapter](/docs/amps-user-guide/enrichment) of our [User Guide](/docs/amps-user-guide)! --- # Crank Up Apache Flume 1960's era wooden logging flume with water running in it 60East customers often integrate their AMPS deployments with the wider ecosystem of cloud storage providers, text search and analysis platforms, and the Apache ecosystem of big data tools such as Spark and Hadoop. One common integration request is the ability to pull an AMPS message stream directly into Apache Flume so that messages can be easily routed to HBASE, Hive, Amazon S3 or Elastic Search for further analysis or processing. AMPS provides sophisticated message processing capabilities that complement Flume in an end-to-end system. AMPS inline [enrichment](/docs/amps-user-guide/optional-modules/experimental-functions#preprocessingenrichment) and powerful [analytics and aggregation](/docs/amps-user-guide/views) (including aggregation of [disparate message formats](/blog/joining-json-bson-xml/)) make it easy to provide processed and enriched data to the destination system. AMPS also provides high-performance [content filtering](/blog/not-using-content-filtering-in-your-messaging-application-youre-doing-it-wrong/) to precisely identify messages of interest, reducing network overhead, storage requirements, and processing time for the destination system. Even better, AMPS provides conflation, paced replay, and resumable subscriptions so that the destination system is never overwhelmed with the volume of incoming messages, and can be populated even in the case of a network outage, planned maintenance, or a failure in a component. Today, we’re making this integration easier by releasing a [custom AMPS source for Apache Flume NG](https://github.com/60East/amps-integration-apache-flume). Apache Flume NG =============== To paraphrase Apache’s project page, Apache Flume NG (simply "Flume" or "Apache Flume" below) is a distributed and reliable service for collecting and aggregating large amounts of data. The “data” routed through Flume can be anything that is adapted to fit Flume’s `Event` abstraction, which is the basic data unit in Flume. Use cases for Flume include social media, trades/executions, and IoT sensor/geo-location data, as well as logging events. Flume is typically run on one or more machines, with each Flume Java process executing a Flume Agent. A Flume Agent is configured to have one or more “sources” that pull in data from various systems. Sources write data to one or more configured “channels”, which usually provide transactional integrity (depending on type). Channels can be configured to have one or more “sinks” that drain their channel of data and write it to some destination system. ![Figure 1: AMPS to Flume Integration](/img/blog/AMPS_Flume_Diagram.png) Flume comes with a variety of built-in source types for importing data from various data sources, such as a generic NetCat source for setting up a network listener socket, to more specialized sources for pulling in data from Syslogs, JMS, Twitter; or in specific data formats such as Avro or Thrift. Flume channels come in a variety of types as well, from the simple and fast (but non-transactional and non-durable) Memory Channel, to slower reliable options that provide transactional guarantees such as a File Channel or JDBC Channel (for committing to any relational database that provides a JDBC driver). Similar to sources and channels, Flume also provides various built-in sink types. These include sinks for writing to the Flume log file (Logger Sink), for writing events to a rolling series of files in a specified filesystem directory (File Roll Sink), and by far its most popular sink, the HDFS Sink, for writing events to the Hadoop Filesystem for downstream batch processing by big data systems such as Apache Hadoop or Apache Spark. Flume also provides the means for third parties to implement their own custom implementations of sources, channels, and sinks, to suit specific needs. For example, connecting to AMPS. The AMPS Flume Source ===================== Flume’s ability to be extended with custom implementations of its components is where the subject of this blog comes in. 60East has released a Flume source capable of pulling in messages from an AMPS subscription and committing those messages to whatever channels the source is attached to in the Flume configuration. The custom AMPS source implements a “pollable” Flume source by sub-classing `org.apache.flume.source.AbstractPollableSource`. This allows the AMPS client created inside the source to read messages from an AMPS subscription and batch them up, committing them to all the channels attached to the source whenever Flume polls the source. This approach is far more desirable from a performance perspective than an event-driven source that commits each message as the message arrives: that approach incurs channel transaction costs on a per message basis. Getting Started =============== To get started you will need at least a Java 7 JDK, Maven 3.3, and Apache Flume NG 1.7.0. First clone the AMPS Flume source repository from GitHub: ```bash git clone git@github.com:60East/amps-integration-apache-flume.git amps-flume ``` Follow the build and installation instructions in the `README.md` file located in the cloned directory (also available under [https://github.com/60East/amps-integration-apache-flume](https://github.com/60East/amps-integration-apache-flume) ). This will walk you through building the AMPS Flume source JAR and how to install it as a plugin in your Apache Flume installation. Once you have the AMPS source installed as a plugin, you can use it within your Flume configuration. Below is an example configuration for an AMPS Flume source: ```bash showLineNumbers # Example config for an AMPS Flume Source agent.sources.amps.type = com.crankuptheamps.flume.AMPSFlumeSource agent.sources.amps.clientFactoryClass = com.crankuptheamps.flume.AMPSBasicClientFunction agent.sources.amps.clientName = FlumeClient agent.sources.amps.bookmarkLog = # For one AMPS server, you can just specify "uri". # For multiple HA AMPS servers, specify multiple URIs with an index number. agent.sources.amps.uri1 = tcp://server1:9007/amps/json agent.sources.amps.uri2 = tcp://server2:9007/amps/json agent.sources.amps.command = sow_and_subscribe agent.sources.amps.topic = Orders agent.sources.amps.filter = /symbol IN ('IBM', 'MSFT') agent.sources.amps.options = projection=[/symbol,avg(/price) as /avg_price,\ avg(/price*/qty) as /avg_total],grouping=[/symbol],conflation=1s agent.sources.amps.subscriptionId = Sub-100 agent.sources.amps.maxBuffers = 10 # maxBatch must be <= the smallest transactionCapacity of all channels # configured on the source. agent.sources.amps.maxBatch = 1000 agent.sources.amps.pruneTimeThreshold = 300000 ``` Here we’re configuring a Flume agent called `agent`, as can be seen above in the configuration key prefix. Under `agent`’s sources we have configured our source to be called `amps`. The type of any AMPS source must be the class `com.crankuptheamps.flume.AMPSFlumeSource`. This is the class from our Apache Flume integration repository that implements our custom AMPS source. ### Client Settings The `clientFactoryClass` configuration key is optional. By default it will use the built-in implementation, `com.crankuptheamps.flume.AMPSBasicClientFunction`, which will make use of the `clientName`, `bookmarkLog`, and `uri[n]` configuration keys to create an AMPS `HAClient` instance. If you have need of customizing the AMPS client creation process, such as to use a custom AMPS authenticator, a custom server chooser, or even a custom `HAClient` sub-class, you can create your own AMPS client factory by implementing the `SerializableFunction` interface from the AMPS 5.2.0.0 Java client. See the source code of `com.crankuptheamps.flume.AMPSBasicClientFunction` for an example of this. You would then need to package your client factory class inside `AMPSFlumeSource-1.0.0-SNAPSHOT.jar`, or in your own JAR that is placed in the `$FLUME_HOME/AMPSFlume/libext/` directory. Then just specify your client factory’s fully qualified class name as the value of the `clientFactoryClass` configuration key. The `clientName` configuration key is required and like any AMPS client name, it must be unique across all AMPS clients connecting to a high availability (HA) cluster. The `bookmarkLog` configuration key is used to specify the absolute or relative path to the bookmark store log file. This configuration key is optional. If this is specified, the source will create a bookmark subscription for reliability purposes. The default client factory implementation will always use a `LoggedBookmarkStore` implementation on the client it creates when this key is specified. The `uri` configuration key is required. It is used to specify one or more AMPS server transport URIs that you would like the AMPS client to connect to. If you only have a single AMPS server you want to connect to, the key may be specified as either `uri` or `uri1`. If you want to specify multiple AMPS servers in an HA cluster, each transport URI should be specified with an index number, starting with “1” and having no gaps. For example, `uri1`, `uri2`, and `uri3` for three URIs. ### Subscription Settings The `command` configuration key is optional. If not specified its value defaults to `subscribe` for an ordinary subscription, though you could also specify values such as `delta_subscribe`, `sow_and_subscribe` or `sow_and_delta_subscribe`. In our example we are using the `sow_and_subscribe` command to query the State of the World (SOW) database topic and then subscribe to it for future updates. The `topic` configuration key is required. This is the topic used in the AMPS subscription. Like any AMPS subscription, this may be a valid regular expression to allow subscribing to multiple topics. The `filter` configuration key is optional. This is the filter expression used on the AMPS subscription. Our example here will filter incoming messages to just those with `/symbol` values of `IBM` or `MSFT`. The `options` configuration key is optional. These are the options to be used on the AMPS subscription. In our example here we are using the new AMP 5.2 aggregated subscription feature to group the data by `/symbol` and project calculated view fields. We are also using a conflation interval of 1 second to collect all updates to the topic within a second and only send our Flume client at most one update per a second. The `subscriptionId` configuration key is used to specify the subscription Id for a bookmark subscription, so when Flume is restarted it can recover from the bookmark log and continue the subscription where it left off. This should be unique amongst all subscriptions in the application. If the `bookmarkLog` configuration key is specified, then this is required, otherwise it’s optional. ### Tuning Settings The `maxBuffers` and `maxBatch` configuration keys control the batching of AMPS messages that are committed to the channel(s). They are both optional and have default values of 10 and 1000 respectively. These can be tuned to trade-off performance verses memory usage of the AMPS source. The `maxBuffers` key determines the maximum number of AMPS message buffers the source will queue up in memory before pausing to let Flume sinks catch up in draining attached channels. The `maxBatch` configuration key is the maximum number of AMPS messages allowed inside a message buffer. This is the maximum batch size that will be committed to all attached Flume channels. As such, this value MUST be less than or equal to the `transactionCapacity` of every channel attached to the source, or you will always get channel commit errors (`ChannelException`) in the Flume log and no messages will ever reach attached channels or sinks. If you multiply `maxBuffers` and `maxBatch` you will get the maximum number of AMPS messages held in source memory waiting to be committed to all attached channels. If sinks don’t drain attached channels fast enough, this limit will be reached and you will see this warning in the Flume log: ```bash Pausing AMPS message processing to let Flume source polling catch-up. Consider increasing maxBuffers; or maxBatch and the transaction capacity of your Flume channel(s). ``` If the source uses a bookmark subscription, the AMPS Flume source doesn’t discard any AMPS message in the bookmark store until it has been committed to all channels attached to the source. So if a Flume agent suddenly goes down with thousands of messages batched in memory, upon restart the subscription will be recovered from the bookmark log and all messages that haven’t been marked as discarded will be redelivered to the AMPS source. For each batch, there is a small window between the time Flume indicates that attached channels are successfully committed and when bookmark store discards take place. If an outage occurs in this window, then all or some of the committed messages will be redelivered upon restart (though messages will always be in the proper order for a given publisher and there will be no gaps). This means that the AMPS Flume source provides at-least-once delivery semantics for bookmark subscriptions. The `pruneTimeThreshold` configuration key is optional and has a default value of 300,000 milliseconds (5 minutes). This key determines the minimum amount of time that must elapse before the client’s `LoggedBookmarkStore` will be pruned of obsolete entries. The elapsed time is measured from the source’s start-up time or from when the last prune operation was performed. For debugging and testing purposes, this value can be set to zero to turn off all pruning (NOT recommended for general production use). The `LoggedBookmarkStore` will also be pruned upon normal Flume shutdown, unless this value is zero. If a custom client factory installs another bookmark store implementation on the client, then this configuration key has no effect. Cranking It Up ============== Included in the project repository is a working example that shows off the powerful new aggregated subscription feature of AMPS 5.2 (for more info see our blog post on [aggregated subscriptions](/blog/same-data-different-view-aggregated-subscriptions/)). Go to your cloned GitHub repository directory. Copy the example Flume configuration file to your Flume 1.7 installation: ```bash cp src/test/resources/flume-conf.properties $FLUME_HOME/conf/ ``` Be sure to rename the file if you already have a config file by the same name. Next, start an AMPS 5.2 server instance with the included AMPS config file at: `src/test/resources/amps-config.xml` Create a directory at `/tmp/amps-flume/` to hold the event output of Flume: ```bash mkdir /tmp/amps-flume/ ``` Now start Flume from your `$FLUME_HOME` directory: ```bash bin/flume-ng agent -Xmx512m -c conf/ -f conf/flume-conf.properties -n agent -Dflume.root.logger=INFO,console ``` Lastly, publish the example JSON messages to the `Orders` topic using the AMPS spark utility: ```bash spark publish -server localhost:9007 -topic Orders -rate 1 -file src/test/resources/messages.json ``` Notice that we are publishing at a rate of 1 message per a second, so that in the output we can see the aggregate fields change over time as updates arrive. These are the 15 messages we are publishing: ```bash {"order_key":1, "symbol":"MSFT", "price":62.15, "qty":100} {"order_key":2, "symbol":"MSFT", "price":62.22, "qty":110} {"order_key":3, "symbol":"IBM", "price":180.40, "qty":125} {"order_key":4, "symbol":"FIZZ", "price":61.77, "qty":4000} {"order_key":5, "symbol":"YUM", "price":64.07, "qty":123} {"order_key":6, "symbol":"IBM", "price":181.02, "qty":200} {"order_key":7, "symbol":"FIZZ", "price":61.45, "qty":2300} {"order_key":8, "symbol":"MSFT", "price":62.52, "qty":1000} {"order_key":9, "symbol":"IBM", "price":180.90, "qty":750} {"order_key":10, "symbol":"MSFT", "price":62.45, "qty":900} {"order_key":11, "symbol":"YUM", "price":64.11, "qty":460} {"order_key":12, "symbol":"IBM", "price":180.85, "qty":150} {"order_key":13, "symbol":"FIZZ", "price":61.50, "qty":600} {"order_key":14, "symbol":"MSFT", "price":62.70, "qty":1200} {"order_key":15, "symbol":"IBM", "price":180.95, "qty":480} ``` After about 15 seconds (due to our 1 second rate of publishing and our 1 second conflation interval) our aggregated subscription view of the data gives us the following results under `/tmp/amps-flume/`: ```bash {"symbol":"MSFT","avg_price":62.15,"avg_total":6215.0} {"symbol":"MSFT","avg_price":62.185,"avg_total":6529.6} {"symbol":"IBM","avg_price":180.4,"avg_total":22550.0} {"symbol":"IBM","avg_price":180.71,"avg_total":29377.0} {"symbol":"MSFT","avg_price":62.2966666666667,"avg_total":25193.0666666667} {"symbol":"IBM","avg_price":180.773333333333,"avg_total":64809.6666666667} {"symbol":"MSFT","avg_price":62.335,"avg_total":32946.05} {"symbol":"IBM","avg_price":180.7925,"avg_total":55389.125} {"symbol":"MSFT","avg_price":62.408,"avg_total":41404.84} {"symbol":"IBM","avg_price":180.824,"avg_total":61682.5} ``` For cases where the entire raw message stream is desired at maximum throughput, you would use a `subscribe` command and wouldn't specify options such as `projection`, `grouping`, or `conflation`. For reliability and restart recovery, you would need to then specify the `bookmarkLog` configuration key to use a bookmark subscription. Conclusion ========== Importing an AMPS message stream into Flume used to require intermediate steps and third party software. Now integrating AMPS with Apache Flume has never been easier. The new AMPS Flume source allows you to plug an AMPS subscription directly into a Flume flow. How do you plan to use the Flume integration? Would you like to see AMPS connected to other software stacks -- either as a sink or source? Let us know in the comments! --- # Hot, Fresh, and Expressive: New AMPS Functions! Microphone. image by drestwn -- CC BY 2.0 AMPS 5.2 has dropped and, like a new Beyonce album, it is so awesome it will probably break the internet. AMPS 5.2 comes with a mind bending amount of new functionality, but I would like to focus on a few key new functions that have been made available to your AMPS expressions. Functions are like the backup singers of the AMPS world: they may not be what you hear first, but they sweeten the mix and you'd miss them if they weren't there. First, a quick review. A key piece of the AMPS workflow is a full featured expression language that is based on XPath and SQL-92. This language is used for: - Content filtering - for client subscriptions - for server configuration settings such as actions - for filtered (content-aware) entitlements - Creating projected fields for views - Constructing fields for message enrichment (Another exciting new feature of AMPS 5.2) If you have used AMPS, you have probably used the AMPS expression language. The AMPS expression language exposes a range of functions that allow you to do computations on your message data right inside the system, with very high performance. These include familiar string query functions such as: - `SUBSTR()` - `INSTR()` as well as the numeric aggregation functions: - `AVG()` - `COUNT()` - `MIN()` - `MAX()` - `SUM()` These functions provide a lot of utility, but we weren't satisfied! AMPS 5.2 has greatly expanded the amount of built in functions available to you in your AMPS expressions. There are now 13 new functions available to for use in your AMPS expressions. Let me break them down for you: ### Numeric Operations: Let's start with the most straight forward additions. These are numeric functions that will make it just a little bit easier to tailor AMPS expressions for your use case without extra effort. | Function | Description | | --------- | ---------------------------------------------------------------------------------------------------- | | `ROUND()` | rounds values to the nearest integer, or to the nearest specified decimal place if one is specified. | | `ABS()` | Returns the absolute value of a number. | ### String Operations: String processing is an important part of a good messaging system, so we created quite a few new string functions. I'm going to break them down by sub-category to more clearly show their utility: #### Search and Replace | Function | Description | | ------------------ | ------------------------------------------------------------------------------------------- | | `REPLACE()` | Search and replace all occurrences of a string and return the resulting string. | | `REGEXP_REPLACE()` | Search and replace all occurrences of a regular expression and return the resulting string. | #### Search and Case Sensitivity Previous versions of AMPS required you to use to regular expressions if you wanted to handle case-insensitive matching. Regular expressions are very powerful but come with a large amount of complexity. AMPS 5.2 gives you new tools that allow you to provide case insensitive matching without resorting to the "big guns" inside regular expression matching. Starting with: | Function | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `INSTR_I()` | Case-insensitive, Returns the position at which the second string starts, or 0 if the second string does not occur within the first string. | | `STREQUAL_I()` | Case-insensitive, Returns true if, when both strings are transformed to the same case, the string to be compared is identical to the string to compare. | These are exactly like their case-sensitive counterparts from previous versions of AMPS, with the very useful distinction that they ignore case. The addition of those functions are theoretically enough to handle most string case related issues, but the AMPS team did not stop there! We wanted to provide you maximum flexibility. In some cases, particularly when using strings with the IN clause, it is more efficient to simply convert the string to a known case. This is why we introduced: | Function | Description | | --------- | ----------------------------------------------------------------------------- | | `UPPER()` | provide the ability to convert ASCII strings to their upper case equivalents. | | `LOWER()` | provide the ability to convert ASCII strings to their lower case equivalents. | One final note about these string functions. It's important to point out that these functions are **NOT** unicode-aware. #### Concatenation I would like to take a moment to talk about CONCAT, because it is particularly flexible and powerful. The function accepts both XPath identifiers and literal values and will return a string composed of all of them. CONCAT also works with non-string values, and will attempt to convert them to strings using the normal AMPS string coercion rules. | Function | Description | | ------------------ | ------------------------------------------------------------------------------------------- | | `CONCAT()` | Allows concatenating the string representations of one or more fields into a single string value. CONCAT may be called with any number of arguments. | `CONCAT()` is be very useful for debugging or status message generation, but it can also be used in more clever ways. For example, you can create a unique record id for a view that is composed of two different underlying topics: ```xml showLineNumbers MechaGodzilla /GiantRobots/body_part = /GiantMonsters/body_part json CONCAT(/GiantRobots/part_id, /GiantMonsters/monster_id) AS /cyborg_id CONCAT("Mecha", /GiantMonsters/monster_name) AS /monster_name /GiantRobots/part_id /GiantMonsters/monster_id ``` ### Filtering AMPS 5.2 has added one very important filtering function: | Function | Description | | ---------- | ----------------------------------------------------------------------------------------- | | `COALESCE()` | returns the first non-NULL argument. COALESCE may be called with any number of arguments. | `COALESCE()` takes a list of field identifiers, and returns the first one that is not Null. This can be used in several powerful ways. By using `COALESCE()` in your views, you can create very fine grained aggregated fields. For example: ```sql showLineNumbers COALESCE(/userCategory, /employeeCategory, /vendorCategory, 'restricted') ``` `COALESCE()` can be used as a cleaner alternative to IF chaining. For example, you may want to determine a total for an order, but it may have several possible values for a price. In older versions of AMPS you would write the expression like this: ```sql showLineNumbers /Order.Qty * IF(/Order/NormalPrice IS NOT NULL, /Order/NormalPrice, IF(/Order/SpecialPrice IS NOT NULL, /Order/SpecialPrice, IF(/Order/SuperFriendsDiscount, /Order/SuperFriendsDiscount, 0) ) ) ``` With `COALESCE()` in AMPS 5.2, you can simply write: ```sql showLineNumbers /Order/Qty * COALESCE(/Order/NormalPrice, /Order/SpecialPrice, /Order/SuperFriendsDiscount, 0) ``` Here are a few tips for working with `COALESCE()`. First, notice that, to make the intent of the filter clear, this example provides a constant value for AMPS to return from the COALESCE if all of the field values are NULL. Second, note that `COALESCE()` is a scalar function. It takes a list of scalar values or scalar valued fields. This means that arrays will get converted to "scalar context"! In the most simple terms this means that `COALESCE()` will use the first value of the array as the value, and ignore the rest of the array. ### AGGREGATION For all of you data scientists out there, AMPS 5.2 provides a few new functions that you are going to love: | Function | Description | | ------------------ | --------------------------------------------------------------------------------------------- | | `COUNT_DISTINCT()` | takes a single argument and returns the number of distinct values within the aggregate group. | | `STDDEV_POP()` | return the population standard deviation. | | `STDDEV_SAMP()` | return the sample standard deviation. | These functions are specifically for use with View fields. These functions return a single value for each distinct group of messages, as identified by distinct combinations of values in the Grouping clause. This class of functions existed in previous versions of AMPS with functions such as `AVG()`, `COUNT()`, `MIN()`, `MAX()`, and `SUM()`. The addition of these new aggregation functions allow you to perform more advanced statistical analysis on your data right inside AMPS, allowing you to get the analysis you need with lower latency and less complexity. ### Conclusion The AMPS expression language was always very powerful. It's one of the key pillars of the flexibility of AMPS. These new functions have expanded that flexibility, giving you even more options to tailor AMPS perfectly to your use case. AMPS 5.2 has brought a huge amount of new features; too much for a single blog post! look for the rest of this series of blog posts as we begin to highlight more new AMPS 5.2 features. AMPS Speed! --- # Is Your App Solid? (Does it need SSD?) man on tricycle with rocket strapped to his back It’s 2017 and this is the year Gartner estimated more revenue will come from selling Solid State Drives (SSDs) than their slower, spinning, ancestral Hard Disk Drives (HDDs). [^registerarticle] To some, this seems like a no-brainer, but many experts believe the estimate is way off as HDDs continue to improve capacity, MTBF (Mean Time Between Failure), performance, and price. As more enterprises adopt SSDs, there have been shortages in flash memory used in their construction, which has driven prices up and created supply chain delays. That’s why we’re here, to explore situations where you really need SSDs vs when something else may be good enough (or better!) for your use. Let’s quickly review the primary differences that make up the choice: Capacity, Failure rate, Performance (both sequential and random I/O), Price and Sourcing. This is a high-level comparison only, data-center architects are often considering many more factors such as heat, power, form factor, compatibility, vendor preference, and ease of installation. * **Capacity**: HDD is currently the winner here. You can easily buy +10TB drives today and it’s estimated we could see 100TB drives by 2020. If your goal is to cram as much storage as you can into the smallest space possible, then the density of HDDs is a winner. * **Failure Rate**: Both SSDs and HDDs are making amazing improvements in reliability, with some SSDs and HDDs reporting MTBFs of >1.5M hours, we consider these equivalent. This means if you operate 167 drives for a year, you’ll expect to see an average of 1 fail. * **Performance**: Both SSDs and HDDs are great at sequential I/O throughput. SSDs are a clear winner over HDDs on random I/O and have service times that are measured in microseconds, not in milliseconds where moving parts must seek to the I/O location on each operation. If you’re primarily doing sequential I/O (like recording video or time series data), then HDDs could serve you well. However, if you’re requiring the fastest service times and optimal performance for random I/O, then you need SSDs. It’s important to understand your workload and isolate your HDDs for sequential I/O workloads, otherwise other tasks that will move the device’s write head and you’ll see performance degrade to that of the random I/O workloads. * **Price**: Price is a tricky subject – are you wanting to optimize spend over space or time? That is, do you measure “capacity” in terms of gigabytes or in transactions per second? If you’re looking at transactions per second per dollar, then you may need SSDs. However, most comparisons use $/GB metrics and with HDDs giving between 8-10x more capacity per dollar spent, I’d consider HDDs here the winner in most cases. * **Sourcing**: We’ve already seen flash shortages that disrupt the availability of SSDs making it difficult to make large purchases at times – meaning higher costs and longer lead times to get them when you need them. There are many reasons for flash shortages (material shortages, demand, etc.) and it’s difficult to predict when they’ll happen. If you’re needing SSDs, make sure you plan your sourcing process far ahead to get the devices you need when you need them! HDDs are currently easier to source, so in some urgent storage scenarios, you may not even have a choice of going with SSDs. ![HDD compared to SSD](/img/blog/ssd_post/options_sliders.png) _HDD vs SSD on Common Attributes_ What's This Mean for Customers of AMPS? --------------------------------------- AMPS can benefit from fast storage on several dimensions. The most obvious one is transactional volume in its transaction log. Since AMPS writes to the transaction log durably (it doesn’t acknowledge it’s stored until the device has completed the write) then the throughput of the device dictates the transaction throughput of the system. For this comparison, we’re using commodity devices locally attached to AWS cloud instances and the device was completely isolated, so this was the best-case scenario for sequential write performance for these devices. (Important: Using higher-end, enterprise SSDs AMPS can do easily 5x these numbers.) ![Peak througput](/img/blog/ssd_post/peak_throughput.png) _Peak throughput comparison_ You can see that an HDD does well at 300K messages per second. There are many messaging problems where that’s easily enough capacity -- however, you need to be careful to isolate the AMPS transaction log to prevent other applications or services from moving that write head. If you’re over 100-300K messages a second, you’ll want to look at an SSD. If you require more than 300K/s, you may want to start considering higher end SSD devices as well where you can achieve 2-3 million transactions per second. If you have a transactional State-of-the-World (SOW) topic, then you’ll have a workload that’s considered random I/O and the performance of your AMPS deployments can greatly benefit from SSDs. When AMPS updates records for transactional SOW topics it will typically update the on-disk record image in-place, which has all the characteristics of random I/O for most workloads and abysmal HDD performance. This also means that instance recovery or SOW topic rebuilds can be done much faster on SSDs. ![Transactional SOW performance](/img/blog/ssd_post/transactional_sow_performance.png) _Transactional SOW performance_ Event Stream Replay Performance ------------------------------- AMPS has a sophisticated transaction log replay system that allows users to replay content filtered event streams at a specific rate – for example, you could replay an event stream at 4x the real-time rate, 5MB/s, or 50,000 messages per second. Some users have 1000’s of these replay consumers at any given time, which can put tremendous stress on the storage device in environments where the total persisted event stream is much larger than the amount of memory available in the host environment. The AMPS Transaction log is composed of individual journal files, where the latest journal file is being written to and any of the journal files could be concurrently read from. These event stream replay consumers, even though they’re individually sequential, will perform like a randomized I/O workload in aggregate. Because of the randomized I/O pattern, the write performance to the head journal will suffer in HDD environments with several concurrent replay consumers. In the following diagram, you can see a high-level view of the AMPS transaction log, showing replay consumers reading from “cold” journals as well as the head journal being written to. We’ve designed AMPS so that cold journals can be archived to cheap-n-deep HDD (or even NAS) so that certain deployments only need SSD storage to cover their “hot” data where the data is being read and written to. ![Transaction log flow](/img/blog/ssd_post/transaction_log_flow.png) _Transaction log overview_ Warnings & Tips --------------- **Slow SSDs**: Don’t go buying SSDs assuming _any_ SSD will be better than _any_ HDD, there do exist SSD devices that are far slower/worse than HDDs – check the drive specs before buying! **File System**: The choice of file system can make a huge difference in performance for some work-loads, but more importantly are the mount options for the device. For example, if you’re using the popular ext4 file system, do you need to have journaling or can you turn it off for a performance benefit? Can you use `noatime` and `nodiratime` to turn off file metadata updates to update the access time for every read? **Multi-Tenant Environments**: If you plan on running multiple AMPS instances on the same host or other applications alongside an AMPS instance, you’ll want to be careful to isolate the load or provide enough capacity to achieve your objectives. It’s a common error to benchmark systems independently and then deploy collectively not aware of how the resources degrade non-linearly under load. So, Do I need an SSD or not? ---------------------------- _**TLDR;**_ Here are some questions to help determine if you truly need an SSD for your AMPS deployment. If you answer “Yes” to any of these, then you’re a great candidate for an SSD and it’s unlikely you’ll regret paying up for the SSD device. _**Will you be operating multiple I/O heavy applications on the host?**_ Multi-tenant environments where performance matters can benefit from SSD and minimize the disruption of one service from I/O bursts of another. _**Will you be storing more than 2x the hosts available memory on the device?**_ When the amount in storage vastly exceeds the host’s memory, system performance can be lost to OS virtual memory paging activity. _**Do you plan on making heavy use of bookmark/replay subscriptions?**_ _**Or**_ _**Do you plan on using transaction log backed SOW topics?**_ _**Or**_ _**Do you plan on using high-velocity message queues?**_ The features above can approximate a randomized I/O workload and can benefit from SSD devices. _**Will you be publishing bursts of events or messages at more than 100MB/s?**_ If you’re publishing message bursts at greater than 100MB/s and you’re latency sensitive, then you’ll need an SSD. If you’re publishing bursts of more than 200MB/s, you may want to look into a higher end SSD/flash device. _**Can you afford the monetary costs and sourcing lead-time?**_ SSDs are currently more expensive and the sourcing lead-time can be significant for some firms. If you need ample storage today, you may not have SSD as an option. Summary ------- Because of the differences in performance, cost, and availability, both HDD and SSD options continue to thrive. Every quarter the HDD vendors offer more capacity and a lower price while SSD vendors are cranking up capacity and performance. If you’re even asking the question “Do I need an SSD?” and you can buy an SSD device that matches your criteria for performance, capacity, budget, and time-to-market, then you should’ve already done it. _[Updated 08 September 2017: Clarify that queues can have a disk usage pattern similar to bookmark replay.]_ [^registerarticle]: [https://www.theregister.co.uk/2016/01/07/gartner_enterprise_ssd_hdd_revenue_crossover_in_2017/](https://www.theregister.co.uk/2016/01/07/gartner_enterprise_ssd_hdd_revenue_crossover_in_2017/) --- # Beat the Traffic With Conflation congested freeway Does your network bandwidth ever feel like this congested freeway? Do you ever have a subscription that gets so many messages that it really cannot process them all? If you answered yes, why haven’t you tried using conflated subscriptions? Conflated subscriptions help to reduce the bandwidth for a subscription and may reduce the processing resources required for that subscription as well. They work just like conflated topics, which are defined in the AMPS config, but with a couple of differences. The differences are you can set the conflation interval on a per subscription basis and you can subscribe to any topic, not just conflated topics. This is great if you have multiple subscriptions that may have different intervals in which they need to process data. Let's look at an example where conflated subscriptions might be useful. Imagine there is an application which displays selected stocks with their current prices. This display refreshes every two seconds. Since the application only refreshes the display every two seconds, it only needs new data every two seconds, rather than a full stream of the price changes. On the other hand, a trading desk might need the full stream of messages and therefore will not set a conflation interval or conflation key(s) on their subscriptions. To accomplish this, the display application can place a subscription that updates the price based on the `tickerId` with a conflation interval of two seconds and a conflation key of `tickerId`. For example, in Python: ```python showLineNumbers client = AMPS.Client("myConflatedClient") client.connect(uri_to_amps) client.logon() command = AMPS.Command("subscribe").set_topic('prices') \ .set_options('conflation=2s, conflation_key = [/tickerId]') message_handler = messageHandler() client.execute_async(command,message_handler) ``` AMPS will hold the data for each distinct `tickerId` for two seconds before sending a message to the application. For example, AMPS will receive the following messages within the conflation interval: ```js showLineNumbers {"tickerId":"IBM","price":150.34} {"tickerId":"IBM","price":149.76} {"tickerId":"MSFT","price":70.76} {"tickerId":"IBM","price":149.32} {"tickerId":"MSFT","price":70.94} {"tickerId":"IBM","price":151.10} ``` Any messages that are received by AMPS with the same `tickerId` will be replaced until the two second interval is reached and then the data is sent to the subscriber. Only the last message for each `conflation_key` is delivered by AMPS at the end of the interval: ```js showLineNumbers {"tickerId":"IBM","price":151.10} {"tickerId":"MSFT","price":70.94} ``` This means the application does not have to process many updates that will ultimately never end up being displayed, it will not waste time parsing unnecessary updates and an up to date price is always shown on the refresh interval. This will free up some of your bandwidth and let you unleash the horses under the hood of your other subscribers. This also means that your application may not receive messages strictly in the order published, but will receive the last message for each distinct conflation key during the interval. ![fast car on luminous highway](/img/blog/horsepower_car.jpg "Unleash your application's true speed with conflation")

    Even More Power

    Conflated subscriptions are new in AMPS 5.2. They're just one of the ways that AMPS can provide conflation. Conflated subscriptions are the simplest to demonstrate, since they don't require changes to the AMPS configuration. Conflated subscriptions require AMPS to calculate conflation individually for each subscription that requests conflation. If several applications (or several instances of an application) all need conflation at the same interval, use a [Conflated Topic](/docs/amps-user-guide/conflated-topics) instead. The results are the same for each individual subscriber, but AMPS keeps track of conflation for the topic as a whole rather than an individual subscription, which is more efficient. Conflated topics are also available in older versions of AMPS, so if you're on an earlier version, conflated topics are the available option. --- # Do-It-Yourself SOW Keys key cutting machine The AMPS State-of-the-World (SOW) depends on being able to identify distinct updates to a message. AMPS does this by creating a SOW key for each message: subsequent updates that have the same key are updates to the same message. In many cases, it's convenient to have AMPS determine the SOW key for a message based on the contents of the message. That's not the only way to get a SOW key, though. An explicit SOW key allows a publisher to specify the SOW key of a message at the time the message is published. AMPS does not try to interpret the data within the message to generate a SOW key, or to determine if the message is unique. Instead, AMPS relies only on the provided SOW key to determine the identity and uniqueness of the message. This can be useful for a variety of different purposes. ### Common Uses There are two common uses for explicitly defining your own SOW keys that I would like to mention. The first common use is for binary message types. Explicit SOW keys are necessary for binary messages because AMPS will not attempt to parse a binary message, and therefore cannot obtain a SOW key from within the message. With that in mind, explicit SOW keys are the only way to build a SOW topic that contains binary messages. The second common use of explicit SOW keys is situations where a message does not contain a unique key, and you aren't able to add one to the message itself. In this situation, you would give AMPS a different key for each unique message published. Though these may not be the only situation that an explicit SOW key could be used, they are the most common. ### Considerations When Generating SOW Keys There are a few things to keep in mind when generating your SOW keys: 1. All publishers should use a consistent method for generating SOW keys. This is to guarantee that all publishers can update the proper message in the SOW. 2. SOW keys must contain only characters that are valid in Base64 encoding. 3. The application must ensure that messages intended to be logically different do not have the same SOW key. As long as you are able to meet this criteria, your keys can be anything that you want. ### Configuring AMPS Configuring AMPS to use explicit SOW keys is quite similar to configuring any other topic in the SOW, with one important difference. The `Key` option will not be provided. Here is an example a SOW configured to use explicit keys: ```xml showLineNumbers Foo path/to/sow/Foo.sow json ``` If you provide a `Key` in the configuration, any explicit SOW key set on a publish message will be ignored and the configured `Key` will be applied. ### Publishing the SOW Key Explicit SOW keys are set on a publish message using the Command Interface. This is done by creating a publish Command object, then calling `set_sow_key` on that object. A python example of this would be as follows. ```python showLineNumbers # Let's build a simple publish Command cmd = AMPS.Command("publish").set_topic(topic).set_data(data) # Now let's set the SOW key cmd.set_sow_key(key) # and finally send it, using # execute_async with no handler # for efficiency client.execute_async(cmd, None) ``` That's all there is to it! Once the message is published, AMPS will handle the rest. ### SOW Queries Data published with an explicit SOW key can be queried with a content filter just like any other non-explicit SOW topic. You can also query this data using the SOW key that you provided. In order to issue a query by the explicit SOW key, you build a SOW `Command` and set the SOW key on that `Command`. Here is an example of this: ```python showLineNumbers cmd = AMPS.Command("sow") cmd.set_sow_keys(key).set_topic(topic) for m in client1.execute(cmd): print "%s: %s" % (m.get_sow_key(), m.get_data()) ``` Though this might not be useful in every case, the query will be faster for those applications that need it. If your application only needs to store and retrieve values using a single key, using an explicit SOW key is the most efficient approach. ### Further Reading For more details, see [How Does the State of the World Work?](/docs/amps-user-guide/sow/how_does_sow_work) and the section on [SOW Keys](/docs/amps-user-guide/sow/sow_keys) in the [AMPS User Guide](/docs/amps-user-guide). ### Closing Thoughts Using an explicit SOW key can be useful when you have no way of identifying a key within the message itself. This is particularly the case with Binary messages, but it may have many other uses. As long as your application has a way of consistently generating a unique SOW key for each unique message, then setting an explicit SOW key might be for you! --- [Edit: 2024-04-29 Fix the SOW query example code snippet.] --- # Pirates of AMPS: Dead Man's Queue burning letter with skull and cross-bones Queues are the bread and butter of a good messaging system. AMPS provides a powerful queue system that is fast, resilient, and flexible, taking work from publishers and feeding them to consumers as fast as your network will allow. The real world, unfortunately, has time constraints. AMPS Queues are extremely performant and are often used in very time sensitive use cases such as the processing of market data. It is extremely important that consumers of these time sensitive queues don't receive stale data or _poisoned_ data that can cause the system to get stuck. I will talk about what I mean by poisoned data later, but lets take the basic case of stale data first. AMPS solves this problem by allowing an expiration to be set on queue messages. If a message outlives its expiration it is forced to walk the plank and is removed before another client can receive it. This is great! It assures that consumers only have the latest messages, and everything is smooth sailing on the AMPS seas. ![Normal Queue and Dead Letter Flow](/img/blog/queue-flow-opaque.png "Normal Queue and Dead Letter Flow") While keeping only the latest messages in the queue is important for time sensitive applications, those stale messages can still be useful! Statistics on those stale messages can contain important clues about possible problems and improvements to consumer clients. In use cases where consumer speed is of the utmost importance, these statistics are a map to hidden treasure for your business. If only there was a way to save those expired messages... ### The Dead Letter Queue A secondary queue that tracks expired messages from one or more primary work queues is known as a **dead letter queue**. The powerful automated actions platform of AMPS allows you to not only build a dead letter queue, but using AMPS views, you can create conflated statistics of your dead letter queue. You can use this queue and view setup to gain unpreceded insight into your AMPS queue performance! This whole system hinges on the `on-sow-expire-message` action. This powerful action acts like a life boat for those stale messages walking the plank. You might be saying to yourself, "`on-sow-expire-message`? But I thought we were talking about queues not sows?" Yes! But this is part of the power of the AMPS platform. Queues are implemented as a view over an underlying topic or set of topics that are backed by a transaction log. Among other things, this allows AMPS actions to receive events from Queues just like they were a SOW. It requires no extra effort on your part to get this functionality. ##### The AMPS Actions Platform Just a quick aside. The AMPS actions platform is extremely flexible and modular. This dead letter queue is just one particular (rather simple) example of an AMPS action design pattern. The full functionality of AMPS actions is available for you to use with the expired messages. You even have full access to the message data through `amps-action-do-extract-values`. You could trigger a log rotation, a SOW compaction, write to a log, turn on your smart toaster, etc... See the [AMPS User Guide chapter on Actions](/docs/amps-user-guide/actions) for all the gory details. ### AMPS Views: the secret to unlocking the treasures of the Dead Letter Queue Other platforms offer dead letter queues. It is a common design pattern. The innovative differentiator of our queue is that it can show more than just _queue depth_. The AMPS dead letter queue is a full fledged queue, with complete access to its data. You can perform arbitrary queries and views over the data in the queue. Here is an example of the simple case: a view that shows the queue depth, last received stale message, and the time that message was received. ```xml showLineNumbers DeadLetterStats DeadLetters json COUNT(/data/id) AS /totalDeadLetters /data/id AS /lastProcessedId /received AS /lastReceived /totalDeadLetters ``` Let's see how we can spice it up! We have full access to the message data, so how about using `GROUPBY` to nicely group our dead messages: ```xml showLineNumbers DeadLetterStatsByRegion DeadLetters json /data/region AS /region SUM(/data/rumPrice * /data/rumQty) AS /totalLostRumCost /data/region ``` With this aggregated the view, the Royal Navy could very easily see the total value of Rum lost, grouped by each region. ##### Use-Case Ideas Any kind of SQL style query you can think of can be performed over your dead letter queue. Here are some possible examples to get your gears turning: - Your queue receives messages from different regions. `GROUPBY` the region identifier to determine which region is losing the most messages from the queue. - In a market order system, you can aggregate the monetary value of all the dead order messages with `SUM` to give an aggregated value for the lost messages. (This can be used as a powerful incentive metric for your team) - If `SUM` is too simplistic, you can get the average value of `/order_price` * `/quantity` over all the orders in the dead letter queue. - If you use variable length messages in your system, you can compute the `AVG` and `STDDEV_POP` over the byte size of the dead messages to get an idea of the variance of the size of the problem messages. A little bit of self promotion: check out [my earlier blog post on AMPS actions](/blog/hot-fresh-expressive-new-functions/) for an in depth discussion of the kinds of processing you can do over the data in your dead letter queue. ### Poison! a.k.a Forced Expiration in AMPS Queues #### A good pirate always checks their rum! While stale messages are an important consideration for queues, there are other cases that can necessitate the forced expiration of a message from a queue. This could be the extreme case of a message with corrupted data, or simply a message containing a request that could not be fulfilled by a consumer, such as being unable to commit the message to an external database for some reason. We refer to these as _poisoned_ messages, and as of AMPS 5.2.1.0, we have set of features that allow you to safely handle them, making your AMPS queue's even more robust! ![Poisoned Letter Disrupting a Queue](/img/blog/queue-flow-poison-opaque.png "Poisoned Letter Disrupting a Queue") #### Forced Expiration Features Let's break down the new features added to AMPS queues and explain why you will love them. First, there are two new options that have been added to the AMPS queue configuration: How they add safety to your queues? - `MaxDeliveries`: an upper bound to the number of times AMPS may deliver a queue message before automatically expiring it. - `MaxCancels`: a limit to the number of times a subscriber may cancel a lease on a message before it is expired. At first it might seem like these two limits are redundant, but they work in conjunction. `MaxDeliveries` is checked when a message is submitted to a consumer from AMPS. `MaxCancels` is checked when a message is returned to the queue. This gives you the maximum flexibility to determine the behavior that best fits your use-case. For some use-cases it's more important to check canceled messages before they hit the queue again, and for other use cases it may be fine that they re-enter the queue, but they should only be allow so many delivery attempts. You can define both `MaxDeliveries` and `MaxCancels` on the same queue. Note: Delivery is counted each time a message is delivered, so a message that is delivered and then cancelled counts both a delivery and a cancellation. In this case, a message is initially sent out, which counts as a _delivery_. When the message is then canceled, it will increment its cancel count right before it hits the queue again. `MaxDeliveries` is checked when a message leaves, `MaxCancels` is checked when a message returns. [See the AMPS User Guide for more details about `MaxDeliveries` and `MaxCancels`](/docs/amps-user-guide/queues/handling-unprocessed-messages) These new Queue limits are extremely valuable to keeping your queues operating safely and at peak performance, but that's just one part of the solution! If things do go wrong and we get slow queues or _poisoned_ messages, we need as much data as we can get to solve the problem! We got you covered on that end too! The `on-sow-expire-message` action has been supercharged with a new context variable: - `AMPS_REASON`: A comma-delimited string indicating one or more reason(s) the message was expired. Here are the values that `AMPS_REASON` can contain: - `time_limit`: This is the standard Queue timeout condition. - `max_cancels`: The message exceeded the `maxCancels` limit. - `max_deliveries`: The message exceeded the `maxDeliveries` limit. - `forced_expire`: A consumer forced the message to expire from the queue immediately. `AMPS_REASON` gives a detailed picture of why messages are being expired from your queue. And remember, you can use the full power of AMPS views and aggregation with `AMPS_REASON` inside your dead letter queue. Not only can you see which messages are dying, you can now see why they are dying, and calculate a full range of queries and statistics over that data! `forced_expire` is particularly powerful because it gives you a feedback mechanism between your consumers and AMPS. If a consumer detects some kind of problem with a message, it can `force_expire` the message, giving you a discrete flag that you can immediately take action on. For a more in depth discussion of handling _poisoned_ messages in your consumer code, check out [this excellent blog from my colleague Dirk](/blog/yuck-stateless-poison-message-handling/). A message that hit `max_cancels` could just be the result of a down-stream network slowdown, but a message with `force_expire` was a client very explicitly telling you that it could not process that message. This kind of granularity brings a whole new level of power to your monitoring infrastructure, allowing you to respond to problems faster and keep your queues running longer. Let's view some sample snippets from a config that implements this: 1) The main queue definition: ```xml showLineNumbers WorkToDo json at-least-once Work 60s 2 2 ``` 2) Let's make a `DeadLetters` queue to store these poor dead messages: ```xml showLineNumbers DeadLetters json at-least-once ``` 3) A view over the `DeadLetters` to give you those awesome metrics: ```xml showLineNumbers DeadLettersByReason DeadLetters json /reason AS /reason COUNT(/data/id) AS /numMessages /received AS /lastReceived /reason ``` 4) The action definitions to tie it all together: ```xml showLineNumbers amps-action-on-sow-expire-message WorkToDo json amps-action-do-publish-message DeadLetters json { "data" : {{ "{{AMPS_DATA" }}}}, "received" : "{{ "{{AMPS_DATETIME" }}}}", "reason" : "{{ "{{AMPS_REASON"}}}}" } ``` See the AMPS User Guide for [more details about `AMPS_REASON`](/docs/amps-user-guide/actions/on-elements/on-expire) #### Queue Semantics As another important aside, AMPS queues provide two different modes of delivery semantics. These two modes act very differently, and it's important to understand how queue performance and message expiration is affected by these modes: - `at-least-once` semantics - Messages are _leased_ to a consumer, which must acknowledge the message or the message will be automatically returned to the queue. - It is equivalent to a *don-destructive* get from a traditional queue. `at-least-once` semantics are most susceptible to _poisoned_ messages, because the bad messages can be automatically returned to the queue, continuing to cause processing bottlenecks. This semantic mode is why the forced expiration features described above were created. The advantage of `at-least-once` mode is reliability, and the force expiration `AMPS_REASON` feature allows more detailed analysis of why messages are failing. - `at-most-once` semantics - The simplest way to remember these semantics is: "fire and forget". Messages that are sent to a consumer are immediately removed from the queue and are never returned. - It is equivalent to a *destructive* get from a traditional queue. `at-most-once` semantics favors performance over reliability, and thus does not attempt to make any guarantees about whether a message was successfully processed once it has been sent. This makes it less susceptible to _poisoned_ messages, but it also gives you less insight into why messages may have failed to be consumed properly. In fact, this mode **does not** support the `MaxDeliveries` or `MaxCancels` options. `at-most-once` **only** supports timeout expiration. This makes intuitive sense since a message in an `at-most-once` queue will only ever be delivered once, and can never be canceled by a consumer. The `on-sow-expire-message` action still works with `at-most-once` queues, and dead letter queues work the same way. **The important note** is that there is only one condition in which a message will be sent to the dead letter queue: the message timed out. Note: There are many more subtle differences in behavior between these two modes of operation. [See the manual for more details](/docs/amps-user-guide/queues/understanding-amps-queuing#delivery-semantics) ### Sample Dead Letter Queue Config Finally, here is a complete working sample configuration for a Dead Letter Queue implementation in AMPS 5.2.1.0: ```xml showLineNumbers dead-letter-queue-server ampServer enabled 1s ./dead-letter-queue/stats.db localhost:8085 10s websocket-any json json 5 nvfix-tcp tcp 19090 json amps websocket-any websocket tcp 9008 file trace ./dead-letter-queue/server.log ./dead-letter-queue/journals Work json DeadLetters json Work json ./dead-letter-queue/work.sow /id WorkToDo json at-least-once Work 60s 2 2 DeadLetters json at-least-once DeadLetterStats DeadLetters json COUNT(/data/id) AS /totalDeadLetters /data/id AS /lastProcessedId /received AS /lastReceived /reasion AS /lastReason /totalDeadLetters DeadLettersByReason DeadLetters json /reason AS /reason COUNT(/data/id) AS /numMessages /received AS /lastReceived /reason amps-action-on-sow-expire-message WorkToDo json amps-action-do-publish-message DeadLetters json { "data" : {{ "{{AMPS_DATA" }}}}, "received" : "{{ "{{AMPS_DATETIME" }}}}", "reason" : "{{ "{{AMPS_REASON"}}}}" } ``` How will you use this new features of AMPS to sail the seas and find the treasure? Submit your use case for this capability here or via email and we may send you a cool t-shirt! --- # The Canary Sings! AMPS and ITRS Geneos 4.0 Simple, effective alerting can be yours with AMPS and ITRS Genos 4.0 Alert ! We released a sample AMPS plug-in for ITRS Geneos 4.0 that can be used to build real time reporting and alerting based on customizable rules and thresholds. Many of our customers are already using ITRS Geneos to consume the rich monitoring information from multiple AMPS servers and other systems to provide a real time system-wide view of health metrics. We noted that many customers seemed to be writing their integration piece in their own way so we worked with ITRS to create a generic plug-in that can be used to capture the AMPS metrics by Geneos.   After working with ITRS on Geneos 4.0 support, we are pleased to announced that the AMPS-Geneos plug in is available at this public [Github repo](https://github.com/60East/amps-integration-itrs) .  Brand Hunt, co-founder and President of 60East Technologies, explains: *”inside of our AMPS product, we give visibility into every component. With a product that does millions of transactions per second, 10's of thousands of connected client sessions, and global deployment across 1000's of servers, great monitoring is critical to the health of our customers' business. ITRS makes it easy to consume, monitor, and propagate alerts across a large number of metrics and servers, which is why it's our first recommendation for our largest enterprise customers. In fact, ITRS is the only monitoring system we've found that can take advantage of the full range of metrics exposed by our AMPS product and deliver the operational flexibility required by our most demanding enterprise customers.”*  - [ITRS Blog](https://www.itrsgroup.com/simply-put-faster-integrations) # AMPS Monitoring Whether you use ITRS Geneos, DataDog or other system -   we supply a wide range of 'canaries' or proxy metrics that are critical to preventing and detecting faults impeding service level agreements.   This blog will explore some of the best practices and means to monitoring AMPS including monitoring, logs, actions and internal events.     Let’s take a step back and provide an overview of how AMPS provides this information and how you can take advantage of it.  # Real Time System and Instance Monitoring ![blog Galvanometer](/img/blog/galvanometer.png) AMPS provides several ways of exposing monitoring metrics.   The most prominent tool is our [Galvanometer](/blog/get-more-amps-with-galvanometer/) which provides a GUI for host, system and deployment statistics as well as a query facility to inspect AMPS business content.  Just having a simple view of what clients are connected, and when the server restarted has been a great boon to operational insight. The roadmap for this tool is just full of practical capabilities but we are indeed looking for feedback on how it can help make your days more sane.  For more browser based or programmatic (API) based use cases - customers also leverage our entitlement-based browsable Admin Console  to get a similar set of metrics as provided by the Galvanometer. This is where one can browse our monitor rich statistics such as rates, connection counts and replication performance.  It also enables one to perform changes to the system such as disconnecting clients. One can also use the programmatic RESTful interface to the console to capture information from external systems.  For example, one could monitor replication statistics in this manner : ```bash watch -n 2 "wget -qO- localhost:8087/amps/instance/replication/DR_instance/seconds_behind" ``` ## Historical Data The historical information in Galvanometer and the Admin Console comes from our optional `stats.db` database which allows you to run queries that compare the stats over time. Given that we store system, instance and client level information,  it is a common use case to try to correlate what type of load the system was handling when a flood of a thousand connections arrived.  The interval of sampling can be configured to mitigate database bloat or to facilitate fine-grained detective work. In a typical operational use case, one can poll the Admin Console to produce ad-hoc alerts (and even publish back alerts to AMPS) and run end-of-day or week reports based on the statsdb content (before purging it etc). For example, from the StatsDB, to view which clients have fallen behind at one time, one can run: ```sql showLineNumbers sqlite> SELECT s.client_name, MAX(d.queue_max_latency), MAX(queued_bytes_out) FROM ICLIENTS_DYNAMIC d JOIN ICLIENTS_STATIC s ON (s.static_id=d.static_id) GROUP BY s.client_name; ``` Also, for simple external or even spreadsheet reporting, one can use the Admin Console interface to have the results of specified monitoring range sent to a csv (or text, XML, RNC etc)  which can easily be turned into excel graphs/plots: ```bash http://localhost:8085/amps/instance/processors/all/messages_received_per_sec.csv? t0=20111129T0&t1=20111129T232500 ``` ## Real Time Internal Event: Tracking whom is doing what The AMPS engine publishes metrics to internal Event Topics that provide particularly useful information about client connections and the state of topics in the SOW. > The AMPS engine will publish client status events to the internal /AMPS/ClientStatus topic whenever a client issues a logon command, disconnects, fails authentication, enters or removes a subscription, queries a SOW, or issues a `sow_delete`. This mechanism allows any client to monitor what other clients are doing and is especially useful for publishers to determine when clients subscribe to a topic of interest. [AMPS User Guide: Event Topics](/docs/amps-user-guide/events) Please note that these Event Topics can be configured to be persisted so you can then access the values over a time range. ## Devop Tookit: Actions and Logs AMPS simplifies operations by providing a set of Administrative Actions which can be defined in configuration.   With these powerful mechanisms, one can schedule automatic responses to either time, content or resource threshold based triggers.   For example, one can invoke a user script if the free storage is under 15% or set up weekly archiving or deletion of log files or even trim the database for content that is a week old.  Read more about [Administrative Actions](/docs/amps-monitoring-guide/admin) in the [AMPS User Guide](/docs/amps-user-guide). While some tail logs looking for specific messages or error levels, others leverage logging facilities such as Splunk or a Apache Flume based sink such as a Fluentd stack that ties into an HADOOP store or elastic search facility. See more about our [AMPS Flume integration](/blog/crank-up-flume-with-amps/) .   Often the goal of the system is to be able to leverage the historical logs to deem what is normal and then employ machine learning to alert based on what is unusual (i.e. increased # of client connections or 3X times memory usage).   Such systems are also used to ensure the appropriate maintaining of audit trails for regulatory purposes.    ## Business Application Monitoring: Content Limit Checking and Queue Insights At the application level, most customers either publish information to their own monitoring topic which can be consumed by AMPS clients or stored in the SOW. Others leverage our Views on application metrics to calculate aggregations or counts as well as to implement alerts or limit checks.  For example, one can easily set up complex event processing queries to capture limit checking whenever a single order has a quantity over a million or if `quantity*price  > million` or if the aggregated count of orders thus for that listing has reached a million $.   The view or aggregation can also be formed on a queue so that on can monitor or alert on the queue’s actual contents rather than just queue depth.  For example, one can provide a view over a dead letter queue to see the nature of the aggregates of source and value of the trades/orders that didn't get processed. See the [Dead Letter Queue Blog](/blog/dead-letter-queue/)  .  This feature is growing in popularity due to both the business and operational value of having insight into a queue's content.    # Enter ITRS Geneos While AMPS comes with significant and well-proven monitoring capabilities, one would use Geneos to establish a sophisticated system wide alerting and reporting platform. We have implemented a number of rules that have been recommended to customers for some time. These rules not only note system performance but also to identify when the system is not performing up to expectations.  Note that these aren't necessarily Geneos specific. Here are some examples of rules that we recommend . * `All Processors Last Active  > 20000` This detects if the message processor (i.e. the AMPS engine) has not been used which could be indicative of no input or ingestion related issue or other resource constraint. * `file_system_free_percent < 10` This is the key to trigger an alert when the file system is down to 10% of its capacity.   AMPS also has a "do" Action to provide triggered behavior based on this threshold. * `Clients queue_max_latency > 10` This will help ascertain if there is an issue due to slow consumer behavior where AMPS is detecting TCP push back while trying to send data.   * `Replication seconds_behind > 30` During replay for recovery or back testing, this indicator highlights when there is a constraint impacting the recovery steam.    - `Views queue_depth >20000` The number of messages in the view that have not yet completed processing which can be indicative of a resource limitations (complex view processing can be CPU intensive). These rules were chosen due to their prevalent use in customer environments. There are many many more rules that could be configured however their relevance depends on whether they need stats re: views, replications, the State of the World (SOW) database etc. ## For More Information: The [Monitoring Reference guide](/docs/amps-user-guide/monitoring) lists all the aspects being monitored which can be accessed via the HTTP port.  We also encourage people to use a browser to get familiar with its structure and content. The [Statistics Database Reference](/docs/amps-user-guide/amps-statistics) describes the table structure of the same statistics that you can run queries against. This also has to be enabled in the configuration file and an interval for sampling can be provided. # Alerting 60East ! We would love to hear from you about your monitoring needs.  We know that you have built and ran large systems and have faced hardware and software failures and have proceeded to build up preventative practices.    Please share them with us - we would like to continually improve AMPS monitoring and operational capabilities. Thank you! --- # Grids Without Gridlock: Which is Fastest? Learning to love the grid There are lots of reasons to choose a web interface over a native graphical interface. Web interfaces are universal, work on most devices and platforms, have very flexible and feature-rich design capabilities, and do not require any installation for users. That being said, performance is still a big concern. It is especially challenging if the data source for that application is AMPS itself -- the world's most impressive messaging system that can easily overwhelm the fastest applications. Which web grid components have enough capacity to handle millions of elements and hundreds or even thousands of updates per second? We're about to find out! ### Why and What Our AMPS product is commonly used as a View Server for high-performance financial market systems. Our customers are often looking to us for advise on which components can best meet their needs. In these View Server use-cases, data received from AMPS is often displayed in some form of a grid. This is not static data though -- the contents of the grid can change in real time, as AMPS delivers updates and new data. This article is a comparison to measure which web grids can provide a feature-rich, responsive, and memory-efficient solution to this use-case. Today, we are testing 5 different grids that promise great performance and rich functionality: - [**ag-Grid 13.2.0**](https://www.ag-grid.com/): "The Best HTML 5 Grid In The World" is claimed to be used by over 15% of the Fortune 500 companies. To make it even more interesting, we will also test its *React* and *Angular4* components, to detect how fast they perform compared to a pure JavaScript solution. ![ag-Grid table](/img/blog/grid-comparison-ag-grid.png "ag-Grid table") *ag-Grid 13.2.0* - [**SlickGrid**](https://github.com/6pac/SlickGrid/wiki): a very fast and snappy HTML grid that is designed for maximum performance. ![SlickGrid table](/img/blog/grid-comparison-slickgrid.png "SlickGrid table") *SlickGrid* - [**OpenFin's HyperGrid 2.0.2**](https://openfin.co/hypergrid/): the unique grid that is using Canvas instead of DOM to render its contents. This feature allows **HyperGrid** to handle millions of rows of data and thousands of simultaneous updates without slowing down the user interface. ![HyperGrid table](/img/blog/grid-comparison-hypergrid.png "HyperGrid table") *HyperGrid 2.0.2* - [**Webix DataTable 4.3.0**](https://webix.com/widget/datatable/): part of **Webix** framework, DataTable component provides a highly efficient grid that delivers blazing fast performance. ![Webix table](/img/blog/grid-comparison-webix-datatable.png "Webix table") *Webix DataTable 4.3.0* - [**PrimeNG DataTable 2.0.6**](https://www.primefaces.org/primeng/#/datatable): part of the popular Angular-based UI framework, it provides a rich feature set and great customization opportunities. Unfortunately, turned out it was way too slow in our tests, that's why it was excluded from the final results. Without pagination, the PrimeNG grid can only handle ~5K records without having significant performance issues. ![PrimeNG table](/img/blog/grid-comparison-primeng.png "PrimeNG table") *PrimeNG DataTable 2.0.6* All the above grids have similar functionality, such as: - Tree Tables - Filters / Sorting - Cell Editors - Various selection models - Live Updates - Styling options - Data Binding Detailed feature descriptions are available on the grids' websites. ### How Each grid in the above list claims to have incredible performance, but how would we determine who *actually* is the best? User experience is everything. The grid should look good, feel snappy, fit huge amount of data and yet still go easy on memory as it often is a limited resource. A good grid shows results of a query with subsequent updates that occur to that grid, such as deletes, updates, and new records. An average message size is **140 Bytes** -- it's big enough to show meaningful information and is small enough to fit millions of such messages into the grid without taking all the available RAM. The following set of tests should give us a good picture of how the selected grids will perform in a real world situation: - **Rendering Time**: How much time it will take to render the initial portion of data. Fast rendering is important so that the web application loads and is ready to work as soon as possible. Another situation when fast rendering might be useful is switching data sets to display in the grid. - **Frames Per Second** (**FPS**): The more FPS a grid can produce while being used, the smoother and more responsive it looks and feels. Significant changes in **FPS** are perceived as *freezes* and should be avoided as much as possible. - **Memory Consumption**: If a grid is memory efficient, it can work well and do more on a device with a low amount of RAM, such as mobile devices and laptops. In our test we will measure how many rows/records a grid can render using no more than **4 GB** of RAM. - **Live Updates**: Rendering the initial portion of data is important, but it means nothing if a grid cannot handle changes made to it in real time. According to [MDN](https://developer.mozilla.org/en-US/docs/Tools/Performance/Frame_rate), *"A frame rate of 60fps is the target for smooth performance, giving you a time budget of 16.7ms for all the updates needed in response to some event."* In this test we will measure how many rows per second we can add to the grid while maintaining maximum FPS and experience no lagging. Appending rows to the bottom of the grid is a pretty expensive rendering operation because the grid must update or change its view on every record added. ### Environment ##### Hardware - **CPU:** 6th Generation Intel® Core™ i7-6820HQ Processor (8MB Cache, up to 3.60GHz) - **GPU:** NVIDIA® Quadro® M1000M 4GB - **RAM:** 64GB DDR4 2133 MHz - **Storage:** 1TB PCIe SSD ##### Software - **OS**: Linux mint 4.10.0-35-generic #39~16.04.1-Ubuntu x86_64 GNU/Linux and Windows 10 Pro 64-bit - **Browsers**: - Google Chrome Linux 61.0.3163.100 (64-bit) - Mozilla Firefox Linux 56.0 (64-bit) - Microsoft Edge 40.15063.0.0 (64-bit)* ### Don't Make Your Customers Wait Once the initial data portion of a query is loaded (typically, in a separate WebWorker, especially if it's large), we need to display it in the grid. Considering the size of the initial portion, all grids did great, but as usual, some did better than others. ![Grid Comparison: Rendering Time -- 20,000 records](/img/blog/grid-comparison-rendering-time-20k.svg "Grid Comparison: Rendering Time -- 20,000 records") *Grid Comparison: Rendering Time -- 20,000 records* ![Grid Comparison: Rendering Time -- 200,000 records](/img/blog/grid-comparison-rendering-time-200k.svg "Grid Comparison: Rendering Time -- 200,000 records") *Grid Comparison: Rendering Time -- 200,000 records* ![Grid Comparison: Rendering Time -- 2,000,000 records](/img/blog/grid-comparison-rendering-time-2m.svg "Grid Comparison: Rendering Time -- 2,000,000 records") *Grid Comparison: Rendering Time -- 2,000,000 records* ![Grid Comparison: Rendering Time](/img/blog/grid-comparison-rendering-time.svg "Grid Comparison: Rendering Time") The winner is **SlickGrid** with the mind blowing rendering time that's independent of the number of records. The Angular4 version of **ag-Grid** had the best rendering performance among other **ag-Grid** components, possibly due to the [VM-friendly code](http://blog.mgechev.com/2016/08/14/ahead-of-time-compilation-angular-offline-precompilation/) it produces. Another issue with **ag-Grid** we faced was its limitation when it comes to the maximum height of the grid container. This lead to the situation when rendering just stopped after **1,342,166** rows. This is not exactly a problem with **ag-Grid**, just a [limitation in browsers](https://stackoverflow.com/questions/16637530/whats-the-maximum-pixel-value-of-css-width-and-height-properties) which **ag-Grid** folks [mention](https://www.ag-grid.com/javascript-grid-width-and-height/?framework=angular#gsc.tab=0) on their website, however, other grids don't suffer from it. We solved this by reducing the row height, keeping the total container height below the maximum value. ### Smooth Scroll is Important Okay, we have our data loaded and rendered. Is the grid still responsive and snappy? Can we look through these rows without having a feeling that we're watching a slide show? To make this test more objective, we were measuring FPS using Chrome Developer Tools. Scrolling using Touchpad/Mouse wheel simulates slow scrolling, while scrolling by dragging scrollbar will show how the grids are optimized for very fast scrolling. As before, we test for each dataset size. ![Grid Comparison: Scrolling FPS -- Mouse / Touchpad](/img/blog/grid-comparison-fps-scrolling-mouse.svg "Grid Comparison: Scrolling FPS -- Mouse / Touchpad") ![Grid Comparison: Scrolling FPS -- Scrollbar](/img/blog/grid-comparison-fps-scrolling-bar.svg "Grid Comparison: Scrolling FPS -- Scrollbar") This time, we have no clear winner, rather a group of "cool" kids, which are **HyperGrid**, **Webix**, and **SlickGrid**. **ag-Grid** Angular4 component had a significant FPS drop compared to its plain JavaScript and React analogues which showed good results otherwise. All grids have different mouse scrolling speeds, thus explaining the average FPS results in each case. **HyperGrid** truly fullfills its promise of [Ultra High Performance and smooth scrolling with no deferred painting for unlimited data sets](https://openfin.co/hypergrid/) -- it easily handles 2 million rows while maintaining very smooth and consistent experience. **SlickGrid** completely hides rows if scrolling is faster than some threshold value, thus making it easy to hit maximum results in the scrollbar test. **Webix** performs great with any dataset, providing the most consistent and smooth experience for both mouse and scrollbar scrolling tests while having no deferred painting. Subjectively speaking, all grids (except Angular4 **ag-Grid** component) felt very snappy and responsive. ### How Much One Can Fit in 4 GB of Memory? This a pretty straighforward test that answers one simple question -- how much data can we display in the grid on an entry level desktop? ![Memory Consumption](/img/blog/grid-comparison-memory-consumption.svg "Memory Consumption") The winner in this test is **Webix**, closely followed by **HyperGrid** which has almost similar memory efficiency. The overhead introduced by React and Angular4 can be observed for **ag-Grid**; its plain JavaScript solution performed better than its enterprise-oriented counterparts. ### It's Alive! Now the grid is loaded and rendered all of the original data, but that's not enough -- without real-time updates, the dataset becomes irrelevant. AMPS features, such as `sow_and_subscribe` command which delivers updates to the initial query set, [**Content Filtering**](/blog/not-using-content-filtering-in-your-messaging-application-youre-doing-it-wrong/), [**Preprocessing and Enrichment**](/blog/preprocessing-and-enrichment/), [**OOF**](/docs/amps-user-guide/oof) messages, and [**Conflation**](/blog/beat-the-traffic-with-conflation/) provide fine-grained control over updates. These features, combined with our official [JavaScript client](/blog/introducing-new-javascript-client/) make real-time grid updates a surprisingly trivial task, of course, if a grid *can handle* such power. ![Live Grid Updates](/img/blog/grid-comparison-live-updates.svg "Live Grid Updates") While **ag-Grid** has potential and would satisfy a small stream of updates without lagging, the winner, **HyperGrid**, performed more than **20x** better than **ag-Grid**. It seems like `HyperGrid` and its Canvas-based rendering engine truly is the future of fast web components, delivering incredible performance close to native interfaces. ### Licenses and Pricing Here we want to provide a quick overview of what licenses are available for each grid and what the prices are for commercial technical support at the time of this writing. Most grids also have free editions as well, although they are not always Open Source compatible. | ag-Grid | | | --- | --- | | Free: MIT | Commercial: Single Application: $657/year/dev Multiple Applications: $1,056/year/dev SaaS and OEM available More information here. | | HyperGrid | | | Free: MIT | Commercial: Quote-based individual pricing HyperGrid is a part of OpenFin More information here. | | PrimeNG | | | Free: MIT | Commercial: Quote-based individual pricing DataTable is a part of PrimeNG UI Suite More information here. | | SlickGrid | | | Free: MIT | Commercial: Not available | | Webix | | | Free: GPLv3 DataTable is a part of Webix Standard 63 Widgets and Controls Compatible with Open Source projects only | Commercial: DataTable is a part of Webix Standard 84 Widgets and Controls Developer Pack (single Dev): $469/year Team Pack: $1,299 /year (5 developers) Enterprise Pack: $3,999/year (20 developers) Custom quotes available More information here. | ### Development resources Development resources are not important for users, but are for developers. Great documentation and manuals can save hundreds of hours of valuable developer time otherwise spent browsing through docs, googling, or waiting for a ticket to be addressed by a grid's tech support. So far, here's our rating on how good the development resources are for the grids (subjective, yet pretty accurate): 1. **ag-Grid** -- Excellent documentation and examples. This includes materials about its Angular/React components and integration with third party software, such as **OpenFin**. 2. **Webix** -- Almost as good as **ag-Grid**. The grid provides tons of examples and live demos, API is very well documented and easy to search and navigate. Integration with Angular/React could be covered better. 3. **PrimeNG** -- Nice documentation and examples but lacks built-in search. 4. **SlickGrid** -- Documentation looks a bit fragmented. Plenty of examples and demos are available. 5. **HyperGrid** -- No manuals are available. Documentation is very basic and lacks descriptions. Most of the examples provided in the repo do not work and styling information is hard to find. The project is currently in a transitional phase; the new codebase has emerged, but infrastructure and support is falling behind. After figuring it out and creating a demo, we became very popular on Wall St as the Folks Who Figured it Out. ### Conclusion We took a look at some of the great modern web grids. Test results prove the point that web interfaces combined with modern technologies can provide a highly responsive and well performing user interface. In this testing, the winners are: 1. **SlickGrid**: A bit old, but lightning fast and very easy to use. 2. **HyperGrid**: A very promising technology that may become mainstream in the future. Second place is only due to the situation with documentation/manuals, which we believe will be improved soon. 3. **Webix**: Excellent performance in all categories, great API and documentation. Honorable mention: **ag-Grid**. It wasn't the fastest, but it was *fast enough* for most enterprise users' needs, and its focus on integration with React, Angular and other mainstream frameworks, along with tech support and excellent documentation can be the key advantages to choose this grid. Everybody had a different use case and requirement, hopefully these results can help steer you in the right direction and help manage your expectations. As you can see, some of the grids were more performant along several dimensions so that gives you some choice and ability to cater to your specific requirements. Thanks to the AMPS JavaScript client, AMPS-powered web applications can be easily built and deployed. For users who want to try these grids in action, we've prepared a [GitHub repository](https://github.com/60East/amps-blog-web-grid-bake-off) with the sample projects for each grid used in the above tests. Each project also includes the AMPS configuration file and quick start instructions. Did we miss something? Do you know a great grid we should totally try? Let us know what you think! * While **Chrome** and **Firefox** demonstrated similar performance on both Windows and Linux, **Edge** was different. On average, **Edge** performed 35% worse with **ag-Grid**, 50% worse with **SlickGrid**, had similar performance with **HyperGrid**, but, did notably better with Webix -- **3-5 times** faster than **Chrome**. We would appreciate any comments on why **Edge** works this way and how to improve its performance with other grids. --- # Protobuf: Battle of the Syntaxes fighting robots Google Protocol Buffers, or protobuf for short, is a method for serializing a message using a strict schema. AMPS has supported the Proto2 syntax of protobuf since AMPS 5.0, but up until now has not supported the Proto3 syntax for reasons we will discuss shortly. With the release of support for the Proto3 syntax in AMPS 5.2.1.0, I'd like to highlight the differences between the two syntaxes and the impact those have on AMPS. Let's start by giving a brief overview of protobuf. :::tip This post has been updated to cover the changes made in protobuf syntax 3.15. These changes remove some of the limitations in previous Proto3 versions. ::: ## What is a Protobuf? As stated earlier, Protobuf messages must follow a specific schema. Message schemas are defined in a `.proto` file. The `protoc` compiler is then used to generate a language specific implementation of your message schema. Here is an example of a `.proto` file written using the Proto3 syntax. ```protobuf showLineNumbers syntax = "proto3"; package MyNamespace; message Message { string Foo = 1; string Bar = 2; int32 ID = 3; } ``` As you can see, this schema defines a message with 3 fields, 2 strings and an integer. Fields may also consist of sub-messages that are defined in the same protofile, or any imported protofiles. If this was a schema for a Proto2 message, each field would need to be tagged with either `optional` or `required`, but more on that later. In the Proto3 syntax, everything is optional. If you do not provide a syntax, Proto2 will be used and a warning similar to the following will be displayed in the log file. ```text No syntax specified for the proto file: example.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.) ``` Now, let's go over some of the highlights of the Proto2 and Proto3 syntaxes. ## Proto2 vs. Proto3 The Proto2 and Proto3 syntaxes are quite similar, but have some distinct differences. To begin this comparison, I'm going to highlight some of the features of Proto2 that are not present in Proto3. These features include, but are not limited to: 1. Required message fields. This can be useful for things like SOW keys. 2. Ability to set custom default values for a field. 3. Support for nested groups. The Proto3 syntax contains the addition of many new features that were not present in the Proto2 syntax, as well as the removal of some existing features. These changes include: 1. Removal of required fields. 2. Removal of default values. Primitive fields set to a default value are not serialized. 3. Addition of JSON encoding instead of binary protobuf encoding. 4. Extensions have been replaced by the Any type. 5. Addition of well-known type protos. These include any.proto, timestamp.proto, duration.proto, etc. 6. Strict UTF-8 checking is enforced. At the time of writing this, several Proto3 features have been back-ported to Proto2. These features are the following: 1. Addition of Maps. 2. Addition of a small set of standard types for time, dynamic data, etc. A complete list of the changes between Proto2 and Proto3 can be found in the Proto3 [Release Notes](https://github.com/google/protobuf/releases/tag/v3.0.0). ## Limitations of Protobuf Deciding which version of protobuf to use really comes down to your application's needs. Both versions of Protobuf have limitations in AMPS. let's take a look at the restrictions and differences that apply to Proto2 and Proto3. 1. A protobuf message can be an `UnderlyingTopic` for a View, but it cannot be the outbound message type. 2. Subscriptions to AMPS internal topics cannot be a Protobuf message. An example of this would be the `/AMPS/ClientStatus` topic. 3. The Protobuf module will not be loaded into AMPS if your host does not have GLIBC version 2.5 or newer. In short, these restrictions exist because of the strict, fixed definition of a message. Prior to Protobuf 3.15, Proto3 message types could not be used with `delta publish` or `delta subscribe`. This was because Proto3 has fixed default values, and does not serialize any field that matches the default. This left AMPS in a situation where it was unable to determine if the omitted field was intentionally left out, or if it was intentionally set to the pre-defined default value. As of Protobuf 3.15, Support for explicit field presence tracking was added to Proto3 via the `optional` keyword. With the addition of this feature, AMPS can determine if a field was intentially left out or explicitly set to a default value. This allows AMPS to support `delta publish` and `delta subscribe` on Proto3 fields that utilize the `optional` keyword. ```protobuf showLineNumbers syntax = "proto3"; message Customer { int32 id = 1; // Tracks whether phone was explicitly supplied—even when set to 0. optional int64 phone = 2; // Tracks absent versus explicitly set to an empty string. optional string name = 3; } ``` A few things to keep in mind about Proto3 `optional` semantics. Proto3 scalar fields without the `optional` keyword have implicit presence. AMPS treats these as required for delta processing, so they appear in delta messages even when unchanged. For AMPS delta behavior, fields intended to be independently omitted should be explicitly labeled optional, including nested message fields. Repeated fields and maps do not track presence and therefore cannot express AMPS delta semantics. This is important to keep in mind should you decide to convert from Proto2 to Proto3. With the exception of these limitations, Protobuf is a fully supported message type inside of AMPS. ## Choosing a Syntax It is hard to make a general recommendation of which version of protobuf to use, but there are some recommendations. For those of you already using Proto2 successfully, it is recommended that you stay with the Proto2 syntax since there is some API incompatibility between Proto2 and Proto3. Beyond that, I have created a few questions that you must ask yourself before choosing a syntax: 1. Do you intend on using delta messaging? 2. Are there any fields that you require in every message, and would like to fail if it is not there? 3. Do you have a need to assign custom default values for each field? 4. If a default value is received, custom or otherwise, would you need to know if it was omitted or set to the default? If you require any of these features, then you'll need to either use Proto2 or use Proto3 with `optional` fields. If your application's needs do not include these features, then the choice is yours. ## Configuring AMPS In order to configure AMPS to use protobuf, you will need to import a `.proto` file containing the schema for the messages that will be sent over a transport. This means that you will need to define a `MessageType` for each `.proto` file that you intend to use. If your `.proto` file uses the protobuf method of including another `.proto` file, then you will only need to specify the main `.proto` file. That being said, all `.proto` files must be located at one of the `ProtoPath` locations. Here we only have one `ProtoPath`. ```xml showLineNumbers my-protobuf-messages protobuf proto-archive;/mnt/shared/protofiles proto-archive/person.proto MyNamespace.Message ``` This example, taken from the AMPS User Guide, shows a standard way to define your protobuf message type. AMPS will import the `.proto` file and identify the syntax of the file automatically. There is no change to the configuration between the Proto2 and Proto3 syntaxes. ## Publishing a Message In order to publish a protobuf message into AMPS, you must first create and serialize the message. This is done using the protobuf generated files. Once you have serialized your data, publish it to AMPS like you would any other message type. Here is an example using python: ```python showLineNumbers # Import the generated file import person_pb2 # First we need to create an object of the protobuf message protobuf_record=person_pb2.Message() # Now we need to populate it protobuf_record.Foo = "60East" protobuf_record.Bar = "AMPS" protobuf_record.ID = 60 # Serialize the message serialized_message = protobuf_record.SerializeToString() # Let's publish it client.publish("my-protobuf-messages-topic", serialized_message) ``` ## Closing Thoughts Though AMPS supports both Proto2 and Proto3, Proto2 is more feature rich due to restrictions of the message type. Both versions of protobuf have their advantages and disadvantages, and the best choice really does come down to the needs of your application. If you are already successful with Proto2 and do not require a feature added in Proto3, then you should stick with Proto2. If you're just starting out with Protobuf and the restrictions don't affect you, then you should consider Proto3. Much of the information provided here and more is available in the [Protobuf](/docs/amps-user-guide/message-types/protobuf-message-types) section of the [User Guide](/docs/amps-user-guide). --- # You Shall Not Pass: Banning Misbehaving Clients with fail2ban close up of rusty hammer One of the most enjoyable parts of AMPS is how easy it is to create a client, connect to an AMPS instance, and start building an application. In just a few minutes, you can have applications communicating through AMPS and start working out your application's message flow. (In fact, we've demonstrated live-coding a basic chat application from scratch in under 30 minutes!) If you're responsible for keeping a common development instance of AMPS running, that joy can sometimes turn to frustration as misbehaving applications can end up consuming resources on the instance. In extreme cases, misbehaving applications can consume enough resources that other applications are affected. We've seen instances where a misconfigured application generated millions of failed connections in less than two hours. While AMPS did a reasonably good job of managing the flood, the overhead involved in managing those failed connections consumed bandwidth and CPU time that would have been better spent doing real work for the instance. There's a simple solution, though. fail2ban: Better than a Wizard ------------------------------- Managing misbehaving connections isn't a problem unique to AMPS. Web servers, SMTP servers, ssh servers, databases -- any software that accepts connections over a network needs to handle this problem. Ideally, with as little CPU overhead and network overhead as possible. In fact, most modern Linux distributions already provide a service that does this protection: `fail2ban`. Commonly used to protect critical services like SSH access, Apache, ngnix and so on, `fail2ban` has exactly the features we need to protect an AMPS instance. The idea behind `fail2ban` is simple. The `fail2ban` service monitors logs for entries that indicate a problem from a specific remote IP address. If the number of problem entries exceeds a configured threshold, `fail2ban` updates the firewall rules for the system to block access from the problematic system to the affected port for a period of time. With some simple configuration, you can easily configure `fail2ban` to protect AMPS instances. There are a few pieces of the recipe that we need to use `fail2ban`. * __Setting policy__ First, we need to decide what we consider to be behavior that we want to protect AMPS from. * __Logging events__ Then, we need to log the information that we'll use to identify when connections are misbehaving. * __Identifying problems__ Next, we need to let ``fail2ban`` know what events should be considered bad behavior. * __Configuring protection__ Finally, we'll configure how ``fail2ban`` protects AMPS from bad behavior. Setting policy -------------- For the purposes of this blog, we'll set a policy that protects AMPS from two common problems: repeated connections from clients that have the same name ("name in use" collisions) and repeated connections from clients that do not successfully authenticate to AMPS. When we see a client misbehaving in either of these ways, we want ``fail2ban`` to block connections from that client for two minutes. If a client has more than 20 failures due to entitlement or name in use in 10 seconds, we'll trigger the ban. Logging Events --------------- Now that we have a policy to enforce, we need to capture the events for fail2ban. To enforce the policy we just set, we need to know when a client disconnects and why that client disconnected. For `fail2ban` to successfully ban the client, we'll also need information on the IP address that the client is connecting from. Recent patch versions of AMPS 5.2 (that is 5.2.1.37, 5.2.0.87 and subsequent versions) contain all of the information we need for this monitoring in the `07-0013` log message. To capture this information in a place that's easy for fail2ban to monitor, we add a distinct logging target to AMPS configuration. ```xml showLineNumbers file 07-0013,00-0015 /var/log/amps/ban.log 1MB ``` Notice that there's no logging level specified in this `Target`. The `ban.log` file will _only_ contain the specific error that we need fail2ban to monitor, `07-0013`, and `00-0015`, which ensures that AMPS will write at least one event to the logging file, even if there are no disconnects happening. Also notice that the file has a consistent filename, so ``fail2ban`` can easily find it, and that AMPS will write 1MB of logs and then roll the log over onto the same file name: AMPS will keep the file size at approximately 1MB. This file captures disconnect events from the server. For example, the following set of events shows three clients disconnecting from the server. ```bash showLineNumbers 2017-12-07T16:55:49.9395590-08:00 [26] info: 07-0013 client[queue-reader-dirkm](192.168.0.7:50336) disconnected: connection closed. 2017-12-07T16:56:12.2582980-08:00 [26] info: 07-0013 client[daily-report](192.168.1.1:50344) disconnected: connection closed. 2017-12-07T16:59:57.2596670-08:00 [27] info: 07-0013 client[fast-message-loader-3218](192.168.2.0:50342) disconnected: connection closed. ``` The states that we're looking for are the `auth` state (indicating that a client isn't authorized to log on) or the `name in use` state (indicating that a client was disconnected due to the name being in use). Identifying Problems -------------------- Once AMPS is recording messages in a format that `fail2ban` can consume, the rest of the configuration is easy. The log message is built to be relatively easy to scan for, so the filter to `fail2ban` is simple. Create this `amps.conf` file in the `/etc/fail2ban/filter.d` directory (or the equivalent directory for your Linux distribution): ```bash showLineNumbers # Fail2ban filter for AMPS (action-based file format) [INCLUDES] before = common.conf [Definition] # Notice that you could also update the regex to # track specific reasons for disconnection (for # example, you could specify that fail2ban only # tracks name in use or entitlement failures). failregex = info: 07-0013.*\(:[\d]+\) disconnected: (auth|name in use) ``` The regular expression here indicates that a 07-0013 message with a disconnection reason of `auth` or `name in use` is an event that should be monitored. Further, the IP address for the event is the first part of the `host:port` string that is in parentheses just before the word `disconnected`. Configuring Protection ----------------------- Now that we have a rule that tells `fail2ban` how to identify problem disconnects, all that's left is to configure `fail2ban` to protect the host from the problem. We add an `amps` rule to `/etc/fail2ban/jail.local` (or the equivalent file in your distribution). That file defines rules that are customized for the local system, rather than provided by default with the distribution. ```bash showLineNumbers [amps] enabled = true port = 9007 logpath = /var/log/amps/ban.log maxretry = 20 bantime = 120 findtime = 10 ``` The name at the top of the section, `[amps]`, matches the name of the rule file we created for AMPS in the `filter.d` directory. The `maxretry` setting specifies the number of failures that a given host is allowed to have during the `findtime` period. The `bantime` specifies how long to block connections. The settings here say that if an IP address has logged 20 or more disconnect messages in the last 10 seconds, ban that IP address for 120 seconds. Naturally, you can tune these settings to best fit your installation. The ban applies to the specified ports -- if your AMPS instance uses different (or additional) ports, replace the port parameter in the configuration. Easy As A Banhammer =================== That's all there is to it. Restart fail2ban, and misbehaving clients will no longer be able to disrupt your AMPS instance. --- # Meltdown and Spectre Performance Implications Spectre and Meltdown logos Over the last several days, the technology world has been focused on the impact of the Meltdown and Spectre vulnerabilities. There are several good articles published about these vulnerabilities, among them [coverage from The Register](http://www.theregister.co.uk/2018/01/02/intel_cpu_design_flaw/) and [an overview from Red Hat](https://www.redhat.com/en/blog/what-are-meltdown-and-spectre-here%E2%80%99s-what-you-need-know). In all of these discussions, there's a common thread: the kernel fixes for these vulnerabilities will carry a performance cost. The question is -- how much of a cost? Now that some of the patches are available for Meltdown and Spectre, we're able to provide guidance on how these patches will impact performance critical AMPS deployments. Let's start by saying that the degradation you'll see from these patches is highly dependent on your workload and features of AMPS you're using. Red Hat's Performance Team has provided guidance on workloads and the expected performance degradation which I'm copying here for easier reference: [https://access.redhat.com/articles/3307751](https://access.redhat.com/articles/3307751). In order to provide more detail, Red Hat's performance team has categorized the performance results for Red Hat Enterprise Linux 7, (with similar behavior on Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 5), on a wide variety of benchmarks based on performance impact: > * _**Measureable: 8-19%** - Highly cached random memory, with buffered I/O, OLTP database workloads, and benchmarks with high kernel-to-user space transitions are impacted between 8-19%. Examples include OLTP Workloads (tpc), sysbench, pgbench, netperf (<256 byte), and fio (random I/O to NvME)._ > >* _**Modest: 3-7%** - Database analytics, Decision Support System (DSS), and Java VMs are impacted less than the "Measurable" category. These applications may have significant sequential disk or network traffic, but kernel/device drivers are able to aggregate requests to moderate level of kernel-to-user transitions. Examples include SPECjbb2005, Queries/Hour and overall analytic timing (sec)._ > >* _**Small: 2-5%** - HPC (High Performance Computing) CPU-intensive workloads are affected the least with only 2-5% performance impact because jobs run mostly in user space and are scheduled using cpu-pinning or numa-control. Examples include Linpack NxN on x86 and SPECcpu2006._ > >* _**Minimal: <2 %** Linux accelerator technologies that generally bypass the kernel in favor of user direct access are the least affected, with less than 2% overhead measured. Examples tested include DPDK (VsPERF at 64 byte) and OpenOnload (STAC-N). Userspace accesses to VDSO like get-time-of-day are not impacted. We expect similar minimal impact for other offloads._ AMPS Workload Estimates ----------------------- **Transaction Log/Message Queues/Replication:** Use cases with a Transaction Log concerned with performance are typically using a PCIe or NVMe storage device and involve significant networking, which crosses into the kernel space frequently. Our performance simulator for using AMPS within a large trading system sees just under a 12% performance degradation (both in maximum achievable throughput and increase in median latency), however this extreme simulation spends significant time in the kernel and minimal time in user mode. We expect these use cases to fall in the range between upper-end of the **Modest** and the lower end of the **Measurable** impact range of 5-12%. **Large Scale View Server** Where AMPS is spending most of its time executing user mode code (SOW queries, content filtering, delta publish/subscribe, etc.) should see an impact in the **Small** range. Other Considerations and Mitigation ----------------------------------- **PCID support:** To minimize the impact of these patches, verify your systems have **PCID** enabled. Without this feature, the patches will yield a higher performance impact. You can verify your host environment has **PCID support** by verifying the `pcid` flag is reported in your `/proc/cpuinfo` flags row (or in your `lscpu` flags output if you have that command available): ```bash $ lscpu | grep pcid Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer xsave avx f16c rdrand lahf_lm abm epb invpcid_single spec_ctrl ibpb_support tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts ``` **Kernel Bypass Technologies:** Using Kernel bypass technologies such as OpenOnload is a great way of minimizing the impact from these patches. If you're already using these technologies, then your impact from these patches will be far less than those not using them. What's Next ----------- We'll keep this post updated with any new findings or suggestions. If you have any questions, please don't hesitate writing comments below. --- # Easy Authentication and Entitlements elegant key and keyhole One of the most common requirements for AMPS instances is integration with an enterprise security system. In this blog post, we'll show you the easiest way to get an integration up and running -- by building an authentication and entitlement system from scratch! In versions of AMPS prior to 5.0, the only way to integrate with an enterprise system was to create a server module to handle authentication and entitlement. AMPS version 5.0 and later include an optional module that can use a RESTful web service for authentication and entitlement. Using this module can be the easiest way to integrate into an existing system. In this blog post, we'll: 1. Explain how the module works 2. Show you how to configure AMPS to use a simple authentication web service 3. Give you a simple sample implementation of a web service using Flask That might sound like a lot of ground to cover, but don't worry -- this is a lot simpler than you might expect, and the Flask framework in this blog is a great starting point to use for a production-ready system, whether you're developing a standalone service or a fully-featured integration with an existing system. All of the code discussed here is available in a public [github repository](https://github.com/60East/amps-blog-easy-http-auth). And to give credit where credit is due, the technique for using Basic authentication with Flask is developed from a [post by Armin Ronacher](http://flask.pocoo.org/snippets/8/) on the [Flask snippets](http://flask.pocoo.org/snippets) site. ### AMPS Authentication and Entitlement The AMPS authentication and entitlement system is designed to be straightforward. _Authentication_ happens when a connection logs on. The request for authentication contains a user name and a token (perhaps a password, perhaps a certificate, perhaps a single-use token). _Entitlement_ happens when a particular user name _first_ requests a particular type of access to a particular resource. (For example, reading data from a particular topic by subscribing to the topic, or publishing to a particular topic.) After AMPS has checked entitlements with the module once, AMPS caches the results for future use. So how do those steps translate to a RESTful web request? ### Making it RESTful The RESTful authentication and entitlement system does two things: 1. Converts an AMPS `logon` command into an HTTP request for a permissions document, using the credentials in the `logon` command. 2. Parses the permissions document and uses the permissions in that document to grant entitlements when AMPS requests an entitlement check. That's all there is to it. (The full documentation for the module is in the [Authentication and Entitlement using a Web Service](/docs/amps-user-guide/securing/http-auth-module) chapter of the [User Guide](/docs/amps-user-guide).) ### Basics of Basic Authentication The module supports both Basic and Digest authentication. You can read more about these in [RFC7617](http://tools.ietf.org/html/rfc7617) for Basic authentication, and [RFC7616](http://tools.ietf.org/html/rfc7616) for Digest authentication. For this post, we'll look at Basic authentication. In Basic authentication, the web server (in this case, our web service) denies access to resources unless the requestor (in this case, AMPS) provides an appropriate Authorization header. If the Authorization header isn't present, or the credentials aren't accepted, the web server returns a `401` status with a `WWW-Authenticate` header that indicates that it accepts Basic authentication (and provides a token that indicates the scope of the authentication request). It's common for a requestor to send a request without credentials, then use the response from the server to decide what sort of authentication to use. The AMPS web service module uses this approach. When the web server indicates that it uses Basic authentication, the module returns a Authorization header with the credentials for the logon request. ### GET It Done To be able to pass credentials, the module needs to know the contents of the logon request, and the URI to use to retrieve the permissions document. The URI is set in the module configuration (described below). The module allows you to substitute the _user name_ of the user logging in to AMPS at any point in the URI. To convert the logon request into an HTTP request, the module takes the user name and password and uses those to authenticate the HTTP request. The module supports both Basic and Digest authentication. For example, AMPS receives a logon command along these lines: ```javascript showLineNumbers { "c":"logon", "cid":"1", "client_name":"subscriber-AMPSGrid-Falken-host26", "user_id":"falken", "pw":"joshua", "a":"processed", "v":"5.2.1.4:java" } ``` The _user name_ used for HTTP authentication will be `falken`, and the _password_ used for HTTP authentication will be `joshua`. If the module is configured to use `http://internal-webhost:8090/{{USER_NAME}}.json` as the URI, the module substitutes `falken` for the `{{USER_NAME}}` token to request `/falken.json` from the webserver at `internal-webhost:8090`. To determine whether to use Basic or Digest authentication, the module first sends an unauthenticated request to the server. The server responds with a `401` response that contains a `WWW-Authenticate` header, as described above, and AMPS responds with an authenticated request along these lines: ```bash showLineNumbers GET /falken.json HTTP/1.0 User-Agent: AMPS HTTP Auth/1.0 Accept: */* Authorization: Basic ZmFsa2VuOmpvc2h1YQ== ``` Notice that the `Authorization` header contains a base64-encoded representation of the username and password provided on logon. ### Putting It Together: AMPS Configuration With that background, let's configure AMPS to use the web service module for authentication and entitlement. First, load and configure the module: ```xml showLineNumbers web-entitlements libamps_http_entitlement.so http://localhost:8080/{{USER_NAME}}.json ``` Then, specify that the module is used for authentication and entitlement for the instance. ```xml showLineNumbers web-entitlements web-entitlements ``` With this configuration, AMPS will use the web service module for authentication and entitlement for all connections to the instance, and will be looking for a service at port 8080 on the local machine. All we need now is a web service that can use basic authentication and serve up permissions documents! ### Service, Please! Coding the Web Service While the web service module works perfectly well when requesting static documents from a web service such as Apache or nginx, for the purposes of this post, we'll show how simple it is to use a standard HTTP application framework to build the web service. We'll use [Flask](http://flask.pocoo.org/) for this sample, since it's readily available (and we like Python!) Further, the extension points for Flask are straightforward, which makes it very handy for demonstration purposes. To create the service, we first import the things we'll need from Flask and define the service: ```python showLineNumbers from flask import Flask, request, Response from functools import wraps # create the flask app example app = Flask(__name__) ``` Next, we'll need to define a function to return a `401` response that requests Basic authentication: ```python showLineNumbers def authenticate(): """Sends a 401 response that enables basic auth""" return Response( 'Could not verify your access level for that URL.\n' 'You have to login with proper credentials', 401, {'WWW-Authenticate': 'Basic realm="AMPS Authentication"'} ) ``` This function uses Flask to return a `401` response with an appropriate `WWW-Authenticate` header and a helpful message. We also provide a function that can wrap a response: this function checks to see if the credentials provided are valid by calling a `check_auth` function. If so, the wrapped function is called. If not, the wrapper calls the `authenticate` function to return the `401` response. ```python showLineNumbers def requires_auth(f): @wraps(f) def decorated(*args, **kwargs): auth = request.authorization if not auth or not check_auth(auth.username, auth.password): return authenticate() return f(*args, **kwargs) return decorated ``` This function makes it easy to validate credentials before providing a permissions document. We use `wraps` from the `functools` module to create a `@requires_auth` decorator for a function. What this means in practice is that, once we've defined this function, we can easily require successful authentication for a function by adding the `@requires_auth` decorator on a function. The Flask infrastructure handles decoding and parsing the `Authentication` header, so all we have to do is pass the values along to the `check_auth` function. Now, the infrastructure is in place -- all we have to do is write two functions: `check_auth` to verify the username and password, and a function to return a permissions document given an authenticated username. For sample purposes, we just hard code the username and password in the authentication function: ```python showLineNumbers def check_auth(username, password): """This function is called to check if a username/password is valid.""" return username == 'falken' and password == 'joshua' ``` Of course, in a real system, this check could do anything that it needs to, including calling out to an external system to verify the credentials. Last, we define a function that, given an authenticated user name, returns the permissions for that user. For demonstration purposes, we just return a hard-coded JSON document. We use the wrapper defined earlier to ensure that control only enters this function if the username and password are validated by `check_auth`. ```python showLineNumbers @app.route('/.json') @requires_auth def permission_document(username): return """ { "logon": true, "topic": [{ "topic": ".*", "read": true, "write": true }], "admin": [{ "topic": "/amps/administrator/.*", "read": false, "write": false }, { "topic": ".*", "read": true, "write": true }] } """ ``` This function is called for any HTTP request that matches the `/.json` pattern. The function is only invoked after the `requires_auth` wrapper accepts the credentials in the request. This permissions document grants `logon` access to the instance, and allows full read and write of any topic. For the admin console, the document allows access to any resource except those under the `/amps/administrator` path. As usual, the full syntax for the permissions document is described in the [User Guide](/docs/amps-user-guide/securing/http-auth-module#permissions-document-format). Last, we need to start the server when the script is loaded: ```python showLineNumbers if __name__ == '__main__': app.run(host='0.0.0.0', port=8080) ``` ... and that's it! With those few lines of configuration and code, we have a working authentication and entitlement system. ### Would You Like To Play AMPS? With AMPS configured as shown above and the simple Python script running, we can try out the authentication and entitlement system (remember, both the configuration and script are available from the [github repository](https://github.com/60East/amps-blog-easy-http-auth).) We first start AMPS: ```bash showLineNumbers $ ampServer config.xml AMPS 5.2.1.92.210270.984c1d9 - Copyright (c) 2006-2017 60East Technologies Inc. (Built: 2018-03-17T23:01:09Z) For all support questions: support@crankuptheamps.com ``` And then start the authentication and entitlements service: ```bash showLineNumbers $ python auth/main.py * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit) ``` Then, you can use the `ping` command in `spark` to test the credentials. First, try with valid credentials. ```bash showLineNumbers $ spark ping -server falken:joshua@localhost:9007 -type json Successfully connected to tcp://falken:*@localhost:9007/amps/json ``` Then, try with credentials that the authentication and entitlements service doesn't recognize: ```bash showLineNumbers $ spark ping -server matthew:ferris@localhost:9007 -type json Unable to connect to AMPS (com.crankuptheamps.client.exception.AuthenticationException: Logon failed for user "matthew"). ``` It's just that simple! ### Where Next? You can use this Flask framework as the foundation for a more fully-featured system by replacing the `check_auth` and `permission_document` functions. If you need to integrate with a back-end system, you can use the `check_auth` function to check the credentials with that system and respond accordingly. Likewise, the permission document format has been designed to be easy to construct, so you can translate whatever format your existing permissions system uses into the permissions document provided back to AMPS. Or, if you prefer another framework for creating an HTTP server, you can use the basic pattern here to quickly implement a server in your framework of choice. It's up to you! What HTTP framework do you prefer, and what authentication and entitlement systems are you integrating with? Let us know! --- # Happy Birthday to AMPS! candles with the number 10 This month marks the 10 year anniversary of AMPS being deployed into production environments, helping to fuel the global financial markets. Those first customer deployments built on AMPS are still in production, and are still critical infrastructure today! Since then, AMPS has become a key part of critical trading flow at the top financial institutions. We've done this thanks to our close relationships with users. With their partnership and investments into our R&D, we've grown AMPS to the product it is today. The secret to our success is simple: * Partner with the developers using AMPS and the operations teams supporting AMPS * Consider the whole platform when making changes * Stick to [our principles](/blog/welcome-the-amps-philosophy/) * Every engineer works with customers We have some exciting things coming up to help take AMPS into the future. In the meantime, though, we wanted to take a minute to say THANK YOU to all of the people who have partnered with us to build their most important infrastructure on AMPS. We couldn't have done this without you -- and the next decade will be even better! --- # Managing Large Topics in the SOW > Learn how to manage SOW topics that are larger than physical memory. One of the most popular features of AMPS is the State-of-the-World (or _SOW_), which allows applications to quickly retrieve the most current version of a message. Many applications use the SOW as a high-performance streaming database, quickly retrieving the current results of the query and then automatically receiving updates as changes occur to the data. For maximum performance, 60East recommends making sure that all topics in the SOW (including views and conflated topics) fit into memory. Some applications, though, require larger data sets. We've seen applications in production with data sets several times as large as physical memory. We've successfully tested SOW topics as large as 8TB on a system with 128GB of memory (a topic that is more than 60 times as large as physical memory). AMPS version 5.2.2 introduces a new algorithm for State-of-the-World memory utilization that can greatly improve the memory efficiency and decrease the memory pressure of large SOW topics on hosts. This new algorithm applies to persistent SOW topics covered by the Transaction Log. File-backed SOW topics that are not covered by a Transaction Log are memory mapped such that as a SOW topic grows beyond the available physical memory of the host environment, the OS will "page" the SOW topic data from memory **out** to the storage device and from the storage device **into** memory on-demand. SOW topics covered by a Transaction Log are mapped to memory as "private" mappings which counts any modified topic memory as "anonymous dirty" data that counts towards AMPS' Out-of-memory score (aka "OOM score") and puts the system at increased risk of swapping. Prior to AMPS version 5.2.2, Transaction Log covered SOW topics were left in the dirty state once modified, limiting the growth of a SOW topic to the amount of memory available to the AMPS process (typically the amount of physical memory on the host plus configured swap memory). Starting with version 5.2.2, Transaction Log covered SOW topics are "cleansed" periodically as AMPS flushes the pages to disk manually. This "anonymous clean" data does not count towards AMPS' OOM score and thus allows AMPS SOW topics to far exceed the memory available on the host. ## Considerations for Large SOW Topics Having SOW topics that far exceed the host memory works great for many use cases, but you must be careful, since there are many things that need to be considered: __Decreased Publish Performance__: When publishing to a topic that isn't currently resident in memory, there could be significant VM "page-outs" or "page-ins" that impede publishing performance. Capacity planning up to the host memory limit could show acceptable performance that quickly degrades once the SOW topic exceeds the size of the physical memory of the host environment. __Decreased Query Performance__: When querying data where the query result contains records that are not resident, the OS pages those records into memory before the query result can be constructed and sent to the querying client. Additionally, AMPS auto-indexes fields not previously queried, so executing a query with a new field XPath in a content filter can force a re-indexing of a SOW topic, creating extensive "page-in" activity while the SOW data is brought into resident memory to be parsed and indexed. __Increased Recovery Times__: When AMPS starts, it checks the CRC of every record within a SOW topic to guard against corruption. To do the CRC check, AMPS needs to bring the SOW topic into memory. If you have a 1TB SOW topic stored on a fast storage device that can read 1GB/s, it could take 17 minutes (or more) for AMPS to recover the SOW topic. ## Tuning Tips If you're contemplating running large SOW topics that exceed the memory of your host environment, here are some tips on how to tune your host environment and configurations to get the optimal performance out of your large SOW use. ### Hardware __Use Fastest Available Storage__: We recommend you place the SOW topic on the fastest storage available in your environment to minimize the paging costs and recovery times. It's expected to have increased VM paging activity when in these configurations, so the faster the storage device where these large topics are persisted, the better the performance will be. See [Is Your App Solid? (Does it need SSD?)](/blog/is-your-app-solid/) for a discussion on storage options for AMPS applications. ### Operating System __Apply Recommended OS Tunings__: There are OS tuning parameters that can benefit large SOW use cases, which we list here. Engage your system administration teams to set these when appropriate. __Turning off Transparent Huge Pages__: Transparent Huge Pages can carry a huge cost for memory management, we recommend disabling it or setting it to "madvise". ```bash $ echo never > /sys/kernel/mm/transparent_hugepage/enabled ``` __Reduce Swappiness__: This setting controls the preference of dropping non-dirty pages over swapping. With large SOW files, we always want to prefer dropping rarely touched, non-dirty pages over the system swapping. Therefore, we set this value to the lowest setting without disabling it entirely. ```bash $ sudo sysctl -w vm.swappiness=1 ``` __Increase Minimum Free Bytes__: Interrupt handlers and other critical OS components require memory always being available. In large, enterprise host environments the default values controlling the minimum free for these critical operations is often not large enough. We recommend setting this value to 1% of the available memory rounded up to the nearest 1GB. ```bash $ sudo sysctl -w vm.min_free_kbytes=2097152 ``` __Disable Zone Memory Reclaim__: AMPS is already NUMA aware for it's low-latency components and doesn't benefit from the built-in NUMA optimizations within the Linux OS for large SOWs. We recommend turning off the Zone Memory Reclaim feature. ```bash $ sudo sysctl -w vm.zone_reclaim_mode=0 ``` __Increase Maximum Memory Mappings__: One of the thresholds set in the Linux OS configuration that can cause requests for more memory to fail prematurely is the `max_map_count`. When set too low, AMPS can fail to increase the size of a SOW topic. Increasing this value allows us to grow further. ```bash $ sudo sysctl -w vm.max_map_count=500000 ``` ## AMPS Configuration __Use a Large `SlabSize`__: Using a large `SlabSize` for your SOW topic can minimize the overhead of memory map maintenance and rate of SOW topic file extensions when growing. We recommend using 1GB slab sizes for any of your large topics. __Use a `Durability` of `persistent`__: For large topics, you want to use topics with `persistent` durability (the default). Using `transient` durability topics that are large will increase swap usage and be limited in size to the sum of the physical and swap memory on the host. ## Monitoring When AMPS version 5.2.2 or greater is running on a Linux OS with kernel version 3.14 or greater, it is recommended you monitor how much memory the host has available before swapping. This metric is exposed in the AMPS HTTP Admin in the /amps/host/memory/available path. ## Testing When testing the performance of large SOWs on runtime and recovery performance, it's easy to be fooled by effects of the page-cache that make test results inconsistent between consecutive runs. When testing for recovery performance, we advise that you purge the page caches between AMPS lifetimes to achieve worst-case testing with your data, AMPS version, and host environment. Dropping the Linux page caches require root privileges and can be done as follows: ```bash $ echo 1 > /proc/sys/vm/drop_caches ``` ## Go Big The AMPS State-of-the-World is suitable for large caching and computation modeling applications, for example market data caches or XVA modeling. In this post, you've seen the best practices and tuning recommendations 60East has developed over the course of working with many large data applications. For any system, we highly recommend capacity planning to ensure that the resource needs of the system are well understood, and we recommend spending time carefully testing the throughput and capacity of the system. As always, if you have questions about how best to design your application or configure AMPS, let us know at [support@crankuptheamps.com](mailto:support@crankuptheamps.com). --- # Metadata Magic with New AMPS Functions magician holding a magic top hat and wand From the beginning, AMPS has been content aware. Most AMPS applications use content filtering, and features like the State-of-the-World, delta messaging, aggregation, and message enrichment all depend on AMPS being able to parse and filter messages. The key to content filtering and message manipulation is the [AMPS expression language](/docs/amps-user-guide/amps-expressions). The expression language provides a format-independent way of working with content, and is extensible with server-side plugin modules. As useful as the expression language is, up until AMPS 5.2.4.1 AMPS expressions could only refer to: * The contents of the message itself (using XPath-based identifiers), *or* * Context-free functions (like `UNIX_TIMESTAMP()`) These two types of functions are enough for working with data, but for monitoring and administration, it's sometimes more important to be able to describe the message itself, rather than being limited to the content of the message. For example, it was difficult to filter a subscription to just publishes being submitted by a specific publisher or from a specific set of IP addresses or below a certain payload size. Enter the client and message functions of AMPS 5.2.4.1! With these functions, you can write expressions that describe the message itself or the connection that published the message. These aren't necessarily functions that every application needs. On the other hand, if you've ever tried to quickly find whether a message has been read from a SOW topic, or you want to get all the messages that a specific publisher is submitting, the new functions make magic possible! In this post, I'll show how to use some of these functions to answer some common requests. (Full descriptions of these functions are available in the [AMPS User Guide](/docs/amps-user-guide/builtin_functions/amps-function-overview).) The post will focus on the following functions: | Function name | Description | | --- | --- | | LAST_UPDATED() | The last time a message in a SOW topic was published to. | | MESSAGE_SIZE() | The size, in bytes, of the data within a message. | | REMOTE_ADDRESS() | The address (typically IP address) of the connection executing the command. | | USER() | The authenticated user name of the user executing the command. | ### Finding Older Messages One common request is to be able to find messages in a SOW topic that haven't been updated for a certain interval of time. Before AMPS 5.2.4.1, being able to run this query would depend on adding a `timestamp` field to the data (typically through enrichment). With AMPS 5.2.4.1 (and later), though, we can use the `LAST_UPDATED()` function to access the last time the record was updated. With the `UNIX_TIMESTAMP()` function, it's simple to write a query that finds records that haven't been updated in the last five minutes. Using the `spark` utility, all we have to do is provide the filter `UNIX_TIMESTAMP() - LAST_UPDATED() > 300`. ```bash $ spark sow -server myserver:port -type json -topic sampleTopic \ -filter 'UNIX_TIMESTAMP() - LAST_UPDATED() > 300' ``` For each message in the topic, AMPS subtracts the timestamp for the last time the message was updated from the current timestamp. If the difference is more than 300 (300 seconds, that is, five minutes), the filter is true and AMPS returns the message. A filter like this could also be used in a `sow_delete` command to find and remove messages that haven't been active in a certain interval of time. ##### Query Versus Subscription Notice that for the command above, I used `sow` rather than `subscribe` or `sow_and_subscribe`. There's an important reason for this: AMPS evaluates filters against a message when a `sow` query runs, or when there's a change to the message. That means that a filter like `UNIX_TIMESTAMP() - LAST_UPDATED() > 300` is very useful for a query, but isn't very useful for a subscription. Here's why: imagine a message hasn't been updated in 298 seconds when someone issues a `sow_and_subscribe` with this filter. The message doesn't match during the `sow` part of the command, so it isn't returned with the query results. After 2 more seconds, the message would match the filter. However, AMPS doesn't re-evaluate the message against the filter until the message changes, so the subscription doesn't receive the message. (If someone does publish to the message, that will reset the `LAST_UPDATED()` value, and the message would no longer match.) ### Message Provenance In some applications, it's very important to be able to reliably identify the source of a change. In versions prior to 5.2.4.1, the most commonly used pattern was to require each publisher to annotate changes with the user name of the person making the change, while using an entitlement filter to guarantee that the annotations matched the expected user name. With 5.2.4.1, you can use the `USER()` function with enrichment to have AMPS directly annotate each change. This makes things simpler for publishers, and can also provide information that was not previously verifiable. For example, the following simple topic annotates each publish with the authenticated user name of the connection submitting the publish and captures the address from which the publish originated. ```xml showLineNumbers change-source-sample json ./sow/%n.sow /id CONCAT(USER(), ' connected from ', REMOTE_ADDRESS()) AS /lastChangedBy ``` Since enrichment modifies the message before the message is written to the transaction log, the enrichment is also captured in the transaction log for an embedded audit trail. ### Calculating Message Size Statistics An important factor in capacity planning is understanding message sizes. Estimates before a system goes into production are helpful, but as a system comes online, more precise data is available and can be used to validate (or correct) the initial estimates. Likewise, for views, it can be inconvenient to persist the contents of the view to be able to estimate averages. With the `MESSAGE_SIZE()` function and aggregated subscriptions, it's easy to calculate metrics over a topic in the state of the world. For example, the following `spark` query computes basic message size metrics for a SOW topic or view: ```bash showLineNumbers spark sow -topic test -server localhost:9007/amps/json -opts \ 'projection=[sum(message_size()) as /totalSize,\ max(message_size()) as /biggestMessage,\ avg(message_size()) as /averageSize,\ stddev_samp(message_size()) as /stdDev],\ grouping=[/nullValue]' ``` Notice that, since we want to calculate a single value for the entire topic, the `spark` command deliberately sets `grouping` to a field that isn't in the messages. The result is that all the messages in the topic are in the same group, and the metrics are computed over the entire topic. ### Data Up Your Sleeve Just as every good magic trick relies on skill and knowledge, using the new metadata functions well also requires some understanding of how AMPS works and what functions are available when. The simple rule to follow is that a metadata function returns meaningful results if AMPS has the information to provide, and `NULL` otherwise. For example, AMPS can only provide a `LAST_UPDATED()` value for messages that are in the State of the World. If there's no State of the World (that is -- AMPS isn't persisting messages in a way that they can be queried), then there's no way for AMPS to calculate `LAST_UPDATED()`. Likewise, since AMPS only assigns bookmarks to messages that are stored in the transaction log, the `BOOKMARK()` function only returns a value for that function for topics that are stored in the transaction log. The [AMPS User Guide](/docs/amps-user-guide/builtin_functions/amps-function-overview) lists the circumstances under which each of the functions returns a non-NULL value. ### Tricks of the Trade In this post, we've just scratched the surface of the functions available and the ways that those functions can be used. Have a recipe that isn't listed here? Know a great trick for monitoring AMPS with these functions, or have a cool technique that isn't mentioned here? How will you use the new functions? Let us know in the comments! --- # AMPS 5.3: More Power, More Performance lightning strike in a dark night 60East is proud to announce the release of AMPS 5.3 — the most fully-featured and easy to use version of AMPS yet! ## Production Tested From Day One The 5.3 release marks the full release of the features we've been releasing in previews for the last 18 months. The preview program was designed to quickly provide access to new features, and one measure of the success of the preview program is that most of the new features in this release have already been used in production for months, at multiple customer sites, making this the most well-vetted version of AMPS in our history. ## Advanced Features for Demanding Applications This release includes the following major features, new since 5.2: * Select lists, which allow a subscriber to receive a precise subset of the fields in a message, rather than having to receive the full message. * Priority queues, which provide the ability to deliver more important messages first, rather than providing messages in strict first-in-first-out order. * Full support for paginated `sow_and_subscribe`. With this support, a client can specify a page of messages to monitor, and receive _only_ updates for messages in that page. This feature includes support for providing an `oof` message when messages leave the window. * Dramatically expanded set of functions for use in filters, preprocessing/enrichment, and view construction. These new functions include access to metadata for the message and the connection from which the message originated, when available. * Improved support for dead letter queues and poison message handling in AMPS queues * Performance enhancements throughout the server, including significant enhancements to bookmark replay scalability and the ability of AMPS to handle SOW topics recorded in the transaction log that are much larger than physical memory. * Configurable conflation intervals for acknowledgment messages. This allows applications that cannot tolerate the 1 second conflation interval to decrease the wait for acknowledgment to be returned to a publisher. * An optional authentication module that can use an existing Kerberos or LDAP system to verify the identity of a connection to AMPS. * Support for Protocol Buffers v3 * Support for MessagePack * Support for unsigned long integers * Major improvements to the Admin console, including new statistics, new views in Galvanometer, and a new "dark" theme. * Support for Basic Authentication in the Admin console * Support for HTTPS in the Admin console We'll be following up with more extensive blog posts on many of these features in the weeks to come. Aside from these major improvements, this release includes dozens of smaller improvements and bug fixes. See the release history for the full list. ## Upgrade In Seconds We're also excited about a few things that _aren't_ in this release. In the 5.3 release, you can upgrade your AMPS instances from 5.0 or 5.2 *without* running the `amps_upgrade` tool to process data files. AMPS 5.3 will work seamlessly with data files from a previous version. (Notice, though, that 5.3 may record data that cannot be processed by earlier versions, so while upgrade is seamless, earlier versions of AMPS do not work with the new features in 5.3 and do not support rollback). You can also replicate between 5.2 versions of AMPS and 5.3 for the purposes of rolling and incremental upgrades. This capability is intended to help distributed teams and applications with complex topologies upgrade "in place" without the need for a service to ever completely go offline. All of this means that an upgrade can often be as simple as stopping the running version of AMPS and restarting with the new binary. (60East recommends testing the upgrade in a development or test environment first, of course, since we've also added more error-prevention to the startup process.) ## Cranked Up and Ready AMPS 5.3 is available for download and ready to take on your most demanding workloads. What's your favorite part of AMPS 5.3? Let us know -- whether that's one of the major features mentioned above, a smaller improvement, the new theme in Galvanometer, or something else entirely! _[Updated 5/20/2019 to mention unsigned long support.]_ --- # Select Lists: Data Served Up As You Like It burrito loaded with ingredients Select Lists is a new feature introduced in our 5.3 release of the AMPS server. This feature lets you declare a subset of fields for your application to receive when querying or subscribing to a topic. AMPS client applications no longer need to receive a full message when the application will only use part of the message. This feature, when combined with existing filter functionality, provides developers with new methods to efficiently manage the volume of data flowing back to their applications from the AMPS server. Select lists can be used to replace prior work-arounds such as declaring views on top of underlying topics, simply to provide a method to project a subset of fields from the underlying messages, back to the client application. Likewise, if an application declares an aggregated subscription _only_ for the sake of receiving a subset of fields, select lists is an easier and more efficient way to get the same result. ### Choose Your Fields Select lists introduces a new, expressive grammar to declare which fields an application would like to receive from AMPS. The grammar used to declare a Select List specifies a new options keyword, `select=` followed by a comma-delimited list of XPath identifiers for fields that you wish to grant or remove access to. You express whether you are granting or removing access by prepending a `+` or `-` immediately preceding the XPath identifiers. Examples of each of these tokens are `+/included` and `-/excluded` for requesting that AMPS include the `/included` field and requesting that AMPS exclude the `-/excluded` field, respectively. For additional details, please refer to the AMPS User Guide. ### Building A Better Burrito Let's look at a simple example. Suppose a message lists the ingredients of a burrito. ```js showLineNumbers {"id":123, "menu_item": "basic burrito", "green": "lettuce", "protein": "chicken", "beans": "pinto", "cheese":"cheddar mix", "topping": "sour cream", "salsa": "chipotle", "tortilla": "flour"} ``` An application that wanted to know the full ingredients of the burrito could simply do a `sow` query for a burrito, as follows: ```python showLineNumbers # Assume diner is a connected AMPS Client or HAClient for m in diner.sow('foods', '/menu_item = "basic burrito"'): print m.get_data() ``` This would return the original message. An application could use a select list, though, to simply see what protein and salsa is available on a basic burrito: ```python showLineNumbers # Assume diner is a connected AMPS Client or HAClient for m in diner.sow('foods', '/menu_item = "basic burrito"', options='select=[-/,+/protein,+/salsa]'): print m.get_data() ``` This query explicitly _removes_ all fields from the message with the directive `-/`, then adds back the `protein` and `salsa` fields. This select list produces the following result: ```js showLineNumbers { "protein": "chicken", "salsa": "chipotle" } ``` Notice that the message contains exactly the fields requested in the select list. No other information is included -- not even the key field for the SOW or the field that the query filters on. Following this pattern, you could also easily select other versions of the basic burrito. For example, you could make a vegan burrito: ```python showLineNumbers # Assume diner is a connected AMPS Client or HAClient # Vegan burrito for m in diner.sow('foods', '/menu_item = "basic burrito"', options='select=[-/protein,-/cheese,-/topping]'): print m.get_data() ``` ```js showLineNumbers {"id":123, "menu_item": "basic burrito", "green": "lettuce", "beans": "pinto", "salsa": "chipotle", "tortilla": "flour"} ``` You could also select a burrito without any beans or tortilla: ```python showLineNumbers # Assume diner is a connected AMPS Client or HAClient # Low carb burrito for m in diner.sow('foods', '/menu_item = "basic burrito"', options='select=[-/beans,-/tortilla]'): print m.get_data() ``` ### Keeping Result Sets Lean Dashboard client applications are a perfect fit to use Select Lists in concert with paginated subscriptions, to optimize as responsive a UI as possible! The following example highlights how a dashboard client application would query a `quotes` topic using a paginated `sow_and_subscribe` command to request 10 messages at a time and dynamically selecting the subset of fields it would like to receive. Before select lists, AMPS would either require the user to receive all fields from the topic (possibly many more than required), create a View that would project the desired subset of fields to be returned, or use dynamic aggregation (which created and calculated an aggregation, even when the only result was to provide a few fields. For example, the dashboard might simply return the `ticker`, `price` and `quantity` fields while retrieving 10 messages at a time: ```python showLineNumbers # Assumes dashboard is a connect AMPS Client or HAClient dashboard.sow_and_subscribe('quotes', order_by='/ticker', \ options='top_n=10,skip_n=10,select=[-/,+/ticker,+/price,+/qty]') ``` ### Securing Data with Select Lists Select Lists have been built-in to all aspects of AMPS. Not only are subscribing clients able to specify a Select List for fields to be returned, but AMPS also provides select lists as part of the entitlement system. These select lists can be use to prevent users from retrieving or querying certain fields within a topic. Below are some example Entitlement Select Lists that to demonstrate how they would be declared: ```js showLineNumbers { "userId": 456, "userName": "John Investor", "password": "NeverGonnaGuess", "secret": "Boston Red Sox fan", "optional": "empty" } ``` In this example, the Entitlement Select List starts off by implicitly granting full access to all fields (explicitly, would specify `+/`) then specifying the removal of access to the `/password` and `/secret` expressions for those specific fields. ```bash -/password,-/secret ``` For the message above, the result would be that the user would be able to see the `userId`, `userName`, and `optional` fields. Even better, if the user provides a filter that provides either the `password` field or the `secret` field, those fields will always be treated as `NULL` -- exactly the same as if they did not exist in the original message. Another way to use entitlement select lists is to configure an explicit removal of access to all fields, followed by granting of individual entitlement to specific fields. This expression would grant entitlement to the following fields: userId and userName. ```bash -/,+/userId,+/userName ``` In this case, no matter what other fields were present in the message, this user would only be able to see the `userId` and `userName` fields. For that user, it is as though no other fields exist. ### Choose Your Own Data (or Toppings) As shown above, select lists are one more feature to help applications reduce bandwidth usage and processor consumption. Just like ordering a burrito, you can add or remove data as your application needs. Of course, also like ordering a burrito, you can only choose from the data available -- you can't order mango salsa if there's no mango salsa available. So, for a basic burrito, you can include or remove an ingredient, but you can't change the value of an ingredient. (If you _do_ need to change data in a message, you can use a view, an aggregated subscription, or enrichment/preprocessing to do that, depending on your exact needs.) With the integration into the entitlement system, select lists also provide another facet of an overall strategy for keeping data secure in AMPS. Combined with entitlement filters and the extensibility of AMPS user-defined functions, AMPS has extremely fine-grained control over access to data. How do you plan to use select lists? How much bandwidth and processing power will you save? Let us know in the comments! _[Edited on 5/23/2019 to fix typos in text and code snippets.]_ --- # Secure your AMPS instances with Kerberos kerberos [Kerberos](https://web.mit.edu/kerberos) has been an industry standard for authentication for many years and, as of 5.3, AMPS now ships with Kerberos support. AMPS Kerberos support is provided as one of the authentication mechanism options available via the [libamps_multi_authentication](/docs/amps-user-guide/securing/authentication#provided-authentication-modules) module. Kerberos requires that an authentication token be generated and set by the client, so there are also client-side Kerberos authenticators implemented for each of the AMPS client libraries. Before going any further, it's important to note that for this post (and to use Kerberos in your environment), Kerberos infrastructure is a prerequisite. Setting up Kerberos infrastructure is beyond the scope of this article, and is something that is normally managed by a dedicated team. Assuming that Kerberos is already set up for your environment, you will need the following for this demo to function: - 2 Kerberos Service Principle Names (SPNs) - `HTTP/hostname` - For securing the AMPS Admin Interface (`hostname` must be the fully qualified host name where your AMPS instance is running) - `AMPS/hostname` - For securing the AMPS Transports - A Kerberos Keytab containing the above SPNs - A user with Kerberos credentials (may be obtained via `kinit`, via a Keytab for the user or automatically during logon which is often the case for Windows/Active Directory). Note that for configuring Replication Authentication a Keytab for the user that the AMPS server will authenticate as is required. ## Configuring AMPS As documented in the [Configuration Guide](/docs/amps-user-guide/securing/configuring-authentication) the `Authentication` element can be specifed for the instance as a whole and/or for each `Transport`. In the below configuration Kerberos authentication has been enabled for all transports using the `AMPS` SPN and then overridden for the `Admin` interface to use the `HTTP` SPN. Note that in addition to the `Authentication` element, `libamps_multi_authentication.so` must be specified as a `Module` in the `Modules` section as it is not loaded into AMPS by default. The below AMPS configuration uses environment variables for the Kerberos configuration elements, thus before starting AMPS using this config the following variables need to be set: - `AMPS_SPN` - Set to `AMPS/hostname` where `hostname` is the fully qualified name of the host AMPS is running on. - `HTTP_SPN` - Set to `HTTP/hostname` where `hostname` is the fully qualified name of the host AMPS is running on. - `AMPS_KEYTAB` - Set to the path of a Kerberos keytab containing entries for the `AMPS` and `HTTP` SPNs. ```xml showLineNumbers ... libamps-multi-authentication ${AMPS_SPN} ${AMPS_KEYTAB} 8085 stats.db libamps-multi-authentication ${HTTP_SPN} ${AMPS_KEYTAB} json-tcp 8095 tcp amps json libamps-multi-authentication libamps_multi_authentication.so ... ``` ### Success! When AMPS starts up the following will be written to the AMPS log. ```python showLineNumbers 2019-04-23T13:51:11.2302690 [1] info: 29-0103 AMPS authentication creating authentication context for transport 'amps-admin' 2019-04-23T13:51:11.2302900 [1] info: 29-0103 AMPS Kerberos authentication enabled with the following options: [Kerberos.Keytab=/home/bamboo/blog/instance/HTTP-linux-ip-172-31-46-201.us-west-2.compute.internal.keytab] [Kerberos.SPN=HTTP/ip-172-31-46-201.us-west-2.compute.internal] 2019-04-23T13:51:11.2303670 [1] info: 29-0103 AMPS Kerberos authentication importing service name 'HTTP@ip-172-31-46-201.us-west-2.compute.internal' 2019-04-23T13:51:11.2303720 [1] info: 29-0103 AMPS Kerberos authentication acquiring credentials for service name 'HTTP@ip-172-31-46-201.us-west-2.compute.internal' 2019-04-23T13:51:11.2564880 [1] info: 29-0103 AMPS authentication context created successfully 2019-04-23T13:51:11.2564950 [1] info: 29-0103 AMPS authentication creating authentication context for transport 'json-tcp' 2019-04-23T13:51:11.2565040 [1] info: 29-0103 AMPS Kerberos authentication enabled with the following options: [Kerberos.Keytab=/home/bamboo/blog/instance/AMPS-linux-ip-172-31-46-201.us-west-2.compute.internal.keytab] [Kerberos.SPN=AMPS/ip-172-31-46-201.us-west-2.compute.internal] 2019-04-23T13:51:11.2565270 [1] info: 29-0103 AMPS Kerberos authentication importing service name 'AMPS@ip-172-31-46-201.us-west-2.compute.internal' 2019-04-23T13:51:11.2565280 [1] info: 29-0103 AMPS Kerberos authentication acquiring credentials for service name 'AMPS@ip-172-31-46-201.us-west-2.compute.internal' 2019-04-23T13:51:11.2609840 [1] info: 29-0103 AMPS authentication context created successfully ``` Below is the logging for a successful authentication. In this log snippet we can see the following: - Client connection - Client logon (with the password field set to the base64 encoded kerberos token) - Logging related to the authentication process including success messages - Client session info - Logon response (with the password field set to the base64 encoded kerberos response token) ```python showLineNumbers 2019-04-23T14:20:37.8828830 [30] info: 07-0023 New Client Connection: client info: client name = AMPS_A-json-tcp-3-212422832437882870 description = 172.31.46.201:52502 -> 172.31.46.201:8095 2019-04-23T14:20:37.8830300 [15] trace: 12-0010 client[AMPS_A-json-tcp-3-212422832437882870] logon command received: {"c":"logon","cid":"0","client_name":"KerberosExampleClient","user_id":"60east","mt":"json","a":"processed","version":"develop.c75d0e8.256701:c++","pw":"YIICmwYJKoZIhvcSAQICAQBuggKKMIIChqADAgEFoQMCAQ6iBwMFACAAAACjggGWYYIBkjCCAY6gAwIBBaEUGxJDUkFOS1VQVEhFQU1QUy5DT02iPjA8oAMCAQOhNTAzGwRBTVBTGytpcC0xNzItMzEtNDYtMjAxLnVzLXdlc3QtMi5jb21wdXRlLmludGVybmFso4IBLzCCASugAwIBEqEDAgECooIBHQSCARkk+zZFgqOGk2kNVzI3R8MC839gvYWPUcKeNOLu2vQvHxpfT5eyW272Y6qXrttx2J4S7ccRjlwGRPxjITFGHtiGM4T7CC3DNwPieYH2qhU3pIjsDldBUqVLnNhdkwAFaj+H2gw/UIudc8DHhNAfIL8xXc3qlun/iw1zE4gsSw8NkqJewbrNLY9Q5wgpFScKGGhtmrSTelAERzp4X6Qsju5IGtVTIhzngq45sAmhiW/tqT8u5TS8mSoILYm/e8QseL24FYPwt7mueD/U4Lo27bsD4HkAMQ1OHQXPm0rp+zRz2js5A1dAQXcSLWB67016iGfto01qR+TorjKpGbM/kJX2DzfjVWiu2olcoD0CaApMQSRklN4pzoFoCqSB1jCB06ADAgESooHLBIHI1PJpEsljIQpzSi3tJHA/DIpjj25ODoOYNkbxSvHCZB22t0r+sqgxVycwnGEI8C4b8BLCp7PW8iHm0UWl2r3osNCjT8EuNFc7jAoyQIbIRRMoGH50BUzQbxGjz1th3WKzs7dlG6vOEcKXPJYGHq0hguc2lScBpXDzqw+f/wIGVdcsNGyHoY3yBXvo600pAxVaJv+jxx4X+9FbDoIUd8/l8KLXxt6ocdmMGutAec0IlO8ksUkbIOjDlaYiv9fOeoVbL1mSyOLmWTA="} 2019-04-23T14:20:37.8830810 [28] info: 29-0103 AMPS authentication executing authentication for user '60east' using Kerberos 2019-04-23T14:20:37.8830900 [28] info: 29-0103 AMPS Kerberos authentication accepting Kerberos security context with input token length of 671 bytes 2019-04-23T14:20:37.8855330 [28] info: 29-0103 AMPS Kerberos authentication creating output token length of 156 bytes 2019-04-23T14:20:37.8855780 [28] info: 29-0103 AMPS Kerberos authentication successfully processed security context for user '60east@CRANKUPTHEAMPS.COM' 2019-04-23T14:20:37.8855820 [28] info: 29-0103 AMPS Kerberos authentication successfully authenticated user '60east' 2019-04-23T14:20:37.8855830 [28] info: 29-0103 AMPS authentication successfully authenticated user '60east' using Kerberos 2019-04-23T14:20:37.8856330 [29] info: 1F-0004 [AMPS_A-json-tcp-3-212422832437882870] AMPS client session logon for: KerberosExampleClient client session info: client authid = '60east' client name hash = 15396145143580885467 client version = develop.c75d0e8.256701:c++ last acked client seq = 0 last tx log client seq = 0 correlation id = 2019-04-23T14:20:37.8856600 [29] trace: 17-0002 client[KerberosExampleClient] ack sent: {"c":"ack","cid":"0","user_id":"60east","s":0,"bm":"15396145143580885467|0|","client_name":"KerberosExampleClient","a":"processed","status":"success","pw":"YIGZBgkqhkiG9xIBAgICAG+BiTCBhqADAgEFoQMCAQ+iejB4oAMCARKicQRvupwwcjivlf4T1wY+bfhi4i3qBrTnZ5toPbXyR7O3syB8WyFCA5iepf3bJ/44HRo0PKQQVSmpH1hhWtdx+T9HUWmLOiRG+lu6TCsF7RBr6pjsW+V8/iHzOTaD5/T5gxHlQIFl/nSWH8pyS/1B8EqL","version":"develop.310283.6f81d4a"} ``` ## AMPS Client Support 60East provides an implementation of an authenticator for each of the client libraries. These authenticators use the features of that individual programming language to provide the appropriate Kerberos token to AMPS. The details of how to use each authenticator depend on the language, as described below. ### Authenticating using Python For Kerberos authentication using python there is a single module, `amps_kerberos_authenticator`, for authentication on both Linux and Windows. The Python Kerberos authenticator code can be found in the [amps-authentication-python](https://github.com/60East/amps-authentication-python/tree/master/kerberos) github repo. The Python Kerberos authenticator has two dependencies; the `kerberos` module for authentication on Linux and the `winkerberos` module for authentication on Windows. In order to execute the below code the following needs to be done: - The `USERNAME` and `HOSTNAME` variables need to be set to the user name you are authenticating as and the fully qualified name of the host AMPS is running on. - Kerberos credentials need to be available for the user you are authenticating as, thus `kinit` needs to be executed before the code is run or the `KRB5_CLIENT_KTNAME` environment variable needs to be set to a keytab that has been generated for the user (note that this is for the user logging in, and should be a _different_ keytab than the one AMPS is configured with). ```python showLineNumbers import AMPS import amps_kerberos_authenticator USERNAME = 'username' HOSTNAME = 'hostname' AMPS_SPN = 'AMPS/%s' % HOSTNAME AMPS_URI = 'tcp://%s@%s:8095/amps/json' % (USERNAME, HOSTNAME) def main(): authenticator = amps_kerberos_authenticator.create(AMPS_SPN) client = AMPS.Client('KerberosExampleClient') client.connect(AMPS_URI) client.logon(5000, authenticator) ``` ### Authenticating using JavaScript (Node.js) While Kerberos works automatically through the `Negotiate` mechanism of every browser, server-side Node.js applications might require some extra steps to make the Kerberos authentication work with the JavaScript client. For Kerberos authentication using JavaScript there is a single package, `amps-kerberos-authenticator`, available in NPM, for authentication on both Linux and Windows. The JavaScript Kerberos authenticator code can be found in the [amps-authentication-javascript](https://github.com/60East/amps-authentication-javascript/tree/master/kerberos) github repo. The JavaScript Kerberos authenticator has two dependencies; the `kerberos` package and the `amps` package. Both are available in NPM and will be installed automatically if the authenticator is installed from NPM: ```bash npm install --save amps-kerberos-authenticator ``` In order to execute the below code the following needs to be done: - The `USERNAME` and `HOSTNAME` variables need to be set to the user name you are authenticating as and the fully qualified name of the host AMPS is running on. - Kerberos credentials need to be available for the user you are authenticating as, thus `kinit` needs to be executed before the code is run or the `KRB5_CLIENT_KTNAME` environment variable needs to be set to a keytab that has been generated for the user (note that this is for the user logging in, and should be a _different_ keytab than the one AMPS is configured with). ```javascript showLineNumbers const { Client } = require('amps'); const { AMPSKerberosAuthenticator } = require('amps-kerberos-authenticator'); async function main() { const USERNAME = 'username'; const HOSTNAME = 'hostname'; const AMPS_SPN = 'AMPS/' + HOSTNAME; const AMPS_URI = 'ws://' + USERNAME + '@' + HOSTNAME + ':8095/amps/json'; const auth = new AMPSKerberosAuthenticator(AMPS_SPN); const client = new Client('KerberosExampleClient'); await client.connect(AMPS_URI, auth); } main(); ``` ### Authenticating using Java For Kerberos authentication using Java there are two different `Authenticator` implementations, one for GSSAPI based authentication and one for SSPI based authentication. GSSAPI is the only option for authentication when running on Linux, but it is supported on Windows as well. When using GSSAPI a JAAS configuration file is required. SSPI, on the other hand, uses Windows native system calls and thus is Windows only and does not require a JAAS configuration. In general, we recommend that `AMPSKerberosGSSAPIAuthenticator` is used when running on Linux and `AMPSKerberosSSPIAuthenticator` is used when running on Windows. The Java Kerberos authenticator code can be found in the [amps-authentication-java](https://github.com/60East/amps-authentication-java/tree/master/AMPSKerberos) github repo. The Java Kerberos authenticator has a number of dependencies that are detailed in the Maven POM file for the AMPSKerberos project. Below are two different JAAS configuration options. The first example will use Kerberos credentials in the user's Kerberos credentials cache or will prompt the user for a password to obtain the credentials. The second example utilizes a keytab to obtain the Kerberos credentials. When a JAAS configuration is utilized the `java.security.auth.login.config` property needs to be set to the path to the config file and the config file entry name needs to be passed to the `AMPSKerberosGSSAPIAuthenticator` along with the SPN. ```yaml showLineNumbers TestClient { com.sun.security.auth.module.Krb5LoginModule required isInitiator=true principal="username" useTicketCache=true storeKey=true }; TestClient { com.sun.security.auth.module.Krb5LoginModule required isInitiator=true useKeyTab=true keyTab="/path/to/username.keytab" principal="username@REALM" storeKey=true doNotPrompt=true }; ``` In order to execute the below code the following needs to be done: - If using GSSAPI, create the appropriate JAAS config, uncomment the `AMPSKerberosGSSAPIAuthenticator` line and specify `-Djava.security.auth.login.config=/path/to/jaas.conf` when starting the app. - If running on Windows and using SSPI uncomment the `AMPSKerberosSSPIAuthenticator` line. - The `username` and `hostname` variables need to be set to the user name you are authenticating as and the fully qualified name of the host AMPS is running on. ```java showLineNumbers import com.crankuptheamps.authentication.kerberos.AMPSKerberosGSSAPIAuthenticator; import com.crankuptheamps.authentication.kerberos.AMPSKerberosSSPIAuthenticator; import com.crankuptheamps.client.Authenticator; import com.crankuptheamps.client.Client; import com.crankuptheamps.client.exception.ConnectionException; public class KerberosAuthExample { public static void main( String[] args ) throws ConnectionException { String username = "username"; String hostname = "hostname"; String amps_spn = "AMPS/" + hostname; String amps_uri = "tcp://" + username + "@" + hostname + ":8095/amps/json"; // Authenticator authenticator = new AMPSKerberosGSSAPIAuthenticator(amps_spn, "TestClient"); // Authenticator authenticator = new AMPSKerberosSSPIAuthenticator(amps_spn); Client client = new Client("KerberosExampleClient"); try { client.connect(amps_uri); client.logon(5000, authenticator); } finally { client.close(); } } } ``` ### Authenticating using C++ For Kerberos authentication using C++ there are two different `Authenticator` implementations, one for GSSAPI based authentication and one for SSPI based authentication. GSSAPI is the only option for authentication when running on Linux, but, unlike Java, it is not supported on Windows. Specifically, `AMPSKerberosGSSAPIAuthenticator` is used when running on Linux and `AMPSKerberosSSPIAuthenticator` is used when running on Windows. The C++ Kerberos authenticator code can be found in the [amps-authentication-cpp](https://github.com/60East/amps-authentication-cpp/tree/master/kerberos) github repo. The C++ Kerberos authenticator for Linux (GSSAPI) has a dependency on the GSSAPI libs that are part of the [krb5 distribution](http://web.mit.edu/Kerberos/dist). In order to execute the below code the following needs to be done: - The `username` and `hostname` variables need to be set to the user name you are authenticating as and the fully qualified name of the host AMPS is running on. ```cpp showLineNumbers #include #include #ifdef _WIN32 #include "AMPSKerberosSSPIAuthenticator.hpp" #else #include "AMPSKerberosGSSAPIAuthenticator.hpp" #endif int main () { std::string username("username"); std::string hostname("hostname"); std::string amps_spn = std::string("AMPS/") + hostname; std::string amps_uri = std::string("tcp://") + username + "@" + hostname + ":8095/amps/json"; #ifdef _WIN32 AMPS::AMPSKerberosSSPIAuthenticator authenticator(amps_spn); #else AMPS::AMPSKerberosGSSAPIAuthenticator authenticator(amps_spn); #endif AMPS::Client client("KerberosExampleClient"); client.connect(amps_uri); client.logon(5000, authenticator); } ``` ### Authenticating using C#/.NET For Kerberos authentication using C# there is a single module, `AMPSKerberosAuthenticator`, for authentication on Windows. The C# Kerberos authenticator code can be found in the [amps-authentication-csharp](https://github.com/60East/amps-authentication-csharp/tree/master/AMPSKerberos) github repo. The C# Kerberos authenticator has a single dependency on the [NSspi NuGet package](https://www.nuget.org/packages/NSspi). In order to execute the below code the following needs to be done: - The `username` and `hostname` variables need to be set to the user name you are authenticating as and the fully qualified name of the host AMPS is running on. ```csharp showLineNumbers using AMPS.Client; using AMPSKerberos; class KerberosAuthExample { static void Main(string[] args) { string username = "username"; string hostname = "hostname"; string amps_spn = string.Format("AMPS/{0}", hostname); string amps_uri = string.Format("tcp://{0}@{1}:8095/amps/json", username, hostname); AMPSKerberosAuthenticator authenticator = new AMPSKerberosAuthenticator(amps_spn); using (Client client = new Client("KerberosExampleClient")) { client.connect(amps_uri); client.logon(5000, authenticator); } } } ``` ## Server Chooser When using a server chooser the `get_authenticator` function needs to be implemented and it must return the appropriate authenticator for the server you are connecting to. In the case that the server chooser is configured to return URIs for different hosts, then an authenticator configured with the correct SPN needs to be returned. Below is an example of a server chooser implementation in python that uses multiple hosts. ```python showLineNumbers import AMPS import amps_kerberos_authenticator from urlparse import urlparse USERNAME = 'username' HOSTNAME1 = 'hostname1' HOSTNAME2 = 'hostname2' class KerberosAuthServerChooser(AMPS.DefaultServerChooser): def __init__(self): super(KerberosAuthServerChooser, self).__init__() self.authenticators = {} def get_current_authenticator(self): hostname = urlparse(self.get_current_uri()).hostname if hostname in self.authenticators: return self.authenticators[hostname] spn = 'AMPS/%s' % hostname authenticator = amps_kerberos_authenticator.create(spn) self.authenticators[hostname] = authenticator return authenticator def report_failure(self, exception, connectionInfo): print exception def main(): client = AMPS.HAClient('KerberosExampleClient') chooser = KerberosAuthServerChooser() chooser.add('tcp://%s@%s:8095/amps/json' % (USERNAME, HOSTNAME1)) chooser.add('tcp://%s@%s:8095/amps/json' % (USERNAME, HOSTNAME2)) client.set_server_chooser(chooser) client.connect_and_logon() if __name__ == '__main__': main() ``` ## Admin Interface Authentication With an OS and browser that is properly configured for Kerberos authentication, it is possible to authenticate against the AMPS Admin Interface using Kerberos via a browser. The `curl` command can also be used to authenticate against the admin interface when the OS is properly configured and `curl` has been built with Kerberos support. ```xml `curl -v --negotiate -u : http://hostname:admin_port/amps/instance/config.xml` ``` It is also possible to programmatically authenticate with the admin interface. Below is an example using python. ```python showLineNumbers #!/usr/bin/env python import amps_kerberos_authenticator import requests HOSTNAME = 'hostname' ADMIN_PORT = 0 HTTP_SPN = 'HTTP/%s' % HOSTNAME ADMIN_URL = 'http://%s:%s/amps/instance/config.xml' % (HOSTNAME, ADMIN_PORT) def main(): authenticator = amps_kerberos_authenticator.create(HTTP_SPN) token = authenticator.authenticate(None, None) # No user and no return token r = requests.get(ADMIN_URL, headers={'Authorization': 'Negotiate %s' % token}) print r.text ``` ### Replication Authentication Securing AMPS for replication requires having a replication `Transport` with `Authentication` enabled as well the use of an `Authenticator` in the `Replication/Destination` `Transport`. The [libamps_multi_authenticator](/docs/amps-user-guide/securing/multi-authenticator-module) module provides the client side authentication support for the replication connection. The below sample AMPS configuration uses environment variables for the Kerberos configuration elements. The following variables would need to be set correctly for this config to function as expected. A second instance would also need to be configured. - `AMPS_A_SPN` - Set to `AMPS/hostname` where `hostname` is the fully qualified name of the host AMPS A is running on. - `AMPS_B_SPN` - Set to `AMPS/hostname` where `hostname` is the fully qualified name of the host AMPS B is running on. - `AMPS_A_KEYTAB` - Set to the path of a Kerberos keytab containing entries for the `AMPS` SPN for the A instance. - `AMPS_USER_KEYTAB` - Set to the path of a Kerberos keytab that contains an entry for the user principal that you want to use to authenticate to AMPS B. ```xml showLineNumbers AMPS_A libamps-multi-authentication ${AMPS_A_SPN} ${AMPS_A_KEYTAB} json-tcp 8095 tcp amps json 8105 amps-replication amps-replication AMPS_B sync localhost:9999 amps-replication libamps-multi-authenticator ${AMPS_USER_KEYTAB} ${AMPS_B_SPN} json .* libamps-multi-authentication libamps_multi_authentication.so libamps-multi-authenticator libamps_multi_authenticator.so ... ``` ## To Guard and Protect In this post, I've covered the basics of setting up AMPS for Kerberos authentication. If your site uses Kerberos, the team that manages that installation will have definitive answers on how Kerberos is configured in your environment. In this post, we've intentionally kept the focus on AMPS. We haven't delved into the depths of troubleshooting Kerberos installations, or the considerations involved in setting up a Kerberos infrastructure. We encourage you to work with the team responsible for your company's Kerberos installation when configuring AMPS, since many of the most common problems in setting up Kerberos authentication are a matter of ensuring that the credentials provided match what the Kerberos server expects -- which is most easily done when the owners of the Kerberos configuration are also involved. We're very proud to have Kerberos support available out of the box with AMPS. Let us know how the Kerberos support works for you! _[Edited on 7/3/2019 to add information about the JavaScript Kerberos authenticator for Node.js applications.]_ --- # First Things First: Priority Queues dial labelled 'priority' with hand about to turn the dial AMPS queues provide a simple way to distribute work across a group of consumers. By default, AMPS queues provide work in first-in-first-out fashion: that is, the oldest message in the queue is provided to subscribers first, then the next oldest, and so on. For some problems, though, it's important that the _most important_ work happen first, even if the most important message in the queue isn't actually the oldest. For example, a monitoring system may want to log and analyze all events, but may want to process a critical alert immediately, even if there are hundreds of informational events ahead of that alert in the queue. Likewise, a compute grid may need to have time-critical requests processed before work that is less time-critical. One approach for these problems would be to build a separate queue for the higher-priority messages, or to create dedicated "high priority" processors that use a content filter to retrieve only high-priority messages. Those approaches can work very well, but sometimes it's important to have a solution that doesn't require clients to be aware of the priority, or that allows a more flexible priority system rather than relying on a limited number of priority levels. ## AMPS Priority Queues With AMPS priority queues (available in 5.3.0 and higher releases), you can use the properties of the message to automatically set the priority of the message within the queue. To do this, you simply add a `Priority` to the configuration of the queue. The `Priority` tells AMPS how to calculate the priority of each message in the queue. Unlike some other queueing systems, AMPS does not require the publisher to set an explicit priority (or even know that message delivery will be prioritized), and AMPS does not require a fixed set of priority levels or categories. This gives AMPS the ability to provide priority queues that are simple to set up, easy to use, and extremely flexible and adaptable as your application evolves. The details on how priority queues work are available in the [Priority Queues](/docs/amps-user-guide/queues/advanced-queue-configuration#priority-queues) section of the AMPS _User Guide_. Here are a few examples. ### A Basic Example In the simplest case, creating a priority queue can be as easy as including the priority in the message, and using that field to set the delivery priority: ```xml /info/priority ``` With a priority configuration like the one above, if you publish the following two messages to the AMPS queue, AMPS will deliver the second message from the queue before the first message: ```javascript showLineNumbers {"message":1, "info":{"priority":1, "note":"Low priority message"}} {"message":2, "info":{"priority":1000, "note":"High priority message"}} ``` Notice that, in AMPS, priority is always sorted so that the highest priority value is delivered first. (Even if you are used to considering "priority 1" or "priority 0" as meaning "highest priority", because `1000` is greater than `1`, AMPS delivers the message with priority `1000` first. ### Complex Expressions The `Priority` element can include any AMPS expression that produces an integer. This means that more complicated expressions are possible, and that, in many cases, you can use data that is already contained in the message to calculate the priority. For example, you might want to process orders that have the largest total value before orders with lower values (as computed by the price of the item times the quantity of the item in the order). In this case, there's no need for a publisher to add an extra field to the message, since all of the information necessary to calculate the priority is already there. ```xml showLineNumbers (/price * /quantity) as /priority ``` The format for the expression is the same format used for constructing fields. All of the functions that you can use in a standard AMPS filter are available. AMPS calculates the priority when it starts tracking the message for queue delivery (that is, when the message enters the queue), and delivers the message when all higher-priority messages have been delivered. It's also important to remember that there are no predetermined priority levels: there's no need to determine ahead of time whether a message where `/price * /quantity` is `400` is a high-priority message, a medium-priority message, or a low-priority message. With AMPS, that message is higher priority than a message where `/price * /quantity` is `399`, and lower priority than a message where `/price * /quantity` is `401`. There's no need for complicated configuration or arbitrary boundaries between priority levels. ## Complete Example With that background, here's a complete example that includes a `Queue` definition, a publisher and a consumer. Below is a full example of a `Queue` definition that has higher values of the `/pri` field in a JSON message prioritized over lower values: ```xml showLineNumbers q json /pri ``` Next, we can show how the messages are published to the queue using the AMPS Python Client ```python showLineNumbers import AMPS, random, json, time COUNT = 10 client = AMPS.Client("pri-pub-%d" % os.getpid()) client.connect("tcp://localhost:9007/amps/json") client.logon() for i in xrange(COUNT): client.publish("q", json.dumps({"id":i, "pri":random.randrange(5), "ts":time.time()})) ``` In the example above, we are going to publish 10 messages to the priority queue we configured above `q`. Each message has a `/pri` field receiving a value between 0 and 4, and each message has a timestamp to help us track message order. Consuming messages from the priority queue is identical to consuming regular queue messages. Simply subscribe to the queue topic, process, and ack the messages. ```python showLineNumbers for msg in client.subscribe("q"): body = json.loads(msg.get_data()) print "Message with priority: %s" % (body["pri"]) msg.ack() ``` Notice that when you run this sample, AMPS returns messages in priority order (higher values of the ``pri`` field are returned first) rather than the publication order (which would show a lower ``id`` number first, with the values rising). For example, the subscriber above might produce output such as the following: ```bash showLineNumbers Message with priority: 4 Message with priority: 4 Message with priority: 4 Message with priority: 3 Message with priority: 2 Message with priority: 2 Message with priority: 1 Message with priority: 1 Message with priority: 0 Message with priority: 0 ``` (At this point, the program would block waiting for the next message from the priority queue.) If publishers continued to publish messages, new messages would also be delivered in priority order. ## Views on Queues AMPS provides support for views that use a queue as the underlying topic. These views show messages that are currently available for delivery to a consumer. Adding a materialized view of a queue topic is a common approach to create monitoring over the contents of a queue. In the following example, the view is configured to enumerate each of the distinct priority levels in the priority queue, along with the number of messages in the queue, and the oldest timestamp. This simple view configuration can be used to ascertain whether the priority queue is functioning properly. ```xml showLineNumbers q-status json q /pri /pri as /pri COUNT(/id) as /count MIN(/ts) as /oldest_ts ``` Using the `spark` command that is bundled with AMPS, monitoring the state and progress of the priority queues becomes simple. ```bash watch -n1 "~/spark sow -server localhost:9007 -type json \ -topic 'q-status' -orderby '/pri'" ``` After running the simple publish script above several times, the results of this command would look something like the following (although, of course, the actual output will reflect the messages currently in the queue, so the results will depend on how many times you run the publisher, what random priority values were assigned, and whether you have acknowledged messages from the queue). ```javascript showLineNumbers {"count":22, "oldest_ts":1547650270.46372,"pri",0} {"count":22, "oldest_ts":1547650270.46447,"pri",1} {"count":22, "oldest_ts":1547650270.46455,"pri",2} {"count":22, "oldest_ts":1547650270.46462,"pri",3} {"count":22, "oldest_ts":1547650270.46492,"pri",4} Total messages received: 5 (1666.67/s) ``` For an actual monitoring system, we would use a `sow_and_subscribe` command to receive the current state and process updates. For example purposes, we use `watch` at the command line and re-run the query from scratch every time because this makes the output easy to view in a Linux terminal. ## Conclusion AMPS provides a priority queue that is simultaneously familiar to existing priority queue semantics, yet provides the flexibility for queue semantics to go beyond merely assigning priority. How will you use priority queues? --- [Edit: 2019-11-14 Remove unused imports from python sample.] [Edit: 2019-11-18 Various typo fixes.] [Edit: 2019-11-19 Add sample output for queue section subscriber to make it clear that the output in the view section is for the `watch`/`spark` command, not the subscriber.] --- # Scaled-Out Batch Processing with AMPS Queue Barriers runners at a starting line, waiting for the signal to start Scaling out your data processing using AMPS queues allows you to dynamically adjust how many workers you apply to your data based on your needs and your computing resources. Larger orders coming in or more events to process? Just spin up more subscribers to your AMPS queue and let them take their share of the load. AMPS dynamically adjusts how many messages it gives to each consumer based on their available backlog and message processing rate, and your work gets done without needing costly reconfiguration or re-partitioning. Scale-out with queues works great when each message is an independent unit of work. Each subscriber works independently. The subscribers work in competition with one another, each processing messages as fast as possible, acknowledging messages back to AMPS as quickly as possible, and receiving more messages as quickly as possible. AMPS includes features to intelligently deliver messages to consumers that are processing quickly, to avoid a consumer ever having to wait for work when work is available. Some problems require coordination, however. In this article, we look at a typical situation where you want multiple consumers competing to process messages as fast as possible, but you also need them to just work on the same batch of data and to wait for each other to finish and move to the next batch in unison. We’ll examine a powerful new feature in AMPS queues, *Barrier Expressions*, that allows consumers to coordinate and work in concert to efficiently process data. ## Starting Small Let’s imagine that we run a very successful nationwide chain of spatula stores, Spatula City. Each of our 1200 stores transfers their individual sales and customer interaction data every night. We aim to process this data as quickly as possible so we can make real-time business decisions based on actual spatula trends. We use this data to update a global inventory management system, and then in aggregate, use the data for ever-more-advanced predictive analytics. When our business was small we could get this done with a pretty simple program: ```python showLineNumbers # Simple file processing example # (before adding AMPS queues) # # For sample purposes, assume that # spatula_utils includes an existing # library for database connectivity, # and so forth. from spatula_utils import * import sys database = Database() db_conn = database.connect() # For each line in each file, # the program does a nightly update. # After every file is processed, # predictive analytics start. processed_count = 0 for daily_file in sys.argv[1:]: f = open(daily_file) for line in f.xreadlines(): clean_data = preprocess(line) json_data = convert_to_json(clean_data) update_inventory_system(json_data) db_conn.insert(json_data) processed_count += 1 if processed_count % 100 == 0: print processed_count print "Done for the day! Starting predictive analytics." run_predictive_analytics(db_conn) ``` As we processed each row of data from each store, we'd take a moment to update our inventory control system. Once the whole day's set of files was processed, we'd run our larger analytics. ### Batches Get Bigger This system works great when all of the work can be done by a single process in a reasonable amount of time. As our analytics become more advanced, however, we need to get through our file loading much more quickly. And as the number of stores and their sales volume increases (everyone needs spatulas!), our single process for file loading desperately needs scale. AMPS message queues to the rescue! Let’s configure and build an AMPS message queue and publisher to help us distribute this workload efficiently: #### AMPS Configuration File Here's a simple configuration file to distribute work over a set of subscribers: ```xml showLineNumbers NIGHTLY_SALES_DATA json 24h ... ``` In the AMPS configuration file, we've created a single message queue that will be used to distribute our data in a JSON format. Any number of consumers can connect to this instance and subscribe to the queue to become consumers. This message queue will automatically be configured with "at-least-once" semantics, meaning that a message will remain in the queue until it has been delivered to a consumer and acknowledged. Now let's examine the code for our publisher and consumers: #### Publisher As mentioned above, the publisher reads a set of files and publishes information from each line of the file to AMPS, for a consumer to process: ```python showLineNumbers # Simple AMPS queue publisher # example. # Assume that spatula_utils has # database classes and so on. from spatula_utils import * import sys, os import AMPS database = Database() db_conn = database.connect() client = AMPS.Client("publisher-%d"%os.getpid()) client.connect(...) client.logon() # For each file, publish the work # to be processed to the AMPS queue processed_count = 0 for daily_file in sys.argv[1:]: f = open(daily_file) for line in f.xreadlines(): clean_data = preprocess(line) if processed_count % 100 == 0: print processed_count client.publish("NIGHTLY_SALES_DATA", clean_data) processed_count += 1 ## Wait until everything is processed... ## ... but how long should we wait? time.sleep(...) print "Done for the day! Starting predictive analytics." run_predictive_analytics(db_conn) ``` #### Consumer The consumer is just as simple as the publisher. For each message from AMPS, the consumer processes the message, and then acknowledges to the server that the work is done. (We are not tuning queue delivery for maximum performance here, just showing the simplest way to convert the existing process.) ```python showLineNumbers # Read from an AMPS queue and # process sales data. # Assume that spatula_utils has # database classes and so on. from spatula_utils import * import sys, os, socket import AMPS database = Database() db_conn = database.connect() hostname = socket.gethostname() pid = os.getpid() unique_client_name = "consumer-%s-%d" % (hostname, pid) client = AMPS.Client(unique_client_name) client.connect(...) client.logon() print "%s: connected" % unique_client_name processed_count = 0 for message in client.subscribe("NIGHTLY_SALES_DATA"): data = message.get_data() json_data = json.loads(data) update_inventory_system(json_data) db_conn.insert(json_data) message.ack() # Tell the server the message is complete processed_count += 1 if processed_count % 100 == 0: print "%s: %d" % (unique_client_name, processed_count) # This should exit if nightly processing is done... # ... but how do we know for sure? ``` With just a few lines of code, we're able to convert our single-threaded script into a fully-distributed system based on AMPS queues and scale our work out over any number of processors and hosts. We can now increase processing scale just by running another instance of the consumer. But we have one critical, unsolved problem: how do we know when we're done? ## Endings Can be Difficult Here's what we know: all of the data will eventually be processed and loaded into the database. Even if a processor fails midway through, AMPS will redeliver that work to another processor, until every message in the queue is processed. What we don't know, though, is when the day's worth of data is finished being loaded into the database. Each consumer runs independently, and doesn't need to know whether the other consumers are finished. In fact, that's the whole point: we can add as many consumers as we need to just by starting up another process. They don't need to do any sort of coordination: AMPS queues automatically handle message distribution. In our publisher, we need a way to know once all of our messages are fully processed since we need to run analytics at the end. In our consumers, we'd like to be able to exit gracefully once the batch is completed. This problem is harder than it seems for a few reasons: - If the publisher were to wait for the queue to be completely drained, what happens if a consumer fails and returns a message back to the queue? - Maybe every consumer could somehow keep a separate topic up-to-date with every message that's been processed. But how would that scale, and what if there's a failure writing to that topic? - What if publishers want to start writing the next day's data before the previous day is finished? You wouldn't want consumers to march ahead into the next day's data until the previous day is done. (That's not a problem that Spatula City has now, but we have big dreams!) These aren't new problems, and we've seen all sorts of creative ways to signal the completion of a batch. Some are more failure-prone than others, and all of them require thinking about the myriad ways a job might get stuck or might be seen as completed when it's really not. Some solutions require coordinating between consumers, or require a publisher to absolutely know in advance how much work there is to do, or involve guessing based on probability ("if no new messages have shown up to be processed in 60 seconds, I guess that must mean the publisher is done"). Wouldn't it be great if there was a way to clearly communicate to all consumers when messages up to a certain point had been fully processed, and to keep the next day's messages from being processed until the current one is finished? # Introducing Queue Barriers ![Consumers receive everything before the barrier message](/img/blog/barrier-expression-1.png "Consumers Receive Everything up to the Barrier Message. Nothing after the barrier message is eligible for delivery.") AMPS (starting with 5.3.1) has a unique, built-in way to solve problems like these. AMPS will identify specific messages in a message queue as _barrier messages_ based on whether the message matches a `BarrierExpression` you configure. A barrier message won't be delivered until all prior messages are delivered and acknowledged. Also, unlike typical queue messages (which are delivered to a single subscriber), when the barrier message is delivered, the barrier message will be delivered to _all subscribers to the queue_ (provided their entitlements and content filter match, of course). Messages after a barrier message will not be delivered to any consumer until the barrier message has been released and delivered. ![When released, all consumers receive the barrier message](/img/blog/barrier-expression-2.png "When the barrier message is released (because all previous messages are processed), the barrier message is delivered to all subscribers, and normal queue delivery begins with the message after the barrier message.") Let's use a barrier message in our file-loading application to coordinate and scale-out our post-processing analytics! In our config file, we add one line of configuration to indicate how AMPS should determine if a message is a barrier message. We'll simply add an "is_eof" field to the message when it demarcates the end of the day: ```xml showLineNumbers NIGHTLY_SALES_DATA json 24h /is_eof = 1 ... ``` In our publisher, we write a message that matches the BarrierExpression by including an "is_eof" when we're done publishing the day's spatula data. Unlike all of the other messages we've written, this message will be delivered to _all consumers_, and it will only be delivered once all of the previous messages we wrote are completely processed. This gives us a built-in way to know for sure that the day's work has been processed by the consumers. We alter our publisher to also consume our EOF message so that it knows when to run the analytics step: ```python showLineNumbers # More effective AMPS queue publisher # example. This uses the BarrierExpression # configured for the queue to know # when consumers have finished work. # Assume that spatula_utils has # database classes and so on. from spatula_utils import * import os, sys import AMPS database = Database() db_conn = database.connect() client = AMPS.Client("publisher-%d"%os.getpid()) client.connect(...) client.logon() processed_count = 0 for daily_file in sys.argv[1:]: f = open(daily_file) for line in f.xreadlines(): clean_data = preprocess(line) if processed_count % 100 == 0: print processed_count client.publish("NIGHTLY_SALES_DATA", clean_data) processed_count += 1 # Publish EOF, then wait for it to be sent back to # us indicating all prior messages are consumed # The subscription uses a content filter so only # the EOF matches eof_stream = client.subscribe("NIGHTLY_SALES_DATA", "/is_eof = 1") client.publish("NIGHTLY_SALES_DATA", '{"is_eof": 1}') print "Waiting for consumers to finish..." # Wait for the EOF for message in eof_stream: break print "Done for the day! Starting predictive analytics." run_predictive_analytics(db_conn) ``` Great! Our publisher knows when all of the data it's written has been fully processed. Let's modify our consumer to look for this EOF message and use that message to terminate: ```python showLineNumbers # Read from an AMPS queue and # process sales data. This version # uses a queue barrier to know when # processing for the day is complete. # Assume that spatula_utils has # database classes and so on. from spatula_utils import * import sys, os, socket import AMPS database = Database() db_conn = database.connect() hostname = socket.gethostname() pid = os.getpid() unique_client_name = "consumer-%s-%d" % (hostname, pid) client = AMPS.Client(unique_client_name) client.connect(...) client.logon() print "%s: connected" % unique_client_name processed_count = 0 for message in client.subscribe("NIGHTLY_SALES_DATA"): data = message.get_data() json_data = json.loads(data) # When we see the barrier message, we can exit # because there's nothing left to process. if "is_eof" in json_data: break update_inventory_system(json_data) db_conn.insert(json_data) message.ack() processed_count += 1 if processed_count % 100 == 0: print "%s: %d" % (unique_client_name, processed_count) print "%s: EOF received, exiting" % unique_client_name ``` Once every non-barrier message has been consumed by a queue consumer, AMPS will send the single barrier message from our publisher to every consumer. The consumer does not need to alter its AMPS subscription to receive this message, and the consumer can exit as soon as it receives a message letting it know that the publisher has finished. ## Conclusion Batch processing remains an important part of many data scenarios. AMPS Message Queues let you scale-out processing of batches in a low-latency fashion by allowing you to treat the data in your batch as independent messages, and yet retain the semantics of a "batch" across queue consumers. --- # Best Web Grids for 2020 Learning to love the grid Many things have changed in the webapp world since we last did [a grid comparison](/blog/grid-comparison/), way back in 2017. Chrome is increasing its domination in the market of browsers. Edge [ditched its own web engine](https://blogs.windows.com/windowsexperience/2020/01/15/new-year-new-browser-the-new-microsoft-edge-is-out-of-preview-and-now-available-for-download/) and is essentially a Microsoft clone of Chrome now. Firefox [is in decline](https://andreasgal.com/2017/07/19/firefox-marketshare-revisited/), has less than 10% of the market and Mozilla recently had to [let go 70 of its employees](https://techcrunch.com/2020/01/15/mozilla-lays-off-70-as-it-waits-for-subscription-products-to-generate-revenue/) as its revenue keeps falling. Meanwhile, more companies than ever are working on, or considering creating modern web applications that will replace their legacy desktop products and embrace a ubiquitous model for desktop and mobile platforms. The combination of converging technology and increasing development of web applications makes it an exciting time -- and make technology choices more important than ever. In our last evaluation, we already established that web interfaces can be as flexible, feature-rich, and robust as their desktop analogues. The centerpiece of any modern data-intensive web application is a grid, so now it's time to review the most popular web grids on the market and see which one is the best pick for 2020! ### Contestants Let me introduce the contestants -- these are all powerful, well established grid engines that are widely used to build modern snappy web applications. - [**ag-Grid 22.1.1**](https://www.ag-grid.com): "The Best JavaScript Grid in the World" is an extremely feature rich, good looking and well documented grid that claims to be even better than before. We found that it has definitely improved since [the last time](/blog/grid-comparison/) we tested it. ![ag-Grid table](/img/blog/gcp2/grids/ag_grid.png "ag-Grid table") *ag-Grid 22.1.1* - [**Sencha Ext JS 7.0.0**](https://www.sencha.com/products/extjs): A part of "the most comprehensive JavaScript framework for building data-intensive applications", this grid claims it can "handle millions of records". Perhaps the most upsetting part about testing this grid was its restrictive license and a requirement to provide your email and phone number so they can spam you -- *not cool, Sencha, learn from ag-Grid on how it's done right*. ![Sencha Ext JS table](/img/blog/gcp2/grids/sencha_grid.png "Sencha Ext JS table") *Sencha Ext JS 7.0.0* - [**Kendo UI Grid 2019.3.1023**](https://demos.telerik.com/kendo-ui/grid/index): Allows you to "quickly build eye-catching, high-performance, responsive web applications". We're about to find out if, in fact, it *ken do* it. ![Kendo UI table](/img/blog/gcp2/grids/kendo_grid.png "Kendo UI table") *Kendo UI Grid 2019.3.1023* - [**w2ui 1.5**](http://w2ui.com/web/docs/1.5/grid): A nice contrast to big JavaScript frameworks, this grid can do most of what other grids offer, but is tiny and lightweight. In fact, it is 9 times smaller than Ext JS and 7 times smaller than Kendo UI. ![w2ui table](/img/blog/gcp2/grids/w2ui_grid.png "w2ui table") *w2ui 1.5* - [**FancyGrid 1.7.87**](https://fancygrid.com): A very good looking grid with a large list of features. Perhaps by pure coincidence they never mention performance of the grid. Unfortunately, it was way too slow in our tests, so we excluded it from the final results. Without pagination, FancyGrid can only handle ~10K records without having significant performance issues, and that's well below our testing threshold. ![FancyGrid table](/img/blog/gcp2/grids/fancy_grid.png "FancyGrid table") *FancyGrid 1.7.87* - [**Webix DataTable 4.3.0**](https://webix.com/widget/datatable/): part of **Webix** framework, DataTable component provides a highly efficient grid that delivers blazing fast performance. We included Webix here because it was the best grid in our previous comparison. ![Webix table](/img/blog/grid-comparison-webix-datatable.png "Webix table") *Webix DataTable 4.3.0* We excluded SlickGrid and HyperGrid from this comparison. SlickGrid, one of the best grids from the previous test, despite all of its benefits, is currently in life-support mode, with only a moderately active fork of the original product being available. [HyperGrid](https://github.com/fin-hypergrid/core), a product that we expected to become mainstream by now is still a niche offering that never gained considerable traction in the web development community. Perhaps there's a future for Canvas-based grids, though at the present moment they are losing the battle -- some of the HTML5 grids are getting very close in terms of performance, and also have the benefit of better customization due to the DOM access. Having said that, HyperGrid's [recent experiments](https://perspective.finos.org/) in combining the grid engine with a web-assembly based data engine can result in a significant increase in performance and create a new class of ultra fast web applications. We're still paying attention to Hypergrid. ### Tests we run Our main focus is user experience. The grid should look good, feel snappy, hold a huge amount of data and yet still go easy on memory as it often is a limited resource. A good grid shows results of a query with subsequent updates that occur to that grid, such as deletes, updates, and new records. On average, we used a message size of **140 Bytes** -- it's big enough to show meaningful information and is small enough to fit millions of such messages into the grid without taking all the available RAM on our test system. The following set of tests should give us a good picture of how the selected grids will perform in a real world situation: - **Rendering Time**: How much time it will take to render the initial portion of data. Fast rendering is important so that the web application loads and is ready to work as soon as possible. Another situation when fast rendering might be useful is switching data sets to display in the grid. - **Frames Per Second** (**FPS**): The more FPS a grid can produce while being used, the smoother and more responsive it looks and feels. Significant changes in **FPS** are perceived as *freezes* and should be avoided as much as possible. - **Memory Consumption**: If a grid is memory efficient, it can work well and do more on a device with less memory, such as mobile devices and laptops. In our test we will measure how many rows/records a grid can render using no more than **2 GB** of RAM. Last time we used **4 GB** as the target value, but people in comments made a valid point that some entry level devices, such as office desktops, smartphones, and Chromebooks might only have 4 GB available, so we lowered the threshold accordingly. - **Live Updates**: Rendering the initial portion of data is important, but that's not enough if a grid cannot smoothly render changes in real time. According to [MDN](https://developer.mozilla.org/en-US/docs/Tools/Performance/Frame_rate), *"A frame rate of 60fps is the target for smooth performance, giving you a time budget of 16.7ms for all the updates needed in response to some event."* In this test we will measure how many rows per second we can add to the grid while maintaining maximum FPS and experience no lagging. ### Environment ##### Hardware - **CPU:** 6th Generation Intel® Core™ i7-6820HQ Processor (8MB Cache, up to 3.60GHz) - **GPU:** NVIDIA® Quadro® M1000M 4GB - **RAM:** 64GB DDR4 2133 MHz - **Storage:** 1TB PCIe SSD ##### Software - **OS**: Ubuntu 5.3.0-27 GNU/Linux x86_64 - **Browser**: Google Chrome Linux 79.0.3945.130 (64-bit) We don't test other browsers, due to the factors I mentioned in the beginning of this post. Chrome and its clones (including Safari which is using WebKit) are, at this point, the only significant platform that people use. Some people don't like this fact which reminds them of the sad era of Internet Explorer dominance, however it's not the same situation. This time the platform is free and open for everyone -- and being packaged into independent browsers by multiple *independent* teams. This fact ensures there will be no stagnation, violation of standards, or reliance on vendor-lock-in solutions. On the side note, web developers are happier than ever! ### Time to Render Once the initial data portion of a query is loaded (typically, in a separate WebWorker, especially if it's large), we need to display it in the grid. Considering the size of the initial portion, all grids did great, but as usual, some did better than others. ![Grid Comparison: Rendering Time](/img/blog/gcp2/graphs/rendering_all.svg "Grid Comparison: Rendering Time") Even though every grid in our tests is using virtualization, [**ag-Grid**'s Viewport model](https://www.ag-grid.com/javascript-grid-viewport/) was absolutely the best -- it feels like there's no limit to amount of data it can handle! **Webix**'s results are second best. **w2ui** has a somewhat linear dependency for the rendering time required which might be a limiting factor for extremely large datasets. ### Rendering Performance in Dynamic: Scroll Tests Okay, we have our data loaded and rendered. Is the grid still responsive and snappy? Can we look through these rows without having a feeling that we're watching a slide show? To make this test more objective, we measured FPS using Chrome Developer Tools. Scrolling using Touchpad/Mouse wheel simulates slow scrolling, while scrolling by dragging the scrollbar will show how the grids are optimized for very fast scrolling. As before, we test for each dataset size. ![Grid Comparison: Scrolling FPS -- Mouse / Touchpad](/img/blog/gcp2/graphs/scroll_mouse.svg "Grid Comparison: Scrolling FPS -- Mouse / Touchpad") ![Grid Comparison: Scrolling FPS -- Scrollbar](/img/blog/gcp2/graphs/scroll_bar.svg "Grid Comparison: Scrolling FPS -- Scrollbar") **ag-Grid** and **Webix** are clear winners, followed by **Ext JS**, which demonstrated great performance and is very well optimized out-of-the-box for buttery smooth rendering regardless of the dataset size. On the other hand, **Kendo** didn't perform as well. In our test, every grid except **Kendo** felt snappy and smooth. ### Memory Efficiency This is a test that answers one simple question -- how much data can we display in the grid on an entry level desktop? ![Memory Consumption](/img/blog/gcp2/graphs/memory_consumption.svg "Memory Consumption") Due to the [tricks](https://www.ag-grid.com/javascript-grid-viewport/#example-sequence) that **ag-Grid** provides for its row model, data consumption is extremely efficient and almost identical to **Webix** *again*. The two cool kids absolutely destroyed the rest of competition when it comes to memory efficiency. **Kendo** continues to disappoint us with 4x less memory efficiency than the winners. ### Real Time Updates Now the grid is loaded and rendered with all of the original data, but that's not enough -- without real-time updates, the dataset becomes irrelevant. This test starts with the 20,000 records and applies updates at an increasing rate, until the grid starts lagging. ![Live Grid Updates](/img/blog/gcp2/graphs/live_updates.svg "Live Grid Updates") **w2ui** is simply mind blowing when it comes to real-time updates! Having said that, all contestants (except, again, **Kendo**) demonstrated great performance when it comes dynamic data updates. It's worth mentioning that most grids allow batching grid updates and **ag-Grid** demonstrates [insane performance](https://medium.com/ag-grid/how-to-test-for-the-best-html5-grid-for-streaming-updates-53545bb9256a). ### Learning Curve While users care about performance and look-and-feel of a grid, developers care about how easy it is to work with the grid and implement application functionality with it. Let's see what each grid engine can offer in terms of documentation, examples, and support. I've ranked the grids as follows: 1. **ag-Grid** -- Excellent documentation and examples. Commercial support is available. 2. **Webix** -- Almost as good as **ag-Grid**. The grid provides tons of examples and live demos, API is very well documented and easy to search and navigate. Commercial support is available. 3. **Kendo UI** -- Nice, well structured documentation and examples. Commercial support is available. The only downside is a restrictive license and lack of public access to the framework, as well as a private NPM repository, which makes it harder to begin evaluation. 4. **FancyGrid** -- Nice, clear, and concise tutorials and docs with editable live examples, but lacks built-in search. Commercial support is available. 5. **Sencha Ext JS** -- On the bright side, the docs are very well written, and live examples with the playground are amazing. Having said that, the documentation is in a poor state. It is extremely fragmented; most of it is divided into "classic" and "modern" parts, sample projects are spread into multiple versions of the API. Worse of all, links throughout docs often lead to earlier versions of the API and it's possible to switch to an older API version and potentially get incompatible or deprecated code. Try to google `extjs grid` and you'll see 4-6 various versions of the docs in search results). Some of the example links on the [site](https://examples.sencha.com/extjs/7.0.0/) don't work (for example, ["Buffered Scrolling"](https://examples.sencha.com/extjs/7.0.0/examples/classic/grid/buffer-grid.html)). Similar to Kendo, a restrictive license and lack of public access to the framework as well as a private NPM repository make it harder to begin evaluation and also result in unsolicited calls and emails. Commercial support is available. 6. **w2ui** -- Documentation is present, but feels a bit limited. There are a few examples but for most part it's a trial and error process to make it work as expected. There's no commercial support available, although it's possible to [order a training session](http://w2ui.com/web/support) if your company happens to have an office in California. ### Conclusion We took a look at some of the great modern web grids. Test results prove the point that web interfaces combined with modern technologies can provide a highly responsive and well performing user interface. This time, the winners are: 1. **ag-Grid**: Absolute winner! It's greatly improved since the last time, and I tend to agree that it is perhaps *the* best HTML5 grid available today. 2. **Webix**: Excellent performance in all categories, great API and documentation -- this is a great alternative to **ag-Grid** although not as robust when it comes to integration with modern applications built in React and Angular. 3. **Ext JS**: The documentation is a mess, but it performs really well and working with it was nice and easy. It sure does have every possible feature you can imagine in a grid engine! Honorable mention: **w2ui**. It wasn't the fastest, but it is very compact and simple to work with. If you're looking for a low calorie alternative to the Enterprise giants, this is a great pick. Hopefully, these results can help steer you in the right direction of selecting the best web grid for your next project. As you can see, some of the grids were more performant along several dimensions so that gives you some choice and ability to cater to your specific requirements. For users who want to try these grids in action, we've prepared a [GitHub repository](https://github.com/60East/amps-blog-web-grid-bake-off) with the sample projects for each grid used in the above tests. Did we miss something? Do you know a great grid we should totally try? Let us know what you think! --- # From Zero to Fault Tolerance Hero with AMPS Replication xkcd style stick man comic that describes a boss asking a programmer how their AMPS application handles fault tolerance. In real world systems, networks fail, components need to be replaced, servers need maintenance. Successful enterprise grade applications need to be designed with fault tolerance in mind! AMPS sets you up for success with features designed for robust fault tolerance and high availability. Key to these features is AMPS _replication_ -- ensuring that messages are reliably distributed to more than one server. Planning for a disaster can be hard, but AMPS replication doesn't have to be daunting. That's why we are bringing you a series of detailed blog posts that will take you from fault tolerance newbie to replication pro. ## Starting Simple AMPS supports complex replication topologies containing many AMPS instances, but first, we have to start with the fundamental building blocks. In this first post, we are going to describe the basic configuration settings needed to bring your AMPS deployment into a replicated configuration. We'll set up a simple configuration with a single topic replicated between two AMPS instances. #### The First Instance We start by giving the AMPS instance a name as usual. When we're replicating an instance the name is important, because this is what identifies this instance to other instances of AMPS. It's important that the name be unique among the set of replicated instances, and it shouldn't have any weird characters, but it doesn't have to be especially exciting otherwise. So we'll use `AMPS-Replication-A` for this one. Then we give the instance a **Group** tag. The **Group** tag is used by AMPS to represent regions or clusters of instances. As you will see when we define the replication destination, the Group name is the primary way in which amps verifies that it has connected to the correct destination. Note: The **Group** tag is verified by the upstream AMPS instance at the time of **connection**, not when the server first starts up. (And if you don't set a **Group**, AMPS uses the **Name** of the instance as the group -- a group of one, if you will.) ```xml showLineNumbers AMPS-Replication-A DataCenter-1 ... ``` ##### Let's Define Some Transports Transports allow connections _into_ the AMPS instance. For this sample, we will create two transports. The first transport is for standard client traffic to this instance. There is nothing special here. Pay attention to the second transport we define. This is the important one for replication. AMPS uses a dedicated protocol to replicate messages between instances. The `amps-replication` protocol is a proprietary format that allows AMPS to efficiently compress and multiplex the replicated message streams so you get the most out of your network. The instance can receive messages from any number of upstream instances on this transport, so we only need **one** incoming transport, regardless of how many servers will replicate to this instance. The `Name` field of a transport is an identifier string used for debugging purposes such as log messages. `Name` can be anything, but to help make debugging replication easier, we recommend that the `Name` of the Transport match the value of the `Type` for the `amps-replication` transport. ```xml showLineNumbers any-tcp tcp 9007 amps amps-replication amps-replication localhost:10004 ``` Notice that we're not saying anything here about what the incoming replication messages contain, where they're from, or anything like that. That's not the concern of this instance -- in AMPS, the _source_ of the data controls replication, as we'll see later on. ##### Topics and Transaction logs This is a standard pub/sub topic that is configured to be stored in a transaction log. The one requirement for a replicated topic is that it **must** be stored in a transaction log. AMPS replicates exactly the information that is written to the local transaction log. The topic is not required to be declared anywhere else. ```xml showLineNumbers ./journal-A/ orders json ``` ##### Look Mom, no SOW If you're used to other systems, where you have to predeclare topics, or if you typically use topics in the SOW (which have to be defined so AMPS can handle associating messages into a record), let's just pause here for a minute. ![No SOW Mind Blown](/img/blog/xkcd_mind_blown.png "No SOW Mind Blown") Let me repeat that last point, "the topic **is not** required to be declared anywhere else!" If it's in the transaction log, you can replicate it. ##### The Replication Block All of the building blocks are in place for replication! Now, all we need to do is tell AMPS how to replicate the topic. This is where the magic happens. AMPS replication is push based. Each replication block describes what messages will be replicated and what destination those messages will be pushed to. The most important configuration is the name (and message type) of the topic we want to replicate. For this example we define a single topic, but you can define any number of topics or even a regular expression. All matching topics will be replicated to the destination node. ```xml showLineNumbers json orders ``` ##### Group Since we're sending business data to another instance of AMPS so that it is available there when we need it, it's important to be sure that this connection reaches the expected downstream instance. The group needs to match the name of the group that the downstream instance has defined (that is, the top level `Group` tag in that instance's configuration). At **connection time**, AMPS will report an error and not replicate if the group specified here does not match the group of the downstream instance. ```xml showLineNumbers DataCenter-2 ``` ##### Sync or Async The choice between sync or async determines when the server will respond to the message's source that the message has been persisted. The choice has **no effect** on the transfer of messages from the server to the replication destination. As a good default, 60East recommends starting with `sync`. There are powerful use cases that take advantage of `async` replication, but that is for a future blog post. ```xml sync ``` ##### Destination Transports Last, we set the network properties for the outgoing connection. The last part of the Replication destination is probably the most straight forward. We need to define the address and type of the destination. The `Type` for **both** the source and the destination should **always** be `amps-replication`. Remember that the `amps-replication` protocol is a proprietary format that allows AMPS to efficiently compress and multiplex the replicated message streams so you get the most out of your network. ```xml showLineNumbers localhost:10005 amps-replication ``` ##### Finish off this config There are some parts of the configuration that we always include, no matter what the purpose of the instance is. This includes settings such as the admin interface, standard logging, and the closing `AMPSConfig` tag. - **Note:** This is a simple example to demonstrate replication. Your production config will likely have different logging settings and many other elements configured. ```xml showLineNumbers localhost:8085 file info ./logs/instance-A-%Y%m%d-%n.log ``` #### Take a Breath Congratulations! You have set up the first instance in a replicated pair. Remember, most of it is the same as a standard AMPS configuration. Let's summarize the big highlights for this configuration: - Define a group along with your instance name - Define the incoming replication transport; we only need one for the entire instance - Define the topic in the Transaction Log - Define the replication destination block; define where we are pushing message to (one for each place we replicate to) - Define the standard logging and admin configuration #### The Second Instance The second instance looks much like the first, with only a few differences. The Name and Group are defined just like the first instance. Note, the name is different, since these are different instances (and the instance name needs to be unique across all replicated instances). Also notice that the group matches the group we put in the Destination for the first instance -- after all, this is the group we're planning for that Destination to reach! ```xml showLineNumbers AMPS-Replication-B DataCenter-2 ``` Just like the first instance, a transport for normal clients and a single separate replication transport is defined. - **Note**: the ports are unique, to allow both AMPS instances to run on a single host. In a production system, with multiple hosts, it's common to use the same ports for the same purpose in every configuration. ```xml showLineNumbers any-tcp tcp 9008 amps amps-replication amps-replication localhost:10005 ``` The Transaction log is defined exactly the same way as the first instance. Every replicated topic has to have the transaction log defined for that topic on every node. This is the one requirement for topics. - *Remember* AMPS replicates exactly the information that is written to the local transaction log. ```xml showLineNumbers ./journal-B/ orders json ``` AMPS replication is *single-directional* and *push based*. Messages always flow from a replication destination to a transport configured in the transport section. The second instance has configured a replication destination that mirrors back to the first instance. The Topic name in the destination must match the first instance. ```xml showLineNumbers json orders ``` We use the group of the **first AMPS instance** here, since that is where this destination will deliver messages. Just like the first instance, We use `sync` acknowledgment to be sure the other instance has the message. ```xml showLineNumbers DataCenter-1 sync ``` We set the address and port of the first instance -- this tells the second instance to connect to the first instance and deliver messages there. ```xml showLineNumbers localhost:10004 amps-replication ``` Other configuration for the instance, such as logging, is independent of replication. This configuration does not have to match the replicated instance. As before, we turn on the admin interface and add logging. ```xml showLineNumbers localhost:8085 file info ./logs/instance-B.%Y%m%d-%n.log ``` ##### Some Notes For this example, both instances of AMPS reside on the same physical host. Don't use this configuration for performance testing! When running both instances on one machine, the performance characteristics will differ from production, so running both instances on one machine is more useful for testing configuration correctness than testing overall performance. To get the best performance when running more than one instance of AMPS on the same machine, 60East recommends disabling AMPS NUMA tuning in the AMPS configuration file and relying on the operating system NUMA management. See the [AMPS Configuration Guide](/docs/amps-user-guide/configuring-amps/instance-configuration#tuning) for details on how to disable NUMA in your configuration file. Double check all your port numbers! It’s important to make sure that when running multiple AMPS instances on the same host that there are no conflicting ports. AMPS will emit an error message and will not start properly if it detects that a port is already in use. That's why these samples use different ports for each instance. If the instances were on different systems, we would likely use the same port each instance for a given purpose. ## Conclusion That's it! You now have a replicated AMPS configuration. This will get you started with the basics, but there is a lot more to come. Look for the coming posts in the series, where we build off of these fundamentals to make your AMPS deployment bullet proof in the face of fault tolerance: - Capacity Planning for replication - The AMPS *High Availability* client - Distributed SOWs and Distributed Queues - Server maintenance advice for replication such as optimizing disk space utilization and administrative actions - Complex replication topologies with multiple regions and strategies network degradation mitigation ![Feature Creep](/img/blog/xkcd_feature_creep.png "Feature Creep") For more detailed information check out the AMPS user manual, including the chapter on [replication](/docs/amps-user-guide/ha/ha-details), advice on [capacity planning](/docs/amps-user-guide/operation/capacity-planning), and as always, [support@crankuptheamps.com](mailto:support@crankuptheamps.com) is available to help with specific needs. --- # Bookmark Range clouds over a snowy mountain range Even if you can't make it to the great outdoors, AMPS now makes it easy to visit a range of data in the transaction log. For years, AMPS has had the ability to use a _bookmark subscription_ to replay messages from the transaction log. A bookmark subscription begins at a _bookmark_ -- a point in the transaction log -- and the subscription would continue until the point that the application unsubscribed. Bookmark subscriptions allow applications to resume subscriptions after being disconnected without losing messages, and also provide a way to recreate the sequence of messages to an instance (for example, for auditing or backtesting purposes). ## Come to Think of It, Please Fence Me In But what if an application only wants part of the transaction log (say, messages published between 48 hours ago and 24 hours ago)? The conventional way to solve this with AMPS is for the application to request a bookmark subscription with a timestamp from 48 hours ago to start the replay and then check each message as it arrives to decide if replay has passed the part of the transaction log that was of interest. If the message doesn't already have a timestamp as part of the message data, the subscription would also need to use the `timestamp` option so that AMPS would include the time at which the message was processed on the local instance. This approach requires the AMPS server to send more messages than the application would process, since there is no way for the server to know where the application would stop processing messages. Since replay is typically very fast, this could easily be millions of messages more than the application needs. Starting in the AMPS 5.3.2 preview, AMPS now provides a Bookmark Range feature. This feature lets you decide the exact range of messages for your bookmark subscription to receive. AMPS client applications can now receive an exact set of messages from a subscription during a specific timeframe, including open of business to close using timestamp bookmarks. No longer will clients have to check for that end marker message to see if they should unsubscribe. Now you can specify what specific bookmark you would like AMPS to stop sending messages for the subscription. ## Where to Begin, Where to End AMPS now allows the subscription to set not only the beginning point but also the ending point. Bookmark range also allows you to tell AMPS whether the subscription should receive the initial bookmark, as opposed to regular bookmark replays, where the bookmark provided is not included in the message set returned to the subscription. With a bookmark range, you specify the inclusiveness you would like for your beginning and ending range, and you can choose to include the beginning (and ending) bookmark. To set a beginning and ending point, a subscription provides a subscription range specifier with the bookmark. The format of the subscription range specifier is as follows: ``` : ``` where the `begin_interval_specifier` is one of: | specifier | behavior | | --- | --- | | ( | Exclusive replay. The specified beginning bookmark will not be present in the replay. | | [ | Inclusive replay. The specified beginning bookmark will be present in the replay. | and the `end_interval_specifier` is one of: | specifier | behavior | | --- | --- | | ) | Exclusive replay. The specified ending bookmark will not be present in the replay. | | ] | Inclusive replay. The specified ending bookmark will be present in the replay. | Subscriptions may request a `completed` acknowledgment with their range subscription. AMPS will return the completed acknowledgment when the stopping point is reached. All currently supported bookmark formats are allowed to be used in bookmark range, this includes timestamps. To receive all messages matching your subscription for June 4, 2020 you could specify a bookmark like the following: `[20200604T000000:20200605T000000)`. This bookmark tells AMPS to deliver all the messages that match the subscription beginning at midnight June 4 (inclusive) and ending at midnight June 5 (exclusive). (AMPS timestamps are in the UTC timezone, so if your business is in a different time zone, you can adjust the timestamp as needed.) ## Plan for the Future Currently bookmarks not in the transaction log are treated as unknown bookmarks. Subscriptions with unknown bookmarks are set to begin after the last message recorded in the transaction log at the point of subscription, this includes timestamps in the future. On a regular bookmark subscription with a future timestamp as the bookmark the subscription would begin immediately after the last message in the transaction log. The new bookmark range subscription feature supports future timestamps not being returned as the last message, but rather will hold the subscription open until the given timestamp. Future timestamps can be used in both the `begin_bookmarks` and the `end_bookmarks`. An application can now enter a subscription to collect all records for the full business day ahead and then complete, even when the application is started before business hours. The subscription would just set two future timestamps along with the inclusiveness that the application needs. AMPS will begin delivering messages at the start time specified and stop delivering at the end time specified. Again, a completed acknowledgment will be sent (if requested) at the point where the subscription ends. Bookmark range also supports bookmark lists in both the beginning and ending point. AMPS will find the earliest bookmark in the starting list and the latest bookmark in the stopping list. The replay will then act as if those two bookmarks were given as the range start and stop bookmarks. ## More Than Just Your Normal Open Ended Subscription As you can see above, bookmark range can be used in a variety of ways. You can receive messages from a specific range in the past to retrace your steps or set up a future subscription to gather messages for the work ahead. In either case, AMPS automatically begins and ends the subscription exactly at the point you choose, without your application having to do any extra work, and without sending any unnecessary messages. The new bookmark range feature can be combined with other awesome AMPS features including, but not limited to: filters, select list, and rate. What are you most excited about with this new feature? How will you use this to make your applications more efficient? Let us know in the comments! --- # Great Web Grids for 2021 > A year has passed since our last grid analysis post -- it's time to revisit! Last year we published a very popular [grid comparison](/blog/grid-comparison-2/) which provided a good overview of several web grid engines commonly used for modern web applications. The time has come to expand the review and provide several new options for people who are looking for the best pick for their new stylish ultrafast data intensive web applications. We stand by the notion that web interfaces can be as flexible, feature-rich, and robust as their desktop analogues and should be considered as the first choice when choosing a platform for a user-facing application. Modern data-intensive web applications are essentially web grids so it is extremely important to pick the best option among the many of them available. Our goal is to help you find the grid that powers your next application! ### Contestants We have previously reviewed many great grids and this time we wanted to introduce all new faces that weren't tested before to give you a wider choice for your new shiny high performance web application! The following grids are all excellent and you can't go wrong by choosing any of them for your new project: - [**Tabulator 4.9**](https://tabulator.info/): Their website says it is "The easy to use, fully featured, interactive table JavaScript library". This is a simple, fast, and good looking grid with no external dependencies that is freely available under the MIT license. ![Tabulator table](/img/blog/gcp3/grids/tabulator.png "Tabulator table") Tabulator 4.9 - [**FXB Grid**](https://www.javascript-grid-control.com/): A grid without external dependencies that is simple to understand, and easy to extend, style, and modify. Also, spoiler alert, it is extremely fast! ![FXB table](/img/blog/gcp3/grids/fxb_grid.png "FXB table") FXB Grid - [**GrapeCity Wijmo Grid 5.20203.766**](https://www.grapecity.com/wijmo/demos/Grid/Overview/purejs): A part of the [Wijmo](https://www.grapecity.com/wijmo) framework, this is a great grid component that is good looking, flexible and has an impressive amount of features supported. ![Wijmo table](/img/blog/gcp3/grids/wijmo.png "Wijmo table") Wijmo Grid 5.20203.766 - [**RevoGrid 2.9.0**](https://revolist.github.io/revogrid/): An Excel-like reactive grid that boasts support of huge data load and complex operations. While it is available as a [pure JavaScript library](https://revolist.github.io/revogrid/guide/), it is also providing bindings for most popular modern application frameworks such as [React](https://revolist.github.io/revogrid/guide/framework.react.overview) and [Vue](https://revolist.github.io/revogrid/guide/framework.vue.overview). ![RevoGrid table](/img/blog/gcp3/grids/revo_grid.png "RevoGrid table") RevoGrid 2.9.0 - [**Smart Grid 0.3.1**](https://mukuljainx.github.io/smart-grid/): A lightweight highly customizable React grid. It's very simple and not as feature-rich as other grids in this contest but it has great performance, and a very intuitive reactive interface. It can be a great low calorie open source alternative with only **884 bytes** added to your final application bundle. ![SmartGrid table](/img/blog/gcp3/grids/smart_grid.png "SmartGrid table") Smart Grid 0.3.1 - [**SyncFusion DataGrid 18.4.39**](https://www.syncfusion.com/javascript-ui-controls/js-data-grid): A part of a gigantic [SyncFusion](https://www.syncfusion.com/) component suite that spans through several languages and platforms. Perhaps, it is not the first choice for a standalone web grid project, but would definitely be a great addition that provides a highly efficient grid that delivers excellent performance in case you are already using other SyncFusion components for your applications. ![SyncFusion table](/img/blog/gcp3/grids/sync_fusion.png "SyncFusion table") SyncFusion DataGrid 18.4.39 If you want even more great options to consider for your project I would suggest reviewing our previous posts comparing many other great grids: [Best Web Grids for 2020](/blog/grid-comparison-2/) and [Grids Without Gridlock: Which is Fastest?](/blog/grid-comparison/) ### How We Test Our main focus is user experience. The grid should look good, feel snappy, hold a huge amount of data and yet still go easy on memory as it often is a limited resource. A good grid shows results of a query with subsequent updates that occur to that grid, such as deletes, updates, and new records. On average, we used a message size of **140 Bytes** -- it's big enough to show meaningful information and is small enough to fit millions of such messages into the grid without taking all the available RAM on our test system. The following set of tests should give us a good picture of how the selected grids will perform in a real world situation: - **Rendering Time**: How much time it will take to render the initial portion of data. Fast rendering is important so that the web application loads and is ready to work as soon as possible. Another situation when fast rendering might be useful is switching data sets to display in the grid. - **Frames Per Second** (**FPS**): The more FPS a grid can produce while being used, the smoother and more responsive it looks and feels. Significant changes in **FPS** are perceived as *freezes* and should be avoided as much as possible. - **Memory Consumption**: If a grid is memory efficient, it can work well and do more on a device with less memory, such as mobile devices and laptops. In our test we will measure how many rows/records a grid can render using no more than **2 GB** of RAM. - **Live Updates**: Rendering the initial portion of data is important, but that's not enough if a grid cannot smoothly render changes in real time. According to [MDN](https://developer.mozilla.org/en-US/docs/Tools/Performance/Frame_rate), *"A frame rate of 60fps is the target for smooth performance, giving you a time budget of 16.7ms for all the updates needed in response to some event."* In this test we will measure how many rows per second we can add to the grid while maintaining maximum FPS and experience no lagging. ### Environment ##### Hardware - **CPU:** 6th Generation Intel® Core™ i7-6820HQ Processor (8MB Cache, up to 3.60GHz) - **GPU:** NVIDIA® Quadro® M1000M 4GB - **RAM:** 64GB DDR4 2133 MHz - **Storage:** 1TB PCIe SSD ##### Software - **OS**: Ubuntu 5.8.0-42 GNU/Linux x86_64 - **Browser**: Google Chrome Linux 88.0.4324.150 (64-bit) We don't test other browsers as they are increasingly irrelevant at this point -- they either have a tiny market share (Firefox), don't have decent performance and support for modern technologies (IE 11), or are essentially clones of Chrome (Edge/Opera/etc). ### Time to Render Once the initial data portion of a query is loaded (typically, in a separate WebWorker, especially if it's large), we need to display it in the grid. Considering the size of the initial portion, all grids did great, but one grid is a true king of rendering! ![Grid Comparison: Rendering Time](/img/blog/gcp3/graphs/rendering_all.svg "Grid Comparison: Rendering Time") Every grid in our tests is using virtualization. **FXB Grid** did fantastic, rendering pretty much any dataset in *under* a millisecond! **SyncFusion** and **Smart Grid** performed excellent as well. Other grids demonstrated a near linear dependency for the rendering time required which might be a limiting factor for extremely large datasets. ### Rendering Performance in Dynamic: Scroll Tests Now that we have our data loaded and rendered we want our grid to keep being responsive and snappy with all the data loaded. We want to look through these rows without having a feeling that we're watching a slide show. To make this test more objective, we measured FPS using Chrome Developer Tools. Scrolling using Touchpad/Mouse wheel simulates slow scrolling, while scrolling by dragging the scrollbar will show how the grids are optimized for very fast scrolling. As before, we test for each dataset size. ![Grid Comparison: Scrolling FPS -- Mouse / Touchpad](/img/blog/gcp3/graphs/scroll_mouse.svg "Grid Comparison: Scrolling FPS -- Mouse / Touchpad") ![Grid Comparison: Scrolling FPS -- Scrollbar](/img/blog/gcp3/graphs/scroll_bar.svg "Grid Comparison: Scrolling FPS -- Scrollbar") Every grid nailed the mouse/touchpad scrolling test with buttery smooth near-perfect FPS! The scrollbar scroll, due to its nature, triggers more intensive re-renders of virtualized rows and thus can be somewhat more challenging which we observed in this case. **RevoGrid** and **FXB Grid** are clear winners with **RevoGrid** having the best overall scrolling performance. **Wijmo** and **Smart Grid** demonstrated decent performance which, while being slightly worse, would still provide great user experience. On the other hand, **Tabulator** seems to be not well optimized for this kind of re-rendering case and didn't perform great. **SyncFusion** is a special case -- it felt snappy and smooth when using mouse/touchpad, however, it has a specific re-rendering behavior of lagging for about *200-300ms* after scrolling using the scrollbar, exposing stale data previously populated in the virtual viewport before updating it ([try to scroll the example](https://ej2.syncfusion.com/documentation/grid/virtual/) on their website yourself). Because of that lag, we can't award **SyncFusion** more than 3-5 frames per second for the scrollbar test. ### Memory Efficiency This is a test that answers one simple question -- how much data can we display in the grid on an entry level desktop? ![Memory Consumption](/img/blog/gcp3/graphs/memory_consumption.svg "Memory Consumption") All grids demonstrated great efficiency -- they will fit enough data before running out of memory, however, **Smart Grid**, **SyncFusion**, and **FXB Grid** are significantly more efficient when it comes to memory compared to the rest of the contestants. ### Real Time Random Updates Now the grid is loaded and rendered with all of the original data, but that's not enough -- without real-time updates, the dataset becomes irrelevant. This test starts with the 20,000 records and applies updates at an increasing rate, until the grid starts lagging. ![Live Grid Updates](/img/blog/gcp3/graphs/live_updates.svg "Live Grid Updates") **FXB Grid** is excellent when it comes to real-time updates, more than 5 times outperforming the second place! Having said that, all contestants (except, perhaps, **Tabulator**) demonstrated decent performance when it comes dynamic data updates. ### Quality of Documentation While users care about performance and look-and-feel of a grid, developers care about how easy it is to work with the grid and implement application functionality with it. Let's see what each grid engine can offer in terms of documentation, examples, and support. I've ranked the grids as follows: 1. **Smart Grid** -- The grid is so simple, the documentation is just one small page. The simpler, the better, thus, the first place. 2. **Wijmo** and **SyncFusion** -- both grids provide tons of examples and live demos, API is very well documented and easy to search and navigate. Commercial support is available. 3. **RevoGrid** -- Nice, well structured documentation and examples. Commercial support is available. 4. **FXB Grid** and **Tabulator** -- Great documentation and interactive demos. Commercial support is available for the **FXB Grid**. To be fair, all grids in this comparison have great documentation -- the learning curve won't be too steep! ### Conclusion We've reviewed an array of great web grids that will do great as a heart of your web application. In fact, this time test results demonstrated a much closer competition than [a year ago](/blog/grid-comparison-2/) for another set of contestants. The winners among the newcomers are: 1. **FXB Grid**: Excellent performance in every test -- you simply can't go wrong by picking this grid! 2. **Smart Grid**: An ultralight and ultrafast React grid that will do the job without adding overhead. 3. **RevoGrid**: A great grid engine with the smoothest rendering we've ever seen, with support for most frameworks, great documentation and available technical support for commercial customers. Honorable mention: **SyncFusion**. This is a great pick in terms of performance, however it might not be the first choice if you're not intending to build your entire application using their platform. Our previous winners are also extremely nice grids to consider as well: - **ag-Grid**: An amazing grid engine that has every feature you can possibly imagine, along with great support, documentation and, of course, performance. - **Webix**: Excellent performance in all categories, great API and documentation. Hopefully, these results can help steer you in the right direction of selecting the best web grid for your next project. Before you make the final decision, please also review our previous grid comparison posts: [Best Web Grids for 2020](/blog/grid-comparison-2/) and [Grids Without Gridlock: Which is Fastest?](/blog/grid-comparison/) For users who want to try these grids in action, we've prepared a [GitHub repository](https://github.com/60East/amps-blog-web-grid-bake-off) with the sample projects for each grid used in the above tests. Did we miss something? Do you know a great grid we should totally try? Let us know what you think! --- # AMPS on Windows: WSL2 for Quick Development Cartoon penguin sitting on a window holding the 60East logo AMPS is used for a wide variety of applications, from extreme low-latency applications with a latency budget of less than a millisecond roundtrip to applications that aggregate millions of fast changing records that intentionally conflate updates to reduce load on a user interface. All of these applications have one thing in common, though: AMPS runs on x64 Linux, so application developers need to have access to a Linux installation to develop against AMPS. For sites (or developers) typically focused on Windows development, this can sometimes make it seem more difficult than necessary to get started with AMPS. Installing a virtualization product like VirtualBox or VMWare works well, but involves installing the virtualization software, choosing and manually installing a Linux distribution, and then dealing with what is effectively an entirely separate desktop and development system. On recent builds of Windows 10, the process can now be just a few simple steps. These builds include Windows Subsystem for Linux 2 (WSL2), which runs a Linux kernel on Microsoft's Hyper-V. It's included as an optional feature of Windows 10 starting with the May 27, 2020 update (version 2004 / build 19041 and later). The Linux distributions are provided through the Windows Store, and are simple to download and install for people who don't spend a lot of time managing Linux systems. If you don't already have an existing setup for running a Linux server on your windows system, WSL2 is a great solution. There's no need to have a separate development system or to set up separate Virtual Machine software -- you can do everything you need using the WSL2 system provided with Windows. The advantages of running AMPS on WSL2 are the same as the general advantages of WSL2: * Good performance * An environment that is, effectively, identical to a native Linux install for Linux applications * Tight integration with the Windows environment * Easy installation and setup (especially as compared with installing Linux on a VM) AMPS runs perfectly well on WSL2. All unit and integration tests pass on WSL2 (stress and long-haul-testing suites require several systems with high-speed networking between them, so those aren't really run on any development system). In fact, AMPS (and the development environment for AMPS) run well enough that several members of the 60East development team now use WSL2 as their main development environment when working on the AMPS server. ## AMPS On Windows: Step by Step To set up WSL2 and get AMPS running, follow these steps: 1. First, you need to **make sure you're running a version of Windows 10 that includes WSL2**. As mentioned earlier, any version of Windows 10 more current than version 2004 / build 19041 will work. If you're using an older version of Windows 10, you'll need to allow a Windows feature update to get WSL2 running. 2. **Install WSL2**. There are instructions in the Microsoft documentation for [Installing WSL](https://docs.microsoft.com/en-us/windows/wsl/install). In most cases, this boils down to the following command in an Administrator Powershell: ```powershell PS C:\Users\yourname> wsl --install ``` This installs an Ubuntu distribution by default. If you prefer to use a different distribution, the [Installing WSL](https://docs.microsoft.com/en-us/windows/wsl/install) instructions include information on installing a different distribution. (In our experience, Ubuntu works quite well as a development environment.) To use WSL2 after installing it, you will need to restart your system. 3. Optionally (but strongly recommended), you can **set up other aspects of your development environment**. Among other things, this lets you set a user name and password for the user that shells run under, and so on. You can find full [WSL Environment Setup](https://docs.microsoft.com/en-us/windows/wsl/setup/environment) instructions in the Microsoft documentation. 4. **Install AMPS**. This is a matter of downloading the latest release from the [AMPS release page](https://crankuptheamps.com/releases/amps) and then extracting it. Start the bash shell for your Linux distribution (if you've installed Ubuntu, you can run this by starting Ubuntu from the start menu) or Windows Terminal. Navigate to your home directory, then download and extract AMPS. You can use any current version of AMPS. For example, to install AMPS 5.3.3.30, you would run commands like: Navigate to your home directory: ``` bash $ cd ~ ``` Download the AMPS binary: ``` bash $ wget https://devnull.crankuptheamps.com/releases/amps/5.3.3.30/AMPS-5.3.3.30-Release-Linux.tar.gz ``` Extract the AMPS binary: ```bash $ tar -zxf AMPS-5.3.3.30-Release-Linux.tar.gz ``` 5. **Start AMPS**. To start AMPS, just run the AMPS binary with a valid configuration file. To use the minimal configuration template included with the server, first have the server produce the configuration file and then start AMPS. In a bash shell or Windows terminal window, navigate to your home directory: ```bash $ cd ~ ``` Have the server produce the minimal sample configuration and save it to a file: ```bash $ ./AMPS-5.3.3.30-Release-Linux/bin/ampServer --sample-config > minimal.xml ``` And then run AMPS, using that file as the AMPS configuration: ```bash $ ./AMPS-5.3.3.30-Release-Linux/bin/ampServer minimal.xml ``` Depending on your Windows settings, you may get a warning from Windows firewall when AMPS starts. Check the appropriate boxes for the permissions you want AMPS to have. That's all it takes to get a minimal AMPS configuration up and running! ## Tips for WSL2 You now have a running AMPS instance on your Windows system! If you've used the sample configuration, you can now open a browser on your Windows desktop and navigate to `http://localhost:8085/` to see the Galvanometer for the instance. A few useful tips: * Typically, AMPS will issue a few warnings on startup. The exact warnings depend on the distribution and version you chose for WSL2. The most common warning is that AMPS can't determine the layout of the system for NUMA tuning. In a production system, it would be important to understand these warnings and determine if they need to be resolved. Since the WSL2 setup here is intended for interactive development rather than production, it's generally alright to ignore these warnings and deal with any issues if they emerge. * For developing applications that will run on Linux, [Visual Studio Code](https://code.visualstudio.com/download) works well as a development environment. There's an extension (the [Remote - WSL extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-wsl)) that provides excellent integration with the WSL2 environment. * If, at any point, you want to show the current directory in a File Explorer window, you can type `explorer.exe .` into the bash shell, and Windows will open that path in File Explorer. (This is also a convenient way to get the file path for things like uploading files from or saving files to the Linux environment.) * The WSL2 environment can host multiple AMPS instances, if necessary, to do development with a basic replication and failover scenario. (Of course, the more CPU-intensive processes you have running on the development system, the fewer resources are devoted to each process.) * There is a known issue with IPv4 over IPv6 networking in current builds of WSL2 (see [the github issue](https://github.com/microsoft/WSL/issues/4851) ). This means that, if you are using a recent preview of AMPS and the `Transport` configuration for the server provides an `InetAddr` that only uses a port number (for example, `9007`, as the minimal sample does) then simply connecting to AMPS from Windows processes using `localhost` won't work. To connect to the AMPS server from the Windows side in this case, you have a few different options: 1. You can connect to AMPS over IPv6 (if you are using a current client library and a current preview version of AMPS). Rather than connecting using a string like `tcp://locahost:9007/amps/json`, connect over IPv6 using a string like `tcp://[::1]:9007/amps/json`. 2. Explicitly bind the Transport to IPv4. To do this, you would change the `InetAddr` in the configuration to a value along the lines of `0.0.0.0:9007`, which will tell the AMPS server to listen on all IPv4 interfaces (but no IPv6 interfaces). 3. Find the ethernet address of the WSL2 interface (using `ip addr | grep eth0`in the WSL2 bash shell) and connect using that IP rather than `localhost`. Unfortunately, this address may change if the system is restarted. For local development purposes, we generally take approach 1 or approach 2, since this can be set up once (in the application configuration file or the AMPS server configuration file) and then stay working. ## Crank Up Your Windows! Those of us who run Windows systems at 60East are big fans of WSL2. Although virtual machines work well, the ease of use and simple Windows integration make WSL2 enjoyable to work with. We hope that you'll enjoy it too! Have more tips or tricks when using WSL2? Drop us a line, or let us know in the comments! --- # AMPS Components and CVE-2021-44228 (log4j) Image of a heavy combination lock on a safe Over the last several days, a remote code execution vulnerability ([CVE-2021-4428](https://nvd.nist.gov/vuln/detail/CVE-2021-44228)) has been reported in the popular [Apache log4j](https://logging.apache.org/log4j/2.x/) package. This is an extremely serious vulnerability, and is being actively exploited by attackers as of the publication of this posting. The AMPS server, the AMPS utilities, the AMPS client libraries, and supplemental Java components produced by 60East do not use log4j and are not vulnerable to this issue. * **AMPS server** The AMPS server does not use Java or the log4j product. * **AMPS utilities** The `spark` utility included with AMPS is written in Java, but does not use the log4j product. * **AMPS Java client** The AMPS Java client has no external dependencies outside of the Java Runtime Environment, and does not use the log4j product. * **AMPS Java Kerberos authenticator** This component is provided through a github repository, and is not included with the AMPS distribution or the AMPS client distribution. This example includes code that uses the Simple Logging Facade for Java. This example does not include log4j, but could be configured at deployment time to use log4j. * **AMPS Apache Flume integration** This component is provided through a github repository, and is not included with the AMPS server distribution or the AMPS Java client distribution. This component includes code that uses the Simple Logging Facade for Java. This example does not include log4j, but could be configured at deployment time to use log4j. In summary, no Java code provided by 60East requires or uses log4j. It is possible for applications based on this code to use log4j. However, no modification to 60East-provided components or code is necessary to remove or update any use of log4j. --- # Which Message Type is Best For You? Picture of a box of chocolates. From Day-1, we’ve built AMPS to be content aware, yet message-type agnostic. As such, we’re often asked which message-type we think is best. The best message type, in most situations, is dependent on the use-case. In this article, we drill-down into what factors you should consider when selecting a message type, the benefits/drawbacks of each message-type and the functionality trade-offs specifically when it comes to AMPS. ## Message Type Considerations There are a gazillion message types one could use, each having been created to offer some distinct benefit over other existing message types. For example, the FIX message type is used within financial services and is simple to parse, serialize, and view over a network. The Protobuf message type is designed to be a generic binary message type with a format enforced by a schema definition. When selecting the best message type for your use case, it’s a good idea to consider the following: serialization, parsing, language support, and size. ### Serialization and Parsing Whatever format you select, messages will need to be serialized into that format. If your use-case is performance critical, then you’ll want to look at the serialization performance for the types of messages you’ll be sending. **Make sure the programming languages your team uses has support for efficient serialization from data structures into the message format and parsing back into a data structure.** If you’re a performance critical use case, as many AMPS customers are, you’ll want to pay careful attention to any garbage collection or wasted cycles in the parsing path. For example, if you receive a 2KB message, is there an efficient way to get just a single value out of that message or does the entire message need to be parsed? You’ll want to consider the full path from message construction (serialization) through to the consumption of that message (parsing) when determining which message type is the best fit for your use-case. For example, general message formats such as Protobuf could be a great message type for a back-end system written in a variety of programming languages. However, if the final target of the message is a Javascript application running within a browser, then JSON is likely a better message type choice to optimize for the user experience, UI response time, and even battery lifetime (for mobile apps) – JSON was developed from the Javascript object model, and browsers have built-in parsers for JSON that are much more efficient than the parsers for any other format. ### Message Size Message types encode values and data types in different ways. You’ll want to make sure your message type choice has an acceptable data type “bloat” for the data types you plan using. For example, a message type like a simple “C-struct” or Protobuf message can efficiently encode a large array of 1000 double-precision floating point values in around 8000 bytes. However, using BSON or JSON could easily be 1.5x to 3x larger to store the same large array. There can be a large variance on encoded data size between message types, so you’ll want to test the message types you are considering with the data you plan on using. ## Message Type Properties The 3 most important properties of message types are if the message type is “binary” (as opposed to utf-8, latin-1, ASCII, etc.), whether the type supports a hierarchical structure, and whether the type requires formal schema definition documents. Each property has unique benefits and drawbacks that can dramatically impact system performance, developer productivity, and future flexibility. ![Diagram of Capabilities](/img/blog/message_types/MessageTypeVennDiagram.svg "Diagram of Capabilities") ### Binary Message Types Message types that encode their data in binary form have a distinct advantage of being able to maintain the precision of their data. Message types that are encoded in UTF-8 or ASCII can lose floating point precision during serialization and parsing. If you need to transmit 64-bit floating point numbers without any loss of precision, then using a binary message type may be your only choice. On the other hand, if your data only needs a few decimal places of precision, then this loss of precision may not matter. ### Hierarchical Message Types Some message types work best for “flat” message layouts, while others are designed to work with hierarchical data. Hierarchical messages are, in our experience, more expensive (in time and space) to serialize and parse. If you’re using hierarchical messages, then you’ll typically be leveraging “array” data structures as well, which add to the complexity of your parsing and more surface area to your application testing. Some applications need a hierarchical structure to accurately represent the data. In other cases, though, a hierarchical structure isn’t necessary and using a flat (or flatter) data structure can improve performance. ### Schema Definitions When you application receives a stream of bits, it needs to understand the message framing and how to extract data from the message. Message types with schema support require the serialization into a specific format matching the schema that programs parsing the message can later use to determine the message layout. Messages without explicit schema definitions will have an implicit schema encoded in the message layout itself, otherwise a parser of the message won’t know which bits correspond to which data. For example, schema-based Protobuf, can use the following schema definition for an Order: ```cpp showLineNumbers message Order { int32 id = 1; int32 quantity = 2; float price = 3; int32 product_id = 4; } ``` A single order will take 16-bytes on the wire and the receiver of the Order message will use the Protobuf schema (the .proto file) to decipher what those 16-bytes mean in terms of the Order properties. On the other hand, if we were to send a similar message in a schema-less message type like JSON, the message could look something like the following: ```js {"id": 1, "quantity": 100, "price": 123.45, "product_id": 42} ``` It’s easy to see that the JSON message is much larger than 16-bytes, because the message schema/layout is encoded in each message itself. Schemas are fantastic ways to reduce the size of messages, but they come at a cost with reduced system flexibility and developer agility. For example, adding a “timestamp” attribute to our Order example would require a new Schema is rolled-out to all producers and consumers of the message – otherwise a producer may be producing an older Order without a “timestamp” while a consumer expects the message to have a “timestamp” attribute. Compare that to JSON, where the producers can inject the new timestamp attribute at anytime and the consumer will see it and be able to use it when it exists. Some teams see the schema-based types as too rigid and inflexible, while others see the fluidity of message types like JSON as having a dangerous lack of contract between the producers and consumers of the messages. There’s no single answer that works for all applications, but it’s important to consider the tradeoffs and be aware of what constraints you are choosing. | Message Type | Schema | Binary | Hierarchical | | --- | --- | --- | --- | | JSON | ○ | ○ | ⬤ | | FIX | ◍ | ○ | ◍ | | XML | ◍ | ○ | ⬤ | | ProtoBuf | ⬤ | ⬤ | ⬤ | | MessagePack | ○ | ⬤ | ⬤ | | BSON | ○ | ⬤ | ⬤ | | BFlat | ○ | ⬤ | ○ | | C-Structs | ⬤ | ⬤ | ⬤ | ⬤: Full Capability ◍: Partial Capability ○: No Capability ## End-to-end Performance One of the reasons we’ve built AMPS to be message type agnostic, is because we’ve found in decades of working on high-performance systems, that one of the most common places to unnecessarily introduce latency is message type conversions. For example, if you’re storing FIX data into a classic RDBMS you need to convert the FIX data into SQL insert statements that map the FIX data to columns within the database table. If you want to later read that record and format into JSON, you need to select the data out and then convert into JSON. Even in cases where the message type of the producer and consumer are the same, if the intermediary data store or broker requires a different data format, then you’ll add latency to your messaging with conversions alone. ![Converting formats takes time](/img/blog/message_types/MessageTypeConversionsLatency.svg "Converting formats takes time") When your datastore or broker can natively store data in at least the producer or consumer’s format, this can cut down the latency costs by 1/2. If the datastore or broker, producer, and consumer can all natively use the same format, then there’ll be no latency increase due to format conversions. Therefore, an additional consideration for your message type selection should be the intermediary systems that these messages pass through. Make sure you can store, query, and retrieve data in your message type of choice. ## AMPS and Message Types We’ve worked hard to make the message type you select to use with AMPS a choice of functionality and policies of your choosing. This is reflected in our messaging performance, which at a 10% match rate you can see the content filtered performance of every message type is outstanding – maxing out at nearly 1 million messages per second per CPU core for every message type (except for that darn XML – which is, unfortunately, both complex and verbose!) The graph below shows the results with relatively small messages, using content filtering with a 10% match rate, and running on a relatively fast (as of March 2022) machine. ![Message type performance -- no bad choices](/img/blog/message_types/MessageTypeThroughput.svg "Message type performance -- no bad choices") AMPS is designed to let you choose which message type works best for you. Even better, there’s no need to be restricted to the list of message types provided with AMPS, because we have APIs for extending AMPS functionality and content-awareness to other message types. This means we (or customers) can include new message types without changing the AMPS server itself. Most functionality is supported on every type that ships out-of-the-box with AMPS. However, there are key differences with Protobuf when it comes to delta messaging (you need to be using Protobuf version 2 or greater than 3.15) and with real-time aggregation. Real-time aggregation doesn’t work with strict schema data types, since it’s not necessarily possible for AMPS to guarantee that the aggregation is valid for the schema (except for within real-time aggregated JOINs that have a result message type in one of the other message types supporting real-time aggregation.) | AMPS Message Type | Content Filtering | Delta Messaging | Realtime Aggregation | | --- | --- | --- | --- | | JSON | ⬤ | ⬤ | ⬤ | | FIX | ⬤ | ⬤ | ⬤ | | XML | ⬤ | ⬤ | ⬤ | | ProtoBuf | ⬤ | ⬤ | ○ | | MessagePack | ⬤ | ⬤ | ⬤ | | BSON | ⬤ | ⬤ | ⬤ | | BFlat | ⬤ | ⬤ | ⬤ | | C-Structs | ⬤ | ○ | ○ | | “Unparsed Binary” | ○ | ○ | ○ | ⬤: Full Capability ○: No Capability ## Still Struggling? Hopefully these considerations help with selection of the best message type for your application or solution space. If you’re still struggling with what to choose, then we’d suggest these tips: * Web/mobile applications are increasing in popularity: go with JSON if you can. * If you’re always in a high-frequency feedback loop with your own users/customers, select a flexible (non-schema) message type, such as JSON or MessagePack. * Minimize data hierarchy within your messages (the “flatter” the better) – go at most one level beyond the “root” level. * For serialization and parsing performance, don’t ever use XML or BSON. * If you have the choice of which fields to include in your messages, include fewer fields for better performance. --- # Reloaded: Monitor Your AMPS Instances with Prometheus and Grafana > Learn about integrating AMPS Admin API with Prometheus and Grafana in this blog post! We wrote this several years ago, and it remains true: modern data processing systems are complex and often consist of several sub-systems from various vendors where each individual subsystem typically exposes some sort of monitoring interface with its own metrics, format, authentication and access control. In order to keep such complexity under control and be able to monitor whole system state in real-time and in the past, standard monitoring packages have emerged. More than ever, most customers we work with no longer build end-to-end monitoring systems themselves, but instead build custom dashboards using off-the-shelf monitoring software. It makes sense to focus on the metrics that are important to the business and the application rather than the low-level details of creating the framework. One popular package is [Prometheus](https://prometheus.io/). Prometheus is an open-source product created to collect and aggregate monitoring data and stats from different systems in a single place. Prometheus conveniently integrates with [Grafana](https://grafana.com/), another open source tool that can visualize and present monitoring stats organized in dashboards. In this post, we demonstrate how to integrate the built-in [AMPS Admin API](/docs/amps-monitoring-guide) with Prometheus, and thus, with Grafana in order to monitor and visualize various AMPS metrics. To integrate AMPS data into Grafana, we're going to need to do a few things: - configure [AMPS Admin API](/docs/amps-monitoring-guide) - create a data exporter that exposes data from [AMPS Admin API](/docs/amps-monitoring-guide) in a format recognized by Prometheus - configure Prometheus to use the AMPS data exporter - configure Grafana to use Prometheus as a data source As usual, all of the files used in this article are available on [Github](https://github.com/60East/amps-integration-prometheus). If you've never worked with Prometheus or Grafana before, you can find detailed quick start guides here: - Prometheus: [https://prometheus.io/docs/prometheus/latest/getting_started/](https://prometheus.io/docs/prometheus/latest/getting_started/) - Grafana: [http://docs.grafana.org/guides/getting_started/](http://docs.grafana.org/guides/getting_started/) ### Configure AMPS Admin API AMPS has a built-in Admin module that provides metrics using a RESTful interface. All it takes to enable the monitoring and statistics interface is to add the `Admin` section in the AMPS configuration file: ```xml showLineNumbers ... 8085 stats.db 1s ... ``` If your configuration already has the [Admin API](/docs/amps-monitoring-guide) enabled, just take note of the port number used for the [Admin API](/docs/amps-monitoring-guide). Otherwise, you can simply add the `Admin` section that exposes the [Admin API](/docs/amps-monitoring-guide) at the specified URL. (`http://localhost:8085`) and also stores stats in a file (`stats.db`) Once you've prepared the configuration file, start AMPS. The full configuration file for the demo is available on Github for your convenience. The detailed description of every metric AMPS provides is available in the Monitoring guide [here](/docs/amps-monitoring-guide). ### Plan Data Collection AMPS offers a wide variety of metrics, and not all metrics will be useful for every installation. (For example, although there is a lot of information about message queues available, those metrics aren't useful for applications that use a fan-out messaging pattern rather than using message queues for compettitive consumption). In the updated sample dashboard, we include basic metrics for: * Host level load * Memory * I/O * Disk usage * CPU load * Instance level metrics * Overall incoming messages by processor type * Metrics for topics in the SOW (including views and queues) * Insert, update and delete counts (numbers since AMPS started) * Insert, update, query, and delete counts per second (averaged over each sample interval) * Metrics for views * Number of in-flight updates for each view * Metrics for queues * Age of oldest message in the queue * Current queue depth * Replication-related statistics * Metrics for replication destinations * Connection state (currently connected or not) * Transaction log replay point for this destination (`seconds_behind`) * Messages sent per second (averaged over each sample interval) * Metrics for client connections * Bytes in and out per second (averaged over each sample interval) * Network buffer metrics (for both send and receive buffer) * Messages buffered in AMPS for this client (oldest message and current count) These metrics provide a general-purpose minimal dashboard for AMPS. We encourage you to use this as a starting point. Remove any statistics that don't make sense for your installation, and add any statistics that are important for your application and environment. ### Create an AMPS data exporter for Prometheus In order to add AMPS monitoring stats to Prometheus we will need a custom _exporter_. Exporters are applications that convert monitoring data into a format recognized by Prometheus. The detailed guide on how to write Exporters is available [here](https://prometheus.io/docs/instrumenting/writing_exporters/). Depending on the language you want to use you might utilize one of the official client libraries available [here](https://prometheus.io/docs/instrumenting/clientlibs/). In our demo, we will be using Python since Python is simple to use and allows us to focus on the exporter's logic. As with the configuration file, all the files mentioned in this section are in github. Now we'll need to create the exporter application that you can run as a python app. First, make sure you've installed the dependencies for the Prometheus client: ```bash pip install requests prometheus_client ``` Our exporter will need a custom collector -- a special class that collects data from AMPS upon receiving a scrape event from Prometheus: ```python showLineNumbers from prometheus_client.core import GaugeMetricFamily class AMPSCollector(object): def get_stats(self): """ This method collects stats from AMPS at the moment of the scrape event from Prometheus. It can also handle all the required authentication / custom HTTP headers, if needed. """ return requests.get( 'http://localhost:8085/amps.json' ).json() def collect(self): # load currents stats from AMPS first stats = self.get_stats() # update the metrics -- add # whichever metrics you need to # monitor here. yield GaugeMetricFamily( 'amps_instance_clients', 'Number of currently connected clients', value=len(stats['amps']['instance']['clients']) ) yield GaugeMetricFamily( 'amps_instance_subscriptions', 'Number of currently active subscriptions', value=len(stats['amps']['instance']['subscriptions']) ) yield GaugeMetricFamily( 'amps_host_memory_in_use', 'The amount of memory currently in use.', value=stats['amps']['host']['memory']['in_use'] ) # The repository has more metrics with more # advanced collection -- check it out! ``` To add an exposed metric, we use a `GaugeMetricFamily` object. For example, in the above sample, we expose the metric `amps_instance_clients` that corresponds with the number of `Client` objects reported in the [Admin API](/docs/amps-monitoring-guide) at the `/amps/instance/clients` path. Most AMPS metrics can use the `gauge` metric type since it's a simple value that can be set at each interval. You can read more about Prometheus metrics types [here](https://prometheus.io/docs/concepts/metric_types/). The collector class only has a single required method -- `collect()`. The `collect()` method is called upon a scrape event. Once called, the method is responsible for populating metrics values which are gathered from AMPS via a simple `GET` request to the [Admin API](/docs/amps-monitoring-guide). We request data in the `JSON` format by adding `.json` at the end of URL since JSON is easily convertible into native Python lists and dictionaries. Second, we need to register our AMPS collector within the Prometheus client: ```python showLineNumbers from prometheus_client.core import REGISTRY REGISTRY.register(AMPSCollector()) ``` Finally, we start the HTTP server supplied by the client that will serve the exporter's data: ```python showLineNumbers from prometheus_client import start_http_server if __name__ == '__main__': # Start up the server to expose the metrics. start_http_server(8000) # keep the server running while True: time.sleep(10) ``` The above code uses a custom collector to properly request data from AMPS and expose it to Prometheus at the moment of a scrape event. Depending on the policies at your site, you might modify the `get_stats()` method to add authentication / entitlement handling, if needed. More information about securing [AMPS Admin API](/docs/amps-monitoring-guide) is available [here](/docs/amps-user-guide/securing). Start the exporter application and it will expose an HTTP interface at `localhost:8000` for Prometheus to scrape: ```bash python amps-exporter.py ``` That's it: our custom exporter is complete! For more details on the Prometheus Python client, see the manual, available [here](https://github.com/prometheus/client_python). ### Configure Prometheus to use the AMPS data Exporter Now we need to configure Prometheus to utilize the new scrape target (that is, the service provided by the exporter) that we just created. To do this, add a new `job` to the configuration file: ```yaml showLineNumbers global: # Set the scrape interval to every 10 seconds. # Default is every 1 minute. scrape_interval: 10s scrape_configs: - job_name: 'amps_stats' # Override the global default # and scrape targets to every 1 seconds. # (should match AMPS > Admin > Interval settings) scrape_interval: 1s static_configs: - targets: ['localhost:8000'] labels: group: 'AMPS' ``` In the above example, we add the job and also override the `scrape_interval` value to match the AMPS Admin statistics interval value we set in the first step. Since that's the interval at which AMPS refreshes statistics, it's not especially useful for Prometheus to ask for statistics on a more frequent interval (though if the visualization does not need to be as granular as the statistics interval, it could be reasonable to ask for statistics _less_ frequently). We set the `scrape_interval` at the job level since several AMPS instances can be monitored, and each instance might have a different statistics interval. Once configured, Prometheus can be started with this configuration file: ```bash ./prometheus --config.file=prometheus.yml ``` That's all it takes to start collecting AMPS statistics into Prometheus! ### Configure Grafana to use Prometheus as a data source Of course, statistics are more useful if there's a way to visualize them. That's where Grafana comes in. Once the data is in Prometheus, adding it to Grafana is straightforward. Navigate to Grafana and add Prometheus as a Data Source. The detailed instructions on how to do this are available [here](https://docs.grafana.org/features/datasources/prometheus/). The only setting you'll need to modify for our example is the URL: `http://localhost:9090`. After the data source is added, building the dashboard is pretty straightforward -- you can choose different graphs, thresholds and re-arrange widgets on the page. In this version of the dashboard, we show results for the mertics discussed above. Here's a screenshot of the dashboard: ![AMPS Grafana Dashboard](/img/blog/grafana/grafana_dashboard_2022.png "AMPS Grafana Dashboard") The dashboard is included in the [github repository](https://github.com/60East/amps-integration-prometheus). Notice that, when you load it, you will need to replace the UID of the datasource in the sample dashboard with the UID of the datasource you created in Grafana -- Grafana does not adjust the reference. ### To Infinity and Beyond! In this post, we've just scratched the surface of how the [AMPS Admin API](/docs/amps-monitoring-guide) can be integrated with Prometheus and Grafana. Many additional metrics are available and there are a wide variety of ways those metrics can be visualized. Since Prometheus can collect data from a wide variety of sources, you can also combine data on the AMPS instance with data about other parts of the application, giving you full end-to-end monitoring. For further reading, here are some more articles about AMPS monitoring: - [Get more AMPS with Galvanometer](/blog/get-more-amps-with-galvanometer/) - [The Canary Sings! AMPS and ITRS Geneos 4.0](/blog/monitoring-amps/) Have a recipe that isn't listed here? Know a great trick for monitoring AMPS with Prometheus, or have a cool technique that isn't mentioned here? What dashboard would you build? What other systems would you monitor together with AMPS? Let us know in the comments! --- # Cascading Mesh Replication Configuration Image of network cabling arranged to resemble a blossom There are several popular patterns for creating a replicated set of AMPS instances. One popular pattern is a "cascading mesh", or a set of instances that receives publishes in one set of instances and then distributes those messages to other sets of instances. This blog post describes a common approach to creating a mesh of this type, explains how the configuration works, and discusses the tradeoffs involved in this approach. ## Cascading Mesh Topology The sketch below shows the topology of the cascading mesh. ![Diagram of the replication mesh as described in the following paragraphs](/img/blog/ReplicationMeshDiagram.png "Diagram of replication mesh") The first tier of the mesh has two instances that receive publishes from applications. These instances are part of the "IncomingMessages" group, and replicate both to each other and to the next tier in the mesh. The second tier of the mesh provides services to subscribers. This tier has two instances that are part of the "LiveProdSubscriptions" group. These instances replicate to each other and also the next tier in the mesh. The third tier of the mesh provides a non-production environment for application development and testing. This tier has two instances that are part of the "DevUAT" group. These instances replicate to each other and to the last tier in the mesh. The final tier in the mesh provides an environment for archival and audit. This tier has two instances that are part of the "ArchiveAudit" group. These instances replicate to each other, but do not replicate anywhere else. This topology is designed to meet the following requirements: * All messages are available on all tiers. * The mesh can survive the loss of a server on any tier (and, in fact, could survive the loss of one server on _each_ tier) and still deliver messages. * Different types of usage are isolated to different instances. Only approved, production-ready applications will be allowed to access the "LiveProdSubscriptions" instances. An archive of the configuration for these instances [is available for download](../static/downloads/mesh-configuration.zip). ## AMPS Replication For a basic introduction to AMPS replication, see [From Zero to Fault-Tolerance Hero](/blog/replication-intro) and the [AMPS User Guide](/docs/amps-user-guide) section on [High Availability](/docs/amps-user-guide/ha/ha-details). For the purposes of creating a cascading mesh, the following aspects of AMPS replication are most important: * AMPS replication is always point-to-point, from an originating instance to a receiving instance. * By default, AMPS replication does _not_ replicate messages that arrive over a replication connection. An instance in a mesh uses the ``PassThrough`` directive to further replicate messages that arrive at the instance via replication. This isn't necessary if there are only two instances in use, but is typically _required_ for a multi-instance installation to replicate correctly. `PassThrough` in a given `Destination` specifies that when a message is received via replication from an instance in a matching ``Group``, it is eligible to be replicated by this instance to this `Destination`. If no ``PassThrough`` is present in the `Destination` configuration, replicated messages will not be further replicated by this instance. Notice that the `PassThrough` setting applies to the `Group` name of the instance the message is received from, and not any other instance that the message may have been replicated through. * An outgoing message stream from an originating instance is specified by adding a `Replication/Destination` to the configuration of the outgoing instance, one per outgoing message stream. * AMPS duplicate detection uses the name of the publisher and the sequence number assigned by the publisher. If the same message is received more than once (for example, over different replication paths), only the first copy received is written to the transaction log. Any other copy will be discarded. Notice that, if there are multiple paths to an instance, there is no guarantee as to which path the first message will take. This is especially important in a failover situation or if there is network congestion, but can also be true even when the network, hardware, and AMPS instances are all performing as expected. * AMPS replication validation, by default, ensures that both sides of a replication connection provide a complete replica of the messages being replicated. Since this topology intentionally does not replicate messages from the lower-priority environments to the higher priority environments, the configuration will need to relax some of the validation rules. The diagram above shows one possible arrangement of replication connections for this mesh. To make this application more resilient, each instance in a given tier will be configured to fail over replication to _either_ of the instances in the next tier. ### Determining the Outgoing Destinations To be sure that every message reaches every instance in the mesh, each instance in the mesh replicates to the other instance at the same tier. Each instance also insures that messages are replicated to the next tier. Since the instances in the next tier will also replicate to each other, the configuration uses a single ``Destination`` with the addresses of both the instances in the next tier as failover equivalents. Once a message reaches an instance in the next tier, that instance will be sure that the message is replicated to the other server in the tier. ### Determining the PassThrough Configuration For a mesh like this, where the intent is for each message to reach every instance in the mesh, it's important that each instance pass through messages from every upstream instance. Since the intent is to pass through every message, the easiest way to specify `PassThrough` is to provide a regular expression that matches any group. AMPS configurations, by convention, typically use the regular expression `.*` (any quantity of any character) to match anything. If it were necessary to be more explicit, though, for every instance, the configuration would pass through the group names for every instance from which it could receive an incoming message. For example, as shown in the following diagram, when a message is initially published to `Incoming-A`, the `Live-B` instance could receive the first copy of that message from the `Incoming-B` instance in the `IncomingMessages` group, from the `Incoming-A` instance in the `IncomingMessages` group, or from the `Live-A` instance in the `LiveProdSubscriptions` group. A `PassThrough` configuration for an outgoing `Destination` from this instance must specify at least `IncomingMessages|LiveProdSubscriptions` for the `Live-B` instance to replicate the message further. For convenience, an instance would typically specify `.*` to match any incoming `Group`. ![Diagram of the replication mesh showing that Live-B may receive messages from Incoming-A, Incoming-B, or Live-A](/img/blog/IncomingToLive-B.png "Diagram of replication mesh showing that Live-B may receive messages from Incoming-A, Incoming-B, or Live-A") Likewise, the `Dev-A` instance could receive a replicated message from the `Live-A`instance in the `LiveProdSubscriptions` group, from the `Live-B` instance in the `LiveProdSubscriptions` group, or from the `Dev-B` instance in the `DevUAT` group. A `PassThrough` configuration for an outgoing `Destination` from this instance should specify at least `LiveProdSubscriptions|DevUAT`. Again, for convenience, an instance would typically specify `.*` to match any incoming `Group`. ### Sync or Async Acknowledgement? One other choice that needs to be made for each replication destination is the type of acknowledgement to be used for that destination. The acknowledgement type controls when the instance of AMPS considers a message to be safely persisted and acknowledges that persistence. With `sync` acknowledgement, an instance of AMPS will wait for the replication destination to acknowledge that a message has been persisted before considering the message to be safely persisted and acknowledging the message as persisted (to either a publisher or an upstream replication instance). With `async` acknowledgement, an instance of AMPS will consider the message to be safely persisted when it is persisted to the local transaction log. The acknowledgement type doesn't affect how quickly AMPS replicates messages. However, because `async` acknowledgement may acknowledge a message before the message has been received by a downstream instance, it is important to be sure that a message source (either a publisher or an upstream instance) does not fail over between two AMPS instances that replicate over an `async` connection. For this mesh, this means that we can use `async` replication between tiers, but we need to use `sync` replication within a tier. For the `IncomingMessages` tier, a publisher may fail over between `Incoming-A` and `Incoming-B`, so those instances must use `sync` acknowledgement to replicate to each other. In the other tiers, a replication connection may fail over between an `-A` instance and a `-B` instance, so those connections must use `sync` acknowledgement. Since a connection will not fail over from one tier to another, and since every instance fully replicates the message stream (thanks to the PassThrough configuration discussed in the previous section), it's reasonable to use `async` acknowledgement between the tiers. This can reduce the storage needs for the applications that publish to the `IncomingMessages` instances, since messages could potentially be acknowledged to those publishers before the messages are replicated to all of the downstream instances. ### Validation Rules As mentioned earlier, by default AMPS replication validation tries to ensure that messages published to a replicated topic on any instance will be delivered to all other replicated instances. In many cases, this is exactly what replication is intended to do. This replicated mesh, though, does not replicate messages from a lower-priority environment to a higher-priority environment. Because of this, the configuration will need to relax some of the validation checks. Between tiers (for example, replication between `Live-A` and either `Dev-A` or `Dev-B`), messages are only replicated in one direction. The `replicate` validation check ensures that topics that are replicated *to* a given instance are also replicated *from* a given instance. To allow messages to be replicated in only one direction, the configuration excludes the `replicate` validation check. Within a tier (for example, replication between `Live-A` and `Live-B`), every message will be fully replicated, so there is no need to disable the `replicate` validation check within a tier. However, the `cascade` validation check ensures that every destination that messages are replicated to enforces the same replication checks to its own destinations. This means that, within a tier, the configurations need to exclude the `cascade` validation check. Because the configurations exclude the `cascade` validation check within a tier, this means that the tier that replicates to each tier must also exclude the `cascade` validation check. So, for each tier, we need to relax replication validation as follows: * Within a tier, the configurations will exclude the `cascade` validation check. * Between tiers, the configurations will exclude the `replicate` and `cascade` validation checks. ### Putting The Plan Together To summarize the steps we go through to put the plan together, we have the following steps for creating a cascading mesh: * Sketch out the mesh: * Each instance should replicate to every other instance in the same tier. * Each instance should have an outgoing connection that can fail over to any of the instances on the next tier. * For every instance, ensure that the passthrough configuration is either `.*`, or includes the Group name for every incoming replication connection. * If there are any replication connections where it is not possible for either a publisher or a replication connection to fail over from one side of the connection to the other side of the connection (in this example, connections between tiers), consider whether using `async` acknowledgement might reduce the resource needs for publishers by providing acknowledgement more quickly. For example, the Replication section of the `Incoming-A` instance would be as follows in the example configuration: ```xml showLineNumbers Incoming-B IncomingMessages .* .* json cascade sync localhost:4002 amps-replication LiveProdSubscriptions .* .* json replicate,cascade async localhost:4101 localhost:4102 amps-replication ``` The destination configuration for `Live-A` follows a similar pattern. Notice that since the `LiveProdSubscriptions` group is not intended for new messages to be published, the instance does not specify replication back to any instance in the `IncomingMessages` group. The `Replication` for `Live-A` looks like this: ```xml showLineNumbers LiveProdSubscriptions .* .* json cascade sync localhost:4102 amps-replication DevUAT .* .* json replicate,cascade async localhost:4201 localhost:4202 amps-replication ``` ### Try it Yourself An archive of the configuration for these instances [is available for download](../static/downloads/mesh-configuration.zip). These configuration files are set up to run on a single system, but can easily be adapated to a multi-server configuration (by adding the host name for each instance to the replication InetAddr). --- # Bookmark State Without a Filesystem: Ultimate Director's Cut Image of books on a shelf with bookmarks between the pages AMPS bookmark subscriptions provide a way for applications to resume subscriptions in the event of a disconnection or failure of either the application or server. With a bookmark subscription, the application side manages the correct point to resume the subscription by setting a `BookmarkStore` on the client, and then discarding messages from the store once the application is done processing them. The clients offer both a memory-based bookmark store (which does not persist when the application restarts) and a file-based bookmark store (which is persisted on the file system). Those options work well for many AMPS deployments. In some deployments, though, no filesystem is available, but the application still needs to resume a subscription when it is restarted. This post explains how current versions of the AMPS client can use a topic in the AMPS State-of-the-World to store the point at which the application should be resumed. The post assumes a good working knowledge of resumable subscriptions (covered in detail in the AMPS User Guide and the AMPS Java Developer Guide), and also assumes some familiarity with the implementations of the AMPS clients. In older versions of the AMPS clients, you could do this by creating your own `BookmarkStore` (typically by wrapping a `MemoryBookmarkStore` to do the heavy lifting), as described on the 60East blog at [No Filesystem? No Problem! Keeping State in AMPS](/blog/no_filesystem_no_problem_keeping_state_in_amps) and [Keeping State in AMPS, Rebooted](/blog/keeping-state-in-amps-rebooted) blog posts. Current versions of the AMPS client libraries make this much simpler. The AMPS client libraries now include a new interface, `RecoveryPointAdapter`, that allows you to implement _only_ the storage for a bookmark store. Even better, the clients include a `SOWRecoveryPointAdapter` to store bookmark state in AMPS without having to develop an adapter at all. In this blog post, we'll cover what a recovery point is, what a recovery point adapter is, and how to use the built-in recovery point adapter to restore bookmark subscription state from a `Topic` in the State-of-the-World. If all you need to do is store recovery points in AMPS, feel free to skip ahead to `Bookmark Subscription Recovery from a SOW` ### What's a Recovery Point? A recovery point is a bookmark, or set of bookmarks, that the application provides to AMPS when a subscription is restarted so that the AMPS server can find the correct point in the local transaction log to resume that subscription. In the AMPS client libraries, a recovery point is represented by a combination of the subscription identifier and a string that includes the current stable recovery point for that application. A helpful way to think about the contents of the recovery point string is "one or more bookmarks that provide a useful place to resume a subscription". This can include one or more of: * The last bookmark discarded by the application * For a set of publishers, the last bookmark discarded by the application for each of those publishers * The last bookmark AMPS has provided as being fully persisted * A timestamp indicating the last time the store produced a recovery point Depending on the failover situation, one (or all) of these may be the correct point to recover from. For example, if the application goes for a long stretch of time without receiving messages (and all previously received messages have been discarded), it may be most helpful to start from the last bookmark that AMPS has provided as being fully persisted rather than the last message that the application discarded. If the bookmark store knows that there are no messages for this subscription between those two points, this could save a large amount of work on the server side to replay journals with no messages that match the subscription. Likewise, some applications may be offline longer than the retention period for the transaction log. In those cases, none of the bookmarks seen by the application (discarded or not) will be present in the AMPS instance, and it would be most helpful to restart at the timestamp, which will be before the beginning of the transaction log. If no timestamp were provided, no bookmark in the recovery point would be present, and AMPS would restart the subscription from the _end_ of the transaction log rather than the _beginning_ of the transaction log. The `MemoryBookmarkStore` includes logic for managing these situations and determining the best recovery point to use for a given subscription. One of the advantages of a `RecoveryPointAdapter` is that your application doesn't need to figure out the best recovery point for the subscription. The `MemoryBookmarkStore` provides the best current recovery point to your adapter when the recovery point changes. ### What's a Recovery Point Adapter? A recovery point adapter has two responsibilities to the `MemoryBookmarkStore` that uses the adapter: * Persist recovery points provided by the Store. The adapter itself chooses how and where the recovery points are persisted. When the ``MemoryBookmarkStore`` updates the recovery point for a subscription -- for example, when a message is discarded or a persisted ack arrives from AMPS -- the store calls the adapter with the current recovery point. * Produce the current set of recovery points when requested to by the Store. This happens when a subscription is created or restored. The exact interface that a recovery point adapter uses to provide the recovery points for a set of subscriptions varies depending on the programming language, but it is intended to be easy to implement. For example, the Java language recovery point adapter provides a set of recovery points using the standard `java.util.Iterable` and `java.util.Iterator` interfaces. When the `MemoryBookmarkStore` needs to update the recovery point for a subscription, the store calls the `update` method of the adapter with the recovery point that needs to be persisted. Notice that the adapter doesn't need to concern itself with deciding what the correct recovery point for a subscription should be. All of that logic is in the Store itself: the adapter just needs to store the recovery points provided by the Store and provide those recovery points to the Store when necessary. ### Bookmark Subscription Recovery from a SOW The `SOWRecoveryPointAdapter` in the AMPS clients uses a SOW topic in an AMPS instance for storage and retrieval of recovery points. Recovering from a SOW with a recovery point adapter is a matter of: * Configuring AMPS with a SOW topic for recovery points * Creating an AMPS client to use for storing and retrieving recovery points (separate from the client that will be used for the application) * Installing the recovery point adapter when the Store is constructed To be able to store recovery points in a SOW topic, the AMPS configuration needs to define the topic. The SOW needs to be able to store the recovery point for each distinct subscription for each distinct client. By default, the SOW recovery point adapter produces a JSON document that includes a `clientName` field with the name of the client and a `subId` field with the identifier for the subscription (the names and message format are customizable, see the API documentation for the interface). This means that the configuration for the SOW topic should be along the lines of: ```xml showLineNumbers /ADMIN/bookmark_store json /clientName /subId ./sow/%n.sow ``` This creates a topic named `/ADMIN/bookmark_store` in the instance. To use this topic to save and restore recovery points, you create an AMPS client that connects to the instance that stores the topic, and then use that client to create a `SOWRecoveryPointAdapter` for the Store to use. This client must be a different client than the one used by the application: it is reserved for recovery point storage. It's often convenient to wrap this creation in a separate method. For example, the following Java method sets up an HAClient to use a `SOWRecoveryPointAdapter` that will use the instance at `tcp://bookmark-storage-host:9007` to store recovery state: ```java showLineNumbers // Call setSowBackedBookmarkStore before calling connectAndLogon on the // HAClient that will use the SOWBookmarkStore. public void setSowBackedBookmarkStore(HAClient client_) throws AMPSException { // Construct a new client (to help ensure naming uniqueness, // this name is based on the name of the client the adapter will // be saving state for). HAClient recoveryPointStorage = new HAClient(String.format( "recoveryPointStorage-%s", client_.getName() )); // Substitute this with a production-ready ServerChooser // and an appropriate ReconnectDelayStrategy. Notice that // the bookmark store does not need to be on the same // instance as the client that works with actual data. DefaultServerChooser sc = new DefaultServerChooser(); sc.add("tcp://bookmark-storage-host:9007/amps/json"); recoveryPointStorage.setServerChooser(sc); // Connect the recovery point storage client recoveryPointStorage.connectAndLogon(); // Create a recovery point adapter SOWRecoveryPointAdapter adapter = new SOWRecoveryPointAdapter(recoveryPointStorage, // name of the client this adapter is tracking client_.getName(), // close recovery point client when Adapter is closed true, // Adjust these options as necessary // include last update timestamp (journal removal protection) true, // do not throw exceptions false ); // Set the bookmark store using the adapter. // In this case, initialize the store to expect up to // 5 simultaneous subscriptions (as a reasonable default) client_.setBookmarkStore(new MemoryBookmarkStore(5, adapter)); } ``` That's all that's needed to enable the SOW recovery point adapter for an HAClient. ### Reducing Bandwidth for Updates The `SOWRecoveryPointAdapter` will save the recovery point to the State-of-the-World each time a new recovery point is created. For an application that is actively processing messages, this could produce a relatively high volume of updates to the State-of-the-World topic. To reduce the bandwidth required to maintain the recovery point, the AMPS clients include a `ConflatingRecoveryPointAdapter` that wraps another adapter and passes along recovery points to that adapter at the specified interval. The interval consists of a number of updates and a time period: when the time period expires, or the number of updates is reached, the `ConflatingRecoveryPointAdapter` calls the wrapped adapter with the recovery point to be persisted. For example, to persist the recovery point to the AMPS server every 100 updates or 250ms, the example above could be modified to set the bookmark store this way: ```java showLineNumbers // The adapter object is a SOWRecoveryPointAdapter // constructed as above ConflatingRecoveryPointAdapter conflater = new ConflatingRecoveryPointAdapter( adapter, 100, // Number of updates 250, // Update interval in milliseconds 50 // Check thresholds every 50ms ); // Provide the conflating adapter to the MemoryBookmarkStore // instead of providing the SOWRecoveryPointAdapter directly client_.setBookmarkStore(new MemoryBookmarkStore(5, conflater)); ``` The `ConflatingRecoveryPointAdapter` starts its own background thread to manage timeouts and updates. When the adapter is closed, it writes all updates to the underlying adapter before closing that adapter. Setting the threshold for updates is a matter of how much tolerance the application has for receiving messages that have already been processed in the event that the application fails before an updated recovery point is delivered to AMPS. ### Other Considerations To the AMPS server, the topic that stores the recovery points is the same as any other topic in the State of the World. The topic can be replicated to other instances of AMPS to improve reliability and allow the client that saves the recovery point to fail over. As mentioned earlier, the topic can be included in the same instance that has application data, or can be stored in a completely different instance of AMPS. Like a `MemoryBookmarkStore` itself, an application that uses a `SOWRecoveryPointAdapter` must be able to handle some overlap in subscriptions (duplicate messages) in the event that the application fails (or the connection fails over) with messages that have not been discarded. --- # HTTP Preflight- Proxy Play: AMPS Unlocked Imagine you’re at an exclusive club. To get in, you first check with the bouncer (the HTTP preflight request) to make sure you’re allowed in. Only after approval can you step inside and enjoy the party. That’s exactly how HTTP preflight works when connecting TCP clients to AMPS via an HTTP proxy! With this feature, TCP clients can seamlessly pass through an HTTP proxy—just like WebSockets—while maintaining AMPS’s high-performance messaging capabilities. Whether you’re working with Python, Java, C++, or C#, this feature opens up new possibilities for flexible and efficient integrations. ### What is HTTP Preflight? Think of the HTTP preflight feature as a clever hack that lets TCP clients sneak into AMPS through an HTTP proxy by mimicking a WebSocket handshake. Not only does this open the door for seamless connections, but it also gives you the flexibility to pass custom HTTP headers. #### Real-World Use Case Consider a financial trading firm that relies on the AMPS Java client for secure messaging over TCPS while also using JavaScript-based view servers that connect via WebSockets. Their challenge? Minimizing open ports on their firewall while still supporting multiple AMPS transports. To address this, we introduced routing based on the URI using an Nginx proxy. This is where HTTP preflight becomes crucial. By leveraging HTTP Upgrade requests, we can allow TCP clients to pass through the same proxy used for WebSockets. This means trading infrastructure can maintain its AMPS-based architecture without exposing multiple ports. The proxy handles all routing, ensuring secure and efficient connectivity. We initially explored using Server Name Indication (SNI) over SSL/TLS to differentiate between transports. However, the Java Virtual Machine’s session caching behavior introduced unpredictable issues, making debugging difficult. By shifting to HTTP preflight, we eliminate this complexity while keeping the connection streamlined and secure. ### How It Works: The Handshake AMPS now supports HTTP Upgrade preflight requests for TCP/TCPS transports. This allows proxies like nginx to recognize these requests and route both WebSocket and TCP/TCPS connections through a single exposed port. * Client sends an HTTP Upgrade request to initiate the connection: ```http GET /user/can/put/path/here/amps/json HTTP/1.1 Host: host Connection: upgrade Upgrade: tcp ``` * AMPS responds with a 101 Switching Protocols message, allowing the client to proceed with a standard TCP handshake: ```http HTTP/1.1 101 Switching Protocols Upgrade: tcp Connection: Upgrade ``` * Client then sends its standard AMPS logon request and begins normal message exchange. ### AMPS Server Changes The AMPS TCP transport has been updated to: - Accept an HTTP GET request containing an Upgrade header. - Verify that the upgrade request is for tcp or tcps. - Respond with: ```http HTTP/1.1 101 Switching Protocols Upgrade: tcp Connection: Upgrade ``` Transition the connection to standard TCP/TCPS processing. * No AMPS configuration changes are required to enable http_preflight! Just set up Nginx reverse proxy. ### Client-Side Changes To enable HTTP preflight, clients must: - Set `http_preflight=true` (default is false). - Modify the client URI to include `/amps/?http_preflight=true`. Before: ```bash tcp://host:port/amps/json ``` After (with HTTP Preflight): ```bash tcp://host:port/user/can/put/path/here/amps/json?http_preflight=true ``` This signals the proxy to treat the connection as an HTTP Upgrade request. ### Ready to Try It Yourself? #### First: Set up Nginx (Fedora/RedHat) ##### - Install Nginx Proxy ```bash sudo dnf install nginx ``` ##### - Enable and Start the nginx service ```bash sudo systemctl enable --now nginx # enable: Configures nginx to start automatically at system boot. # --now: Starts the nginx service immediately ``` ##### - Configure nginx as a reverse proxy Modify `/etc/nginx/nginx.conf` (`sudo` access is required). The first step in configuring Nginx is setting the number of worker processes. This determines how many processes Nginx will use to handle incoming requests. For small applications, one worker process is usually sufficient. However, in production environments, it is recommended to set this to match the number of CPU cores on your server for better performance. You can determine the number of CPU cores using the command: `nproc`. Then, set `worker_processes` to that number. Next, we define how Nginx handles concurrent connections. This is done inside the `events` block. The `worker_connections` directive sets the maximum number of simultaneous connections a worker process can handle. If you expect high traffic, consider increasing this number, but ensure your server has enough resources to handle it. Next, add HTTP requests in the `http` block. The `include mime.types;` directive ensures that Nginx sets the correct `Content-Type` headers based on file extensions. `default_type application/octet-stream;` is a fallback for unknown file types. `sendfile on`; allows Nginx to send files efficiently by reading them directly from disk. `keepalive_timeout 65;` keeps idle connections open for 65 seconds before closing them. WebSockets require a persistent connection. To properly handle WebSocket upgrades, add a `map` directive inside the `http` block. This checks if the `Upgrade` header is set in an incoming request. If it is, Nginx will upgrade the connection; otherwise, it will close it. This is necessary for WebSocket applications. If your application consists of multiple services running on different ports, you can define upstream blocks within the `http` block to manage them efficiently. Each `upstream` block defines a backend server that Nginx will route requests to. This setup makes it easier to scale your application in the future by adding more servers to each upstream group. Now, we define the main `server` block within the `http` block, which tells Nginx how to handle incoming requests. `listen 80;` makes Nginx listen for requests on port 80 (HTTP). `server_name localhost;` means the server will respond to requests sent to `localhost`. Replace `localhost` with your domain if hosting publicly. Inside the `server` block, add location blocks to define how different types of requests should be handled. First, add a block that will proxy connections to `http://amps_admin` to the backend service running on port `8085`, as defined in the upstream `amps_admin` block earlier. Next, add a block that will proxy connections to `http://amps_tcp` to the TCP port on the server, which we defined in the upstream `amps_tcp` block earlier. Since this connection may involve WebSockets, we also include necessary headers to support upgrades. Similarly, we define a location block to handle WebSocket connections. This block will proxy requests from `/ws/` to the WebSocket server running on the TCP port we specified in `amps_ws`. ##### Nginx Configuration Example: ```nginx showLineNumbers worker_processes 1; # Event Handling events { worker_connections 1024; } # HTTP Configuration http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; # Websocket Upgrade Mapping map $http_upgrade $connection_upgrade { default upgrade; '' close; } # Upstream Server Definitions upstream amps_admin { server localhost:8085; } upstream amps_tcp { server localhost:9007; } upstream amps_ws { server localhost:9008; } # Server Block (Handles Incoming Requests) server { listen 80; server_name localhost; location /admin/ { proxy_pass http://amps_admin/; } location /client/ { proxy_pass http://amps_tcp/; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header Connection $connection_upgrade; proxy_set_header Upgrade $http_upgrade; } location /ws/ { proxy_pass http://amps_ws/; proxy_http_version 1.1; proxy_set_header Host $host;V proxy_set_header Connection $connection_upgrade; proxy_set_header Upgrade $http_upgrade; } } } ``` ##### Run following commands: ```bash sudo systemctl restart nginx # To restart nginx ``` ```bash sudo systemctl status nginx # Check if nginx is running ``` ```bash sudo systemctl stop nginx # Stop nginx ``` ```bash sudo systemctl start nginx # Start nginx ``` #### Second: Start AMPS Server ```bash .//ampServer config.xml ``` If you need help installing and starting AMPS, check out the [Getting Started With AMPS](/docs/intro-guide/getting_started) guide for instructions. #### Finally: Connect AMPS Clients to the AMPS Server - Install AMPS Python Client: ```bash pip install --user --break-system-packages amps-python-client ``` - Install AMPS JavaScript Client: ```bash npm i amps ``` Python Client with HTTP Preflight code sample: ```python showLineNumbers import AMPS import time # Create an AMPS client instance client = AMPS.Client('test') # Connect using HTTP preflight client.connect( 'tcp://localhost:80/client/amps/json?http_preflight=true' ) # Log on to AMPS client.logon() # Publish messages in a loop try: while True: client.publish('test', '{"hello":"world"}') time.sleep(1) except KeyboardInterrupt: print('Stopped by user. Closing connection...') client.close() print('Connection closed.') ``` What This Code Does: - Uses `http_preflight=true` in the connection URI. - Establishes a connection via AMPS and an HTTP proxy. - Publishes JSON messages every second. JavaScript Client code sample: ```javascript showLineNumbers import { Client } from 'amps' async function main() { const client = new Client('test') await client.connect('ws://localhost/ws/amps/json') console.log('connected!') await client.subscribe( message => console.log(message.data), 'test' ) } main() ``` What This Code Does: - It connects to the WebSocket server at `ws://localhost/ws/amps/json` :::info The default port for WebSocket `ws://` is `80` for non-secure connections. ::: - Subscribe to an AMPS topic called test. :::tip If the clients fail to connect after following the above steps on **Fedora/RedHat**, try running the following command: ```bash setsebool -P httpd_can_network_connect 1 ``` This command modifies a security setting in **SELinux** (Security-Enhanced Linux) to allow the **httpd** (web server) process to make network connections. It also ensures that this setting persists across system reboots. ::: HTTP preflight feature is available as of the following client releases: - C++ Client: 5.3.4.5 - C# Client: 5.3.4.0 - Java Client: 5.3.4.0 - Python Client: 5.3.4.5 For additional information, check out the [HTTP Preflight](/docs/amps-user-guide/transports/http-preflight) guide. ### Bringing It All Together With HTTP preflight, TCP clients get access to AMPS through an HTTP proxy. This feature: - Minimizes the number of open ports needed. - Enables flexible routing. - Works with existing WebSocket-compatible infrastructure. So next time you need to connect your TCP clients to AMPS via an HTTP proxy, just flip the swtich on `http_preflight`—and let the magic happen!