The AMPS Bridge

What if your application needs a light-weight way to move messages between AMPS instances?

AMPS Replication already reliably propagates messages to downstream instances. Individual replication links are configured with settings to filter out which topics and messages are replicated. Replication makes use of transaction logging to ensure that no messages are missed during temporary server or network outages, and ensures that messages are delivered as received.

Some applications have different requirements, though. For example, you might need to enrich data as it is moved from one instance to another, using reference data in a database or local cache. Or you might simply want to make a live stream of data from one instance available on another, without the logging overhead associated with replication (and without the reliability guarantees). The AMPS Bridge, a new sample application built on the AMPS C++ Client, meets these needs and more.

AMPS Bridge uses the AMPS C++ library to establish connections to a set of source and destination instances, and bridges messages from the message source to one or more destinations. Source code is included so you can transform or enrich messages as they move through AMPS Bridge. Since AMPS Bridge is built on our C++ client, it automatically takes care of detecting disconnects and reconnecting to the server when a connection is interrupted.

AMPS Bridge allows configuration of the each source/destination pair in a number of ways. First, you can choose which topic or topics to subscribe to, by supplying a topic name or regular expression – the AMPS server takes care of identifying which topics you mean when you supply a regular expression. You can choose any downstream topic you’d like to receive the messages. You can also supply an AMPS filter string to filter the messages that are bridged; that filter is processed on the source AMPS instance, so destination instances only receive relevant messages. Finally, you can opt-in to using bookmark subscriptions and store-and-forward publishing, if your source and destination are enabled with a transaction log.

Included here are the source code, a makefile, and a configuration file. You’ll need the AMPS C++ Client, version 3.2 or greater, installed and built on your machine; change the AMPS_CPP_DIR value in the makefile to point to where your client is installed. You’ll also need one or more AMPS servers, version 3.3 or greater, installed and running. Try running it as-is by modifying config.csv to point to an amps instance and topic you’d like to bridge, and then just execute:

./amps_bridge config.csv 

to start it.

If you open the code, you’ll find a few classes of interest:

• amps_bridge is the “main” class of the application; it manages a map of client connections and kicks off subscriptions.
• bridge_subscription reads a line from the CSV configuration file supplied, and then subscribes to the specified topic to get things started. Note that when HA is specified in the configuration file, we use bookmarkSubscribe() instead of regular subscribe(): this is a heavier weight operation on the server and client, but ensures there are no gaps between messages when reconnecting to a server after a disconnect. Also notice that when we use bookmarkSubscribe(), there is an additional, required, call to discard() in the message handler to indicate that the message has been processed. Finally, the filter() method is called by the message handler to create an outgoing message from the incoming one. If you’d like to add more complex transformation or enrichment logic to AMPS Bridge, filter() is the place to do it.
• bridge_server_chooser is an AMPS::ServerChooser implementation that produces messages to stderr when a disconnect occurs and ten consecutive attempts to reconnect have all failed. This is a simple implementation: you could make this far more intricate, choosing to shut down the program after a period of disconnect, fire off an alert to a monitoring system, etc. Both “HA” and regular subscriptions use the HAClient, which provides automatic reconnection and resubscription even when using plain subscriptions. HAClient interacts with a ServerChooser to indicate failed connections and acquire additional servers for fail-over. In this implementation, we use the ServerChooser interface to find out when the HAClient loses a connection, but AMPS Bridge only supports one server URI per subscription.

To wrap it up, the AMPS Bridge solves different problems than AMPS Replication, and gives you a light-weight, customizable way to move, transform, and enrich messages. Let us know how you use it!

Download

Download the AMPS Bridge source code and makefile here: amps_bridge.tar.gz