Bookmark State Without a Filesystem: Ultimate Director's Cut

Image of books on a shelf with bookmarks between the pages AMPS bookmark subscriptions provide a way for applications to resume subscriptions in the event of a disconnection or failure of either the application or server. With a bookmark subscription, the application side manages the correct point to resume the subscription by setting a BookmarkStore on the client, and then discarding messages from the store once the application is done processing them. The clients offer both a memory-based bookmark store (which does not persist when the application restarts) and a file-based bookmark store (which is persisted on the file system).

Those options work well for many AMPS deployments. In some deployments, though, no filesystem is available, but the application still needs to resume a subscription when it is restarted. This post explains how current versions of the AMPS client can use a topic in the AMPS State-of-the-World to store the point at which the application should be resumed. The post assumes a good working knowledge of resumable subscriptions (covered in detail in the AMPS User Guide and the AMPS Java Developer Guide), and also assumes some familiarity with the implementations of the AMPS clients.

In older versions of the AMPS clients, you could do this by creating your own BookmarkStore (typically by wrapping a MemoryBookmarkStore to do the heavy lifting), as described on the 60East blog at No Filesystem? No Problem! Keeping State in AMPS and Keeping State in AMPS, Rebooted blog posts.

Current versions of the AMPS client libraries make this much simpler. The AMPS client libraries now include a new interface, RecoveryPointAdapter, that allows you to implement only the storage for a bookmark store. Even better, the clients include a SOWRecoveryPointAdapter to store bookmark state in AMPS without having to develop an adapter at all.

In this blog post, we’ll cover what a recovery point is, what a recovery point adapter is, and how to use the built-in recovery point adapter to restore bookmark subscription state from a Topic in the State-of-the-World. If all you need to do is store recovery points in AMPS, feel free to skip ahead to Bookmark Subscription Recovery from a SOW.

What’s a Recovery Point?

A recovery point is a bookmark, or set of bookmarks, that the application provides to AMPS when a subscription is restarted so that the AMPS server can find the correct point in the local transaction log to resume that subscription. In the AMPS client libraries, a recovery point is represented by a combination of the subscription identifier and a string that includes the current stable recovery point for that application.

A helpful way to think about the contents of the recovery point string is “one or more bookmarks that provide a useful place to resume a subscription”. This can include one or more of:

The last bookmark discarded by the application
For a set of publishers, the last bookmark discarded by the application for each of those publishers
The last bookmark AMPS has provided as being fully persisted
A timestamp indicating the last time the store produced a recovery point

Depending on the failover situation, one (or all) of these may be the correct point to recover from. For example, if the application goes for a long stretch of time without receiving messages (and all previously received messages have been discarded), it may be most helpful to start from the last bookmark that AMPS has provided as being fully persisted rather than the last message that the application discarded. If the bookmark store knows that there are no messages for this subscription between those two points, this could save a large amount of work on the server side to replay journals with no messages that match the subscription.

Likewise, some applications may be offline longer than the retention period for the transaction log. In those cases, none of the bookmarks seen by the application (discarded or not) will be present in the AMPS instance, and it would be most helpful to restart at the timestamp, which will be before the beginning of the transaction log. If no timestamp were provided, no bookmark in the recovery point would be present, and AMPS would restart the subscription from the end of the transaction log rather than the beginning of the transaction log.

The MemoryBookmarkStore includes logic for managing these situations and determining the best recovery point to use for a given subscription. One of the advantages of a RecoveryPointAdapter is that your application doesn’t need to figure out the best recovery point for the subscription. The MemoryBookmarkStore provides the best current recovery point to your adapter when the recovery point changes.

What’s a Recovery Point Adapter?

A recovery point adapter has two responsibilities to the MemoryBookmarkStore that uses the adapter:

Persist recovery points provided by the Store. The adapter itself chooses how and where the recovery points are persisted. When the MemoryBookmarkStore updates the recovery point for a subscription – for example, when a message is discarded or a persisted ack arrives from AMPS – the store calls the adapter with the current recovery point.
Produce the current set of recovery points when requested to by the Store. This happens when a subscription is created or restored.

The exact interface that a recovery point adapter uses to provide the recovery points for a set of subscriptions varies depending on the programming language, but it is intended to be easy to implement. For example, the Java language recovery point adapter provides a set of recovery points using the standard java.util.Iterable<RecoveryPoint> and java.util.Iterator<RecoveryPoint> interfaces.

When the MemoryBookmarkStore needs to update the recovery point for a subscription, the store calls the update method of the adapter with the recovery point that needs to be persisted.

Notice that the adapter doesn’t need to concern itself with deciding what the correct recovery point for a subscription should be. All of that logic is in the Store itself: the adapter just needs to store the recovery points provided by the Store and provide those recovery points to the Store when necessary.

Bookmark Subscription Recovery from a SOW

The SOWRecoveryPointAdapter in the AMPS clients uses a SOW topic in an AMPS instance for storage and retrieval of recovery points.

Recovering from a SOW with a recovery point adapter is a matter of:

Configuring AMPS with a SOW topic for recovery points
Creating an AMPS client to use for storing and retrieving recovery points (separate from the client that will be used for the application)
Installing the recovery point adapter when the Store is constructed

To be able to store recovery points in a SOW topic, the AMPS configuration needs to define the topic. The SOW needs to be able to store the recovery point for each distinct subscription for each distinct client. By default, the SOW recovery point adapter produces a JSON document that includes a clientName field with the name of the client and a subId field with the identifier for the subscription (the names and message format are customizable, see the API documentation for the interface). This means that the configuration for the SOW topic should be along the lines of:

<SOW>

  <!-- other topics, views, etc. here -->

  <Topic>
     <Name>/ADMIN/bookmark_store</Name>
     <MessageType>json</MessageType>
     <Key>/clientName</Key>
     <Key>/subId</Key>

     <!-- Storage/persistence configuration here.
        In most cases, this topic should be
        persisted to a file, but that is not
        a requirement.  -->

      <FileName>./sow/%n.sow</FileName>
  </Topic>

</SOW>

This creates a topic named /ADMIN/bookmark_store in the instance.

To use this topic to save and restore recovery points, you create an AMPS client that connects to the instance that stores the topic, and then use that client to create a SOWRecoveryPointAdapter for the Store to use. This client must be a different client than the one used by the application: it is reserved for recovery point storage.

It’s often convenient to wrap this creation in a separate method. For example, the following Java method sets up an HAClient to use a SOWRecoveryPointAdapter that will use the instance at tcp://bookmark-storage-host:9007 to store recovery state:

// Call setSowBackedBookmarkStore before calling connectAndLogon on the
  // HAClient that will use the SOWBookmarkStore.

  public void setSowBackedBookmarkStore(HAClient client_) throws AMPSException
  {
    // Construct a new client (to help ensure naming uniqueness,
    // this name is based on the name of the client the adapter will
    // be saving state for).
    
    HAClient recoveryPointStorage =
      new HAClient(String.format("recoveryPointStorage-%s", client_.getName()));
    
    // Substitute this with a production-ready ServerChooser
    // and an appropriate ReconnectDelayStrategy. Notice that
    // the bookmark store does not need to be on the same
    // instance as the client that works with actual data.
    
    DefaultServerChooser sc = new DefaultServerChooser();
    sc.add("tcp://bookmark-storage-host:9007/amps/json");
    recoveryPointStorage.setServerChooser(sc);
    
    // Connect the recovery point storage client

    recoveryPointStorage.connectAndLogon();
    
    // Create a recovery point adapter. 
    
    SOWRecoveryPointAdapter adapter =
      new SOWRecoveryPointAdapter(recoveryPointStorage,
        client_.getName(),  // name of the client this adapter is tracking
        true,  // close recovery point client when Adapter is closed

        // Adjust these options as necessary

        true,  // include last update timestamp (journal removal protection)
        false  // do not throw exceptions
        );
    
   
    // Set the bookmark store using the adapter.
    // In this case, initialize the store to expect up to
    // 5 simultaneous subscriptions (as a reasonable default)
 
    client_.setBookmarkStore(new MemoryBookmarkStore(5, adapter));
    
  }

That’s all that’s needed to enable the SOW recovery point adapter for an HAClient.

Reducing Bandwidth for Updates

The SOWRecoveryPointAdapter will save the recovery point to the State-of-the-World each time a new recovery point is created. For an application that is actively processing messages, this could produce a relatively high volume of updates to the State-of-the-World topic.

To reduce the bandwidth required to maintain the recovery point, the AMPS clients include a ConflatingRecoveryPointAdapter that wraps another adapter and passes along recovery points to that adapter at the specified interval. The interval consists of a number of updates and a time period: when the time period expires, or the number of updates is reached, the ConflatingRecoveryPointAdapter calls the wrapped adapter with the recovery point to be persisted.

For example, to persist the recovery point to the AMPS server every 100 updates or 250ms, the example above could be modified to set the bookmark store this way:

// The adapter object is a SOWRecoveryPointAdapter
   // constructed as above.

   ConflatingRecoveryPointAdapter conflater =
             new ConflatingRecoveryPointAdapter(adapter,
                   100, // Number of updates
                   250, // Update interval in milliseconds
                   50   // Check thresholds every 50ms
                   );  

    // Provide the conflating adapter to the MemoryBookmarkStore
    // instead of providing the SOWRecoveryPointAdapter directly.

    client_.setBookmarkStore(new MemoryBookmarkStore(5, conflater));

The ConflatingRecoveryPointAdapter starts its own background thread to manage timeouts and updates. When the adapter is closed, it writes all updates to the underlying adapter before closing that adapter.

Setting the threshold for updates is a matter of how much tolerance the application has for receiving messages that have already been processed in the event that the application fails before an updated recovery point is delivered to AMPS.

Other Considerations

To the AMPS server, the topic that stores the recovery points is the same as any other topic in the State of the World. The topic can be replicated to other instances of AMPS to improve reliability and allow the client that saves the recovery point to fail over. As mentioned earlier, the topic can be included in the same instance that has application data, or can be stored in a completely different instance of AMPS.

Like a MemoryBookmarkStore itself, an application that uses a SOWRecoveryPointAdapter must be able to handle some overlap in subscriptions (duplicate messages) in the event that the application fails (or the connection fails over) with messages that have not been discarded.