The sink connector is configured with Kafka Connect properties. These properties can be supplied in a JSON request to a distributed worker or in a properties file for a standalone worker.
Required
The following properties are required to construct a valid AMPS sink connector.
Kafka Connect
| Property | Description |
|---|
connector.class | Must be com.crankuptheamps.kafka.AMPSKafkaSink. |
topics | Kafka topic or comma-separated list of topics to consume from. Use either topics or topics.regex. |
topics.regex | Java regular expression for Kafka topics to consume from. Use either topics.regex or topics. |
tasks.max | Maximum number of sink tasks. A value of 1 is recommended unless the pipeline is designed for parallel consumption. |
Connection to AMPS
| Property | Description |
|---|
uri | AMPS server URI or comma-separated list of URIs, such as tcp://localhost:9007/amps/json. |
clientName | Base name for the AMPS client. Task suffixes are added automatically when multiple tasks are used. |
Topic for AMPS
| Property | Description |
|---|
ampsTopic | AMPS topic to publish to when useTopicHeader is false, or the fallback topic when useTopicHeader is true. |
Optional
AMPS Client
The following properties are optional for functionality related to the AMPS client.
| Property | Description |
|---|
clientFactoryClass | Class used to construct the AMPS client. Defaults to com.crankuptheamps.kafka.AMPSBasicClientFunction. |
maxBatch | Maximum number of Kafka records the sink processes in a batch. Defaults to 1000. |
useTopicHeader | When true, publish each record to an AMPS topic matching the Kafka record topic. When false, publish to ampsTopic. Defaults to true. |
publishFlushTimeout | Timeout passed to AMPS publishFlush at the end of a batch. A value of -1 disables the flush. A value of 0 waits indefinitely for the batch to persist. Defaults to -1. |
logMsgOnWriteError | When true, write the failed message to the Kafka Connect log when an AMPS failed write is detected. Defaults to true. |
Publish Stores
The following properties configure AMPS publish stores for the sink client.
| Property | Description |
|---|
pubStoreType | Publish store type. Valid values are memory and file. |
pubStoreInitialCap | Publish store initial capacity in 2KB blocks. Defaults to 1000. |
pubStorePath | File path for a file-backed publish store. Required when pubStoreType is file. |
pruneTimeThreshold | Minimum time in milliseconds between file-backed store prune operations. Defaults to 300000. |