Configure streaming from Aerospike to Kafka

Configuring streaming from Aerospike to Kafka involves setting up your Aerospike database to send change notifications and modifying the configuration file /etc/aerospike-kafka-outbound/aerospike-kafka-outbound.yml.

Set Up Aerospike Database

In your Aerospike database, you must configure Cross-Datacenter Replication (XDR) and enable change notification. You must also configure your Kafka outbound cluster to be a “connector” XDR datacenter and the namespace must point to this as xdr remote datacenter.

See the change notification configuration parameters and example config.

Modify the Config File

Configure streaming from Aerospike to Kafka by modifying the /etc/aerospike-kafka-outbound/aerospike-kafka-outbound.yml YAML configuration file.

The configuration file has the following sections:

• service Configures the source connector’s listening ports, TLS, and network interface.
• producer-props Configures connection properties for the target Kafka message broker.
• bin-transforms Specifies the bin transformations to apply on the Aerospike record.
• format Specifies the message format to use for the outbound messages sent to the message broker.
• batching Specifies how to collect a group of Change Notification Records into a single batch of outbound message sent to the message broker.
• routing - configures how incoming record updates/deletes from Aerospike are routed to the message broker.
• namespaces Configures bin transforms, format, routing at namespace and set level.
• logging Configures the destination and level for the source connector’s logs.

Caution

If the route or format is not specified for an Aerospike record, then the record is skipped and logged at the warn level.

Cascading Configuration Values

You can configure the bin-transforms, format, and routing sections to include default values, values for each particular namespace, and values for each particular set within a namespace. For each Aerospike record, the appropriate values are applied.

Here is an annotated example:

routing:
  mode: static
  destination: default <1>
‎‎
namespaces:
  users:
    routing:
      mode: static
      destination: users <2>
    format:
      mode: flat-json
      metadata-key: metadata
    sets:
      premium:
        routing:
          mode: static
          destination: premium <3>

1. Specifies that records in all namespaces are to be shipped to the default topic.
2. Specifies that records in the users namespace are to be shipped to the users topic.
3. Specifies that records in the premium set in the users namespace are to be shipped to the premium topic.

Example

A sample configuration is shown below:

# port that the connector runs on
service:
  port: 8080

# kafka producer props https://kafka.apache.org/23/javadoc/org/apache/kafka/clients/producer/ProducerConfig.html
producer-props:
  bootstrap.servers:
    - 192.168.5.20:9092
    - 192.168.5.30:9092

# log location if not stdout
logging:
  file: /var/log/aerospike-kafka-outbound/aerospike-kafka-outbound.log

# one of json, flat-json, message-pack, avro, or kafka
format:
  mode: json

# a list of transformations and mappings.
bin-transforms:
  map:
   yellow: red
   transforms: //will be done in order
    - uppercase

routing:
  mode: static
  destination: default

namespaces:
  users:
    routing:
      mode: static
      destination: users
    format:
      mode: flat-json
      metadata-key: metadata
    sets:
      premium:
        routing:
          mode: static
          destination: premium
        bin-transforms:
          map:
            gold: platinum