Aerospike Kafka source (outbound) connector

Streaming from Aerospike Database Enterprise Edition to Apache Kafka relies on the Aerospike Kafka source (outbound) connector. This connector subscribes to change notifications for new records, updates to records, and record deletions. When the connector receives such notifications, it transforms them from the proprietary Aerospike wire protocol into messages that are in a variety of common data-serialization formats. The connector sends the messages to Kafka topics.

Supported database events

The connector receives data from an Aerospike database by means of a change-notification system which is based on Aerospike’s Cross-Datacenter Replication (XDR). Each message contains a notification of one of these two types of events:

A record was inserted or updated
Each notification contains the full database record, including all, or a subset, of the record’s bins.

Operations on CDT List or Map types are not supported. For example, if a list element is added or a map entry is removed, actions which cause a partial bin update, Aerospike retransmits the full corresponding record bin.

A record was deleted
Each notification contains the corresponding record digest, but not the corresponding record bin.

Note

Note about both types of supported database events: There might be instances where records are re-ordered on the network or across connector instances, in which case a message for an older version of a record could be delivered after a message for a newer version of the same record.