Streaming from Aerospike to Pulsar
Streaming from Aerospike Database Enterprise Edition to Apache Pulsar relies on the Pulsar connector. This connector subscribes to change notifications for new records, updates to records, and record deletions. When the connector receives such notifications, it transforms them from the proprietary Aerospike wire protocol into messages that are in a variety of common data-serialization formats. The connector sends the messages to Pulsar topics.
Message Routing
Pulsar connectors can route messages to one or more Pulsar topics. The routing can be static or dynamic, depending on some property of the message itself.
The connectors support the following routing modes:
- Static: Always routes to the static default topic. This is the default mode.
- Namespace: Uses the namespace of the Aerospike record as the Pulsar topic.
- Set: Uses the set of the Aerospike record as the Pulsar topic. For records without a set, you can specify a fallback route that uses the "static" routing mode.
- Bin: Sets the route based on the value of a bin in the record. Only string, blob, and integer bin types are supported.
Supported database events
The connector receives data from an Aerospike database by means of a change-notification system which is based on Aerospike's Cross-Datacenter Replication (XDR). Each message contains a notification of one of these two types of events:
A record was inserted or updated
Each notification contains the full database record, including all, or a subset, of the record's bins.
Operations on CDT List or Map types are not supported. For example, if a list element is added or a map entry is removed, actions which cause a partial bin update, Aerospike retransmits the full corresponding record bin.
A record was deleted
Each notification contains the corresponding record digest, but not the corresponding record bin.
Note about both types of supported database events: There might be instances where records are re-ordered on the network or across connector instances, in which case a message for an older version of a record could be delivered after a message for a newer version of the same record.