Configure change notifications

The change-notification system informs external systems (such as source connectors, and the XDR Proxy) when records are inserted, modified, or deleted in an Aerospike database. This system uses Aerospike’s cross-datacenter replication (XDR) to publish changes as they occur.

Each message that the system sends contains either an updated/inserted record or a record deletion notification. Each update/insert/touch message contains the full database record including all, or a subset, of the record’s bins. Commands on CDT List or Map types, such as adding a list element or removing a map entry, are currently unsupported.

In case of partial bin updates such as these, the full record bin will be retransmitted. Record deletion notifications for the Aerospike contain the namespace and the record digest for all the server versions until Aerospike Database Enterprise Edition 5.0. However, for the Database EE 5.0 and later, it also ships the userKey and the set name if they are available. Keep in mind that the bins are never shipped with the delete notification.

Only records which have been successfully delivered to an external system are acknowledged by Aerospike.

There might be instances where records are re-ordered on the network or across external systems, such as instances of source connectors and instances of the XDR Proxy. In such cases, a message for an older version of a record might be delivered after a message for a newer version of the same record.

Caution

In rare cases, a change might be lost. For instance, changes may be lost if a node is removed from an Aerospike cluster after data migration between nodes is finished, but before all notifications are sent.

Prerequisites

• Ensure that you are running Aerospike Database Enterprise Edition 4.4 or later.
• Ensure that your digitally signed feature-key-file sets asdb-change-notification to true.

Procedure

Follow these steps on each node of your Aerospike cluster:

Open the /etc/aerospike/aerospike.conf configuration file for editing.

If there is no xdr section in the file, create one.

xdr {

    }

Create an XDR block in the xdr section, placing the connector property at the top of the block and setting its value to true.

Tip

The property ‘connector’ configures change notification for sending messages to both source (outbound) connectors and the XDR Proxy.

Here is an example. The name kafkaDC is arbitrary. You can name your blocks as you see fit. In this example, kafkaDC means only that the target “datacenter” is Kafka.

xdr {
     # Change notification XDR block that round-robins between two connector nodes
     dc aerospike-kafka-source {
             connector true
             node-address-port 10.0.0.1 8080
             node-address-port 10.0.0.2 8080
             namespace test {
             }
     }
}

For information about the properties in this example, see Configuration Parameter Reference.

Save and exit the file.

Restart the Aerospike cluster node.

What to do next

Install, configure, and start the external systems that you plan to use in your system for replicating or streaming data from Aerospike:

• Install the Aerospike Kafka Source Connector
• Overview for the Aerospike XDR Proxy.

Related information

• To learn how changes are logged, hot keys are optimized, and retransmits are handled, refer to the documentation for cross-datacenter replication:
  • Cross-Datacenter Replication (XDR)
  • Cross-Datacenter Replication (XDR) Architecture