Configure change notification for Aerospike Connect for Elasticsearch

The change notification system informs the Aerospike Connect for Elasticsearch connector when records are inserted, modified, or deleted in an Aerospike database. This system uses Aerospike’s cross-datacenter replication (XDR) to publish changes as they occur.

Each message that the system sends contains either an updated/inserted record or a record deletion notification. Each update/insert/touch message contains the full database record including all, or a subset, of the record’s bins.

Caution

Operations on CDT List or Map types, such as adding a list element or removing a map entry, are currently unsupported. In case of partial bin updates such as these, the full record bin is re-transmitted.

For Aerospike Database Enterprise Edition 5.0 and later, record deletion notifications contain:

• Namespace
• Record digest
• userKey (if available)
• set name (if available)

Tip

Bins are never shipped with the delete notification.

There might be instances where records are re-ordered on the network or across external systems, such as instances of source (outbound) connectors. In such cases, a message for an older version of a record might be delivered after a message for a newer version of the same record.

Caution

In rare cases, a change might be lost. For instance, changes may be lost if a node is removed from an Aerospike cluster after data migration between nodes is finished, but before all notifications are sent.

Prerequisites

• Aerospike Database Enterprise Edition, version 4.4 or later.
• asdb-change-notification set to true in your digitally-signed feature-key-file.

Procedure

To configure change notification for use with the Aerospike Elasticsearch Connector, follow these steps on each node of your Aerospike cluster:

Open the /etc/aerospike/aerospike.conf configuration file for editing.

If there is no xdr section in the file, create one.

xdr {

    }

Create an XDR block in the xdr section, placing the connector property at the top of the block and setting its value to true.

Here is an example. The name esDC is arbitrary. You can name your blocks as you see fit. In this example, esDC means only that the target “datacenter” is Elasticsearch.

xdr {
     # Change notification XDR block that round-robins between two connector nodes
     dc esDC {
             connector true
             node-address-port 10.0.0.1 8080
             node-address-port 10.0.0.2 8080
             namespace test {
             }
     }
}

For information about the properties in this example, see Configuration Parameter Reference.

Save and exit the file.

Restart the Aerospike cluster node.

Related information

• To learn how changes are logged, hot keys are optimized, and retransmits are handled, refer to the documentation for cross-datacenter replication:
  • Cross-Datacenter Replication (XDR)
  • Cross-Datacenter Replication (XDR) Architecture