Dynamic XDR configuration

XDR is Aerospike’s inter-cluster replication mechanism, a standard feature in Aerospike Database Enterprise Edition.

This page describes how to configure XDR dynamically from the command line or from a configuration file.

Terminology

• Shipping refers to sending data from one datacenter to another.
• A cluster from which records are shipped is a source cluster.
• A cluster to which records are shipped is a destination cluster.

Note

• For instruction on how to statically configure XDR, see Static XDR configuration.
• For a theoretical and architectural overview, see XDR Architecture.

Version changes

XDR does not impose any version requirement across datacenters. You can ship between connected DCs as long as each is running Database 6.0.0 or later. Different versions of Aerospike Database, however, have the following variations with respect to XDR:

• Starting with Database 8.1.1, recovery-threads configures the number of threads per datacenter (1-32) for use in parallel recovery. This improves recovery performance for large backlogs.
• Starting with Database 7.2.0, ship-versions-policy controls how XDR ships versions of modified records when facing lag between the source cluster and a destination.
• ship-versions-interval specifies a time window in seconds within which XDR is allowed to skip versions.
• Starting with Database 5.7.0, the XDR subsystem always accepts dynamic configuration commands because it is always initialized even if XDR is not configured statically.
• Prior to Database 5.7.0, the XDR subsystem must be configured statically with at least one datacenter.

Dynamic configuration from the command line

Use the asadm manage config command to configure datacenters dynamically without restarting any database nodes.

1. Define the dc and its corresponding node-address-port parameters.

   asadm -e "enable; manage config xdr create dc DC1"
   asadm -e "enable; manage config xdr dc DC1 add node 10.0.0.2:3000"

   When using TLS, append the destination cluster’s TLS name after the port using the format host:tls_port:tls_name:

   asadm -e "enable; manage config xdr dc DC1 add node 10.0.0.2:4333:destination_tls_name"

   To remove a TLS node, specify only the host and port (omit the TLS name):

   asadm -e "enable; manage config xdr dc DC1 remove node 10.0.0.2:4333"

2. Add the namespace to the defined dc.

   asadm -e "enable; manage config xdr dc DC1 add namespace someNameSpaceName"

3. To retain this configuration on restart, add it to your configuration file.

Rewind a shipment

You can rewind records that have been shipped to an XDR destination.

• If you are adding a new namespace to XDR, you can use a single command to add and rewind it at the same time.
• To rewind an existing namespace, you must remove it and then add it back. Be aware that removing a namespace from a datacenter discards the existing XDR queue. If there was a high lag on that namespace/datacenter, some records may not be shipped if the rewind period is less than the lag.
• You can configure rewind to ship only records that were last updated after a certain point in time.
• If you need to ship only specific sets, you must configure ship-only-specified-sets and ship-set after the remove and before the re-add. See Ship specific sets to a new namespace for the asinfo sequence. For large namespaces, asbackup/asrestore or a touch UDF to trigger normal XDR shipping may be more efficient than a set-specific rewind. See the How to rewind XDR for a namespace KB article for details.
• When a namespace is added (with or without rewind), XDR sets a fresh Last Ship Time (LST) for that namespace/DC pair. The LST is communicated across nodes and persisted to SMD. XDR will not ship records older than the LST.

Tip

By default, a rewind will not ship records that were written into this cluster by XDR. If you want the rewind to ship all records, regardless of whether they were written by XDR or a local client, you will need to dynamically set forward true in the destination’s dc configuration, for the duration of the rewind. When the rewind is complete, remember to set it back to false.

Rewind and restart examples

1. In the following example, the rewind 600 clause rewinds shipment to 10 minutes before the current time for datacenter DC1 and namespace biggy:

   asadm -e "enable; manage config xdr dc DC1 remove namespace biggy"
   asadm -e "enable; manage config xdr dc DC1 add namespace biggy rewind 600"

   • For small datasets, you can restart the shipment entirely from the beginning with rewind all.

   • For large datasets, starting a complete rewind can increase server loads.

2. The following example completely restarts shipment for datacenter teenytiny and namespace itsybitsy:

   asadm -e "enable; manage config xdr dc teenytiny remove namespace itsybitsy"
   asadm -e "enable; manage config xdr dc teenytiny add namespace itsybitsy rewind all"

Recovery modes

When you add a datacenter dynamically, the XDR rewind feature scans the index and ships all the records for that namespace, partition by partition. This is XDR recovery mode.

XDR supports the following recovery modes:

Parallel partition-dedicated recovery (when max-recoveries-interleaved = 0):

Parallel partition-dedicated recovery mode spawns multiple recovery threads that operate in parallel. You configure these threads with the recovery-threads parameter. Each recovery thread picks up a partition from the recovery queue and works exclusively on that partition until its recovery is 100% complete. When the entire partition is finished, the thread picks up the next partition from the queue. Multiple threads work simultaneously on different partitions, with each thread fully dedicated to one partition at a time.

Single-threaded interleaved recovery (default when max-recoveries-interleaved >= 0):

Single-threaded interleaved recovery mode uses a single recovery thread to process multiple partitions in an interleaved fashion. Recovery jobs are inserted at position mri-1 in the recovery queue rather than at the end, ensuring fair distribution of recovery resources across partitions and preventing large partitions from monopolizing recovery time. If partitions fall back into recovery due to a high incoming throughput, you can configure max-recoveries-interleaved to limit the number of partitions and make sure a partition does not fall back into recovery after it completes a round.

Dynamic configuration constraints:

• When switching from parallel to interleaved mode: To enable interleaving when max-recoveries-interleaved = 0, first set recovery-threads to 1, then change max-recoveries-interleaved to the desired value.
• When switching from interleaved to parallel mode: To change recovery-threads when max-recoveries-interleaved >= 0, first set max-recoveries-interleaved to 0, then change recovery-threads to the desired value.

Dynamic configuration from a file

With the asinfo set-config command, you can configure datacenters dynamically without restarting any database nodes.

Configure a new datacenter

The following example shows the parameters you can enter on a single asinfo set-config command line, shown separately for clarity.

1. Define the dc and its corresponding node-address-port parameters.

   asinfo -v "set-config:context=xdr;dc=DC1;action=create"
   asinfo -v "set-config:context=xdr;dc=DC1;node-address-port=10.0.0.2:3000;action=add"

2. Add the namespace to the defined dc.

   asinfo -v "set-config:context=xdr;dc=DC1;namespace=someNameSpaceName;action=add"

3. To retain this configuration on restart, add it to your configuration file.

Configure a new datacenter with TLS

When the destination cluster requires TLS and authentication, you must set the TLS and auth parameters before adding node addresses and namespaces. For background on TLS name concepts and static configuration, see Securing XDR with access control, LDAP, and TLS.

1. Create the datacenter.

   asinfo -v "set-config:context=xdr;dc=DC1;action=create"

2. Set the local cluster’s TLS name for this datacenter.

   asinfo -v "set-config:context=xdr;dc=DC1;tls-name=source_tls_name"

3. Set the authentication mode (external or internal).

   asinfo -v "set-config:context=xdr;dc=DC1;auth-mode=external"

4. If using access control, set the password file and user.

   asinfo -v "set-config:context=xdr;dc=DC1;auth-password-file=/path/to/credentials.txt"
   asinfo -v "set-config:context=xdr;dc=DC1;auth-user=xdr_user"

5. Add the destination node addresses. Use the format host:tls_port:tls_name to append the destination cluster’s TLS name after the port.

   asinfo -v "set-config:context=xdr;dc=DC1;node-address-port=10.0.0.2:4333:destination_tls_name;action=add"

   To remove a node, specify only the host and port (omit the TLS name):

   asinfo -v "set-config:context=xdr;dc=DC1;node-address-port=10.0.0.2:4333;action=remove"

6. Add the namespace to start shipping. Include rewind to ship older data.

   asinfo -v "set-config:context=xdr;dc=DC1;namespace=someNameSpaceName;action=add;rewind=900"

7. To retain this configuration on restart, add it to your configuration file.

Rewind or restart shipment with asinfo

With the asinfo set-config command, you can rewind a specific number of seconds’s shipment of records, or restart shipment completely.

When you are rewinding, the namespace to rewind must not be currently configured for that datacenter. That is, if you are adding a new namespace to XDR, you can use a single command to add and rewind it at the same time, but you must remove an existing namespace and add it back to rewind. Be aware that removing the namespace from that datacenter discards the existing XDR queue, so that if there was a high lag on that namespace/datacenter, some records may not be shipped if the rewind period is less than the lag.

In the following example, for datacenter DC1 and namespace biggy, the action=add;rewind=600 clause rewinds shipment to 10 minutes before the current time:

asinfo -v "set-config:context=xdr;dc=DC1;namespace=biggy;action=remove"
asinfo -v "set-config:context=xdr;dc=DC1;namespace=biggy;action=add;rewind=600"

For small datasets, you can restart the shipment entirely from the beginning with action=add;rewind=all. For large datasets, be careful with starting a complete rewind, which can increase server loads.

The following example restarts shipment for datacenter teenytiny and namespace itsybitsy:

asinfo -v "set-config:context=xdr;dc=teenytiny;namespace=itsybitsy;action=remove"
asinfo -v "set-config:context=xdr;dc=teenytiny;namespace=itsybitsy;action=add;rewind=all"

Ship records in a new datacenter’s namespace

asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=all"

Ship records in an existing namespace

1. Disassociate the namespace from the datacenter:

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=remove"

2. If you need to ship only specific sets, configure ship-only-specified-sets and ship-set before re-associating the namespace. These parameters must be set while the namespace is disassociated; they cannot be changed after the action=add step. See Ship specific sets to a new namespace for the full sequence.

3. Associate the namespace to the datacenter with the rewind option set to “all”:

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=all"

4. (Optional) Specify the number of seconds from now to go back and ship records:

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=NUMBER_SECONDS"

Ship specific sets to a new namespace

In some cases when associating (or re-associating) a namespace to a datacenter you can restrict shipping to only specific sets.

1. Disassociate the namespace from the datacenter.

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=remove"

2. Dynamically configure ship-only-specified-sets for that namespace.

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;ship-only-specified-sets=true"

3. Specify which sets to ship.

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;ship-set=SETNAME"

4. Rewind when (re-)associating the namespace to the datacenter.

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=all";