Dynamic XDR configuration

XDR is Aerospike’s inter-cluster replication mechanism, a standard feature in Aerospike Database Enterprise Edition.

This page describes how to configure XDR dynamically from the command line or from a configuration file.

Terminology

Shipping refers to sending data from one datacenter to another.
A cluster from which records are shipped is a source cluster.
A cluster to which records are shipped is a destination cluster.

Version changes

XDR does not impose any version requirement across datacenters. You can ship between connected DCs as long as each is running Database 6.0.0 or later. Different versions of Aerospike Database, however, have the following variations with respect to XDR:

Starting with Database 8.1.1, recovery-threads configures the number of threads per datacenter (1-32) for use in parallel recovery. This improves recovery performance for large backlogs.
Starting with Database 7.2.0, ship-versions-policy controls how XDR ships versions of modified records when facing lag between the source cluster and a destination.
ship-versions-interval specifies a time window in seconds within which XDR is allowed to skip versions.
Starting with Database 5.7.0, the XDR subsystem always accepts dynamic configuration commands because it is always initialized even if XDR is not configured statically.
Prior to Database 5.7.0, the XDR subsystem must be configured statically with at least one datacenter.

Dynamic configuration from the command line

Use the asadm manage config command to configure datacenters dynamically without restarting any database nodes.

Define the dc and its corresponding node-address-port parameters.

asadm -e "enable; manage config xdr create dc DC1"
asadm -e "enable; manage config xdr dc DC1 add node 10.0.0.2:3000"

Add the namespace to the defined dc.

asadm -e "enable; manage config xdr dc DC1 add namespace someNameSpaceName"

To retain this configuration on restart, add it to your configuration file.

Rewind a shipment

You can rewind records that have been shipped to an XDR destination.

If you are adding a new namespace to XDR, you can use a single command to add and rewind it at the same time
To rewind an existing namespace, you must remove it and then add it back. Be aware that removing a namespace from a datacenter discards the existing XDR queue. If there was a high lag on that namespace/datacenter, some records may not be shipped if the rewind period is less than the lag.
You can configure rewind to ship only records that were last updated after a certain point in time.

Rewind and restart examples

In the following example, the rewind 600 clause rewinds shipment to 10 minutes before the current time for datacenter DC1 and namespace biggy:
```
asadm -e "enable; manage config xdr dc DC1 remove namespace biggy"
asadm -e "enable; manage config xdr dc DC1 add namespace biggy rewind 600"
```
- For small datasets, you can restart the shipment entirely from the beginning with rewind all.
- For large datasets, starting a complete rewind can increase server loads.

The following example completely restarts shipment for datacenter teenytiny and namespace itsybitsy:

asadm -e "enable; manage config xdr dc teenytiny remove namespace itsybitsy"
asadm -e "enable; manage config xdr dc teenytiny add namespace itsybitsy rewind all"

Recovery modes

When you add a datacenter dynamically, the XDR rewind feature scans the index and ships all the records for that namespace, partition by partition. This is XDR recovery mode.

XDR supports the following recovery modes:

Parallel partition-dedicated recovery (when max-recoveries-interleaved = 0):

Parallel partition-dedicated recovery mode spawns multiple recovery threads that operate in parallel. You configure these threads with the recovery-threads parameter. Each recovery thread picks up a partition from the recovery queue and works exclusively on that partition until its recovery is 100% complete. When the entire partition is finished, the thread picks up the next partition from the queue. Multiple threads work simultaneously on different partitions, with each thread fully dedicated to one partition at a time.

Single-threaded interleaved recovery (default when max-recoveries-interleaved >= 0):

Single-threaded interleaved recovery mode uses a single recovery thread to process multiple partitions in an interleaved fashion. Recovery jobs are inserted at position mri-1 in the recovery queue rather than at the end, ensuring fair distribution of recovery resources across partitions and preventing large partitions from monopolizing recovery time. If partitions fall back into recovery due to a high incoming throughput, you can configure max-recoveries-interleaved to limit the number of partitions and make sure a partition does not fall back into recovery after it completes a round.

Dynamic configuration constraints:

When switching from parallel to interleaved mode: To enable interleaving when max-recoveries-interleaved = 0, first set recovery-threads to 1, then change max-recoveries-interleaved to the desired value.
When switching from interleaved to parallel mode: To change recovery-threads when max-recoveries-interleaved >= 0, first set max-recoveries-interleaved to 0, then change recovery-threads to the desired value.

Dynamic configuration from a file

With the asinfo set-config command, you can configure datacenters dynamically without restarting any database nodes.

Configure a new datacenter

The following example shows the parameters you can enter on a single asinfo set-config command line, shown separately for clarity.

Define the dc and its corresponding node-address-port parameters.

asinfo -v "set-config:context=xdr;dc=DC1;action=create"
asinfo -v "set-config:context=xdr;dc=DC1;node-address-port=10.0.0.2:3000;action=add"

Add the namespace to the defined dc.

asinfo -v "set-config:context=xdr;dc=DC1;namespace=someNameSpaceName;action=add"

To retain this configuration on restart, add it to your configuration file.

Rewind or restart shipment with asinfo

With the asinfo set-config command, you can rewind a specific number of seconds’s shipment of records, or restart shipment completely.

When you are rewinding, the namespace to rewind must not be currently configured for that datacenter. That is, if you are adding a new namespace to XDR, you can use a single command to add and rewind it at the same time, but you must remove an existing namespace and add it back to rewind. Be aware that removing the namespace from that datacenter discards the existing XDR queue, so that if there was a high lag on that namespace/datacenter, some records may not be shipped if the rewind period is less than the lag.

In the following example, for datacenter DC1 and namespace biggy, the action=add;rewind=600 clause rewinds shipment to 10 minutes before the current time:

asinfo -v "set-config:context=xdr;dc=DC1;namespace=biggy;action=remove"
asinfo -v "set-config:context=xdr;dc=DC1;namespace=biggy;action=add;rewind=600"

For small datasets, you can restart the shipment entirely from the beginning with action=add;rewind=all. For large datasets, be careful with starting a complete rewind, which can increase server loads.

The following example restarts shipment for datacenter teenytiny and namespace itsybitsy:

asinfo -v "set-config:context=xdr;dc=teenytiny;namespace=itsybitsy;action=remove"
asinfo -v "set-config:context=xdr;dc=teenytiny;namespace=itsybitsy;action=add;rewind=all"

Ship records in a new datacenter’s namespace

asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=all"

Ship records in an existing namespace

Disassociate the namespace from the datacenter:

asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=remove"

Associate the namespace to the datacenter with the rewind option set to “all”:
Terminal window
```
asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=all"
```

(Optional) Specify the number of seconds from now to go back and ship records:

asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=NUMBER_SECONDS"

Ship specific sets to a new namespace

In some cases when associating (or re-associating) a namespace to a datacenter you can restrict shipping to only specific sets.

Disassociate the namespace from the datacenter.

asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=remove"

Dynamically configure ship-only-specified-sets for that namespace.

asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;ship-only-specified-sets=true"

Specify which sets to ship.

asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;ship-set=SETNAME"

Rewind when (re-)associating the namespace to the datacenter.

asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=all";