Dynamic XDR configuration

This page describes how to dynamically configure a remote datacenter (DC) destination for Aerospike's Cross Datacenter Replication (XDR) service.

Overview

When you configure XDR dynamically, you do not have to restart any database nodes. You can do it from the command line or from the configuration file. In both cases tools package 6.0.3 or later is required.

• Dynamic configuration from the command line (asadm)
• Dynamic configuration from a configuration file (asinfo)

note

• For instruction on how to statically configure XDR, see Static XDR configuration.
• For a theoretical and architectural overview, see XDR Architecture.

Version changes

• XDR does not impose any version requirement across datacenters.
• Starting with Database 7.2, ship-versions-policy controls how XDR ships versions of modified records when facing lag between the source cluster and a destination.
• ship-versions-interval specifies a time window in seconds within which XDR is allowed to skip versions.
• Starting with Database 5.7, the XDR subsystem always accepts dynamic configuration commands because it is always initialized even if XDR is not configured statically.
• Prior to Database 5.7, the XDR subsystem must be configured statically with at least one datacenter.

Terminology

• Shipping refers to sending data from one datacenter to another.
• A cluster from which records are shipped is a source cluster.
• A cluster to which records are shipped is a destination cluster.

Dynamic configuration from the command line

Use the Aerospike Admin (asadm) manage config command, to configure datacenters dynamically without restarting any database node.

1. Define the dc and its corresponding node-address-port parameters.

   asadm -e "enable; manage config xdr create dc DC1"
   asadm -e "enable; manage config xdr dc DC1 add node 10.0.0.2:3000"

2. Add the namespace to the defined dc.

   asadm -e "enable; manage config xdr dc DC1 add namespace someNameSpaceName"

3. To retain this configuration on restart, add it to your configuration file.

Rewind a shipment

You can rewind records to an XDR destination when you dynamically associate .

When you add a datacenter dynamically, the XDR rewind feature scans the index and ships all the records for that namespace, partition by partition. This is the XDR 'recovery mode'. If partitions fall back into recoveries due to a high incoming throughput, you can configure max-recoveries-interleaved to limit the number of partitions and make sure a partition does not fall back into recoveries after it completes a round.

• If you are adding a new namespace to XDR, you can use a single command to add and rewind it at the same time
• To rewind an existing namespace, you must remove it, then add it back to rewind. Be aware that removing a namespace from a datacenter discards the existing XDR queue, so that if there was a high lag on that namespace/datacenter, some records may not be shipped if the rewind period is less than the lag.
• You can configure rewind to ship only records that were last updated after a certain point in time.

Examples

1. In the following example, the rewind 600 clause rewinds shipment to 10 minutes before the current time for datacenter DC1 and namespace biggy:

   asadm -e "enable; manage config xdr dc DC1 remove namespace biggy"
   asadm -e "enable; manage config xdr dc DC1 add namespace biggy rewind 600"

   • For small datasets, you can restart the shipment entirely from the beginning with rewind all.

   • For large datasets, starting a complete rewind can increase server loads.

2. The following example completely restarts shipment for datacenter teenytiny and namespace itsybitsy:

   asadm -e "enable; manage config xdr dc teenytiny remove namespace itsybitsy"
   asadm -e "enable; manage config xdr dc teenytiny add namespace itsybitsy rewind all"

Dynamic configuration from configuration file

With the asinfo set-config command, you can configure datacenters dynamically without restarting any database node.

Configure a new datacenter

The following example shows the parameters you can enter on a single asinfo set-config command line, shown separately for clarity.

1. Define the dc and its corresponding node-address-port parameters.

asinfo -v "set-config:context=xdr;dc=DC1;action=create"
asinfo -v "set-config:context=xdr;dc=DC1;node-address-port=10.0.0.2:3000;action=add"

2. Add the namespace to the defined dc.

asinfo -v "set-config:context=xdr;dc=DC1;namespace=someNameSpaceName;action=add"

3. Add this configuration to your configuration file.

Rewind or restart shipment with asinfo

With the asinfo set-config command, you can rewind a specific number of seconds's shipment of records, or restart shipment completely.

When you are rewinding, the namespace to rewind must not be currently configured for that datacenter. That is, if you are adding a new namespace to XDR, you can use a single command to add and rewind it at the same time, but you must remove an existing namespace and add it back to rewind. Be aware that removing the namespace from that datacenter discards the existing XDR queue, so that if there was a high lag on that namespace/datacenter, some records may not be shipped if the rewind period is less than the lag.

In the following example, for datacenter DC1 and namespace biggy, the action=add;rewind=600 clause rewinds shipment to 10 minutes before the current time:

asinfo -v "set-config:context=xdr;dc=DC1;namespace=biggy;action=remove"
asinfo -v "set-config:context=xdr;dc=DC1;namespace=biggy;action=add;rewind=600"

For small datasets, you can restart the shipment entirely from the beginning with action=add;rewind=all. For large datasets, be careful with starting a complete rewind, which can increase server loads.

The following example restarts shipment for datacenter teenytiny and namespace itsybitsy:

asinfo -v "set-config:context=xdr;dc=teenytiny;namespace=itsybitsy;action=remove"
asinfo -v "set-config:context=xdr;dc=teenytiny;namespace=itsybitsy;action=add;rewind=all"

Ship records in a new datacenter's namespace

asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=all"

Ship records in an existing datacenter's namespace

1. Disassociate the namespace from the datacenter:

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=all"

2. Associate the namespace to the datacenter with the rewind option set to "all":

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=all"

3. (Optional) Specify the number of seconds from now to go back and ship records:

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=NUMBER_SECONDS"

Ship specific sets to a new namespace

In some cases when associating (or re-associating) a namespace to a datacenter you can restrict shipping to only specific sets.

1. Disassociate the namespace from the datacenter.

   asinfo -v "set-config:context=xdr;dc=<DC_NAME>;namespace=NAMESPACE;action=add;rewind=all"

2. Dynamically configure ship-only-specified-sets for that namespace.

   context=xdr;dc=DC_NAME;namespace=NAMESPACE;ship-only-specified-sets=true"

3. Specify which sets to ship.

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;ship-set=SETNAME"

4. Rewind when (re-)associating the namespace to the datacenter.

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE;action=add;rewind=all";