Dynamic XDR configuration

This page describes how to configure XDR dynamically using asadm or asinfo, and how to persist that configuration across node restarts.

XDR is Aerospike’s inter-cluster replication mechanism, a standard feature in Aerospike Database Enterprise Edition.

Terminology

• Shipping refers to sending data from one datacenter to another.
• A cluster from which records are shipped is a source cluster.
• A cluster to which records are shipped is a destination cluster.

Note

• To learn about how to statically configure XDR, see Static XDR configuration.
• For a theoretical and architectural overview, see Cross datacenter replication (XDR).

Version changes

You can ship between connected DCs as long as each is running Database 6.0.0 or later. Different versions of Aerospike Database, however, have the following variations with respect to XDR:

• Starting with Database 8.1.1, recovery-threads configures the number of threads per datacenter (1-32) for use in parallel recovery. This improves recovery performance for large backlogs.
• Starting with Database 7.2.0:
  • ship-versions-policy controls how XDR ships versions of modified records when facing lag between the source cluster and a destination.
  • ship-versions-interval specifies a time window in seconds within which XDR is allowed to skip versions.

Dynamic configuration

Configure XDR on a running cluster without restarting database nodes using asadm or asinfo.

• asadm manage config applies changes across all nodes in the cluster. Aerospike recommends this for most XDR configuration tasks.
• asinfo set-config applies changes only to the node you connect to. The same command-string format is used when configuring XDR programmatically through the SDK.

For additional command syntax, see XDR syntax in the runtime configuration guide.

Note

Tools package 6.0.x or later is required to use asadm manage config commands.

Configure a new datacenter

1. Create the datacenter and add destination nodes.

   asadm

   asadm -e "enable; manage config xdr create dc DC1"
   asadm -e "enable; manage config xdr dc DC1 add node 10.0.0.2:3000"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC1;action=create"
   asinfo -v "set-config:context=xdr;dc=DC1;node-address-port=10.0.0.2:3000;action=add"

   When using TLS, append the destination cluster’s TLS name after the port using the format host:tls_port:tls_name:

   asadm

   asadm -e "enable; manage config xdr dc DC1 add node 10.0.0.2:4333:destination_tls_name"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC1;node-address-port=10.0.0.2:4333:destination_tls_name;action=add"

   To remove a TLS node, specify only the host and port (omit the TLS name):

   asadm

   asadm -e "enable; manage config xdr dc DC1 remove node 10.0.0.2:4333"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC1;node-address-port=10.0.0.2:4333;action=remove"

2. Add the namespace to the defined dc. This starts shipping new writes from this point forward. XDR does not ship existing records unless you add the namespace with rewind. See Rewind a shipment.

   asadm

   asadm -e "enable; manage config xdr dc DC1 add namespace NAMESPACE_NAME"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC1;namespace=NAMESPACE_NAME;action=add"

3. To retain this configuration on restart, add it to your configuration file. See Persist dynamic XDR configuration.

Configure a new datacenter with TLS

When the destination cluster requires TLS and authentication, set the TLS and authentication parameters before adding node addresses and namespaces. For background on TLS name concepts and static configuration, see XDR security.

1. Create the datacenter.

   asadm

   asadm -e "enable; manage config xdr create dc DC1"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC1;action=create"

2. Set the local cluster’s TLS name for this datacenter.

   asadm

   asadm -e "enable; manage config xdr dc DC1 param tls-name to source_tls_name"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC1;tls-name=source_tls_name"

3. Set the authentication mode.

   Use external for LDAP authentication over TLS, internal for Aerospike access control, or pki for PKI authentication. With pki, do not set auth-user or auth-password-file. See XDR security for authentication mode details and credential setup.

   asadm

   asadm -e "enable; manage config xdr dc DC1 param auth-mode to external"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC1;auth-mode=external"

4. If using access control or LDAP, set the password file and user.

   asadm

   asadm -e "enable; manage config xdr dc DC1 param auth-password-file to /path/to/credentials.txt"
   asadm -e "enable; manage config xdr dc DC1 param auth-user to xdr_user"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC1;auth-password-file=/path/to/credentials.txt"
   asinfo -v "set-config:context=xdr;dc=DC1;auth-user=xdr_user"

5. Add the destination node addresses. Use the format host:tls_port:tls_name to append the destination cluster’s TLS name after the port.

   asadm

   asadm -e "enable; manage config xdr dc DC1 add node 10.0.0.2:4333:destination_tls_name"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC1;node-address-port=10.0.0.2:4333:destination_tls_name;action=add"

   To remove a node, specify only the host and port (omit the TLS name):

   asadm

   asadm -e "enable; manage config xdr dc DC1 remove node 10.0.0.2:4333"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC1;node-address-port=10.0.0.2:4333;action=remove"

6. Add the namespace to start shipping new writes from this point forward. To ship older records, see Rewind a shipment.

   asadm

   asadm -e "enable; manage config xdr dc DC1 add namespace NAMESPACE_NAME"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC1;namespace=NAMESPACE_NAME;action=add"

7. To retain this configuration on restart, add it to your configuration file. See Persist dynamic XDR configuration.

Persist dynamic XDR configuration

To retain dynamic XDR settings across node restarts, add the equivalent xdr section to your Aerospike configuration file after you finish dynamic configuration. See Rewind a shipment for how to control backfill when associating a namespace.

Caution

On a cluster with no existing XDR configuration, add namespace entries to the configuration file only after you dynamically create the datacenter and add the namespace. If the configuration file contains a namespace association before XDR is dynamically enabled, the next node startup scans the entire primary index and ships all records, as if you had specified rewind all. See Rewind a shipment for how to use rewind intentionally.

Rewind a shipment

Use rewind when you associate a namespace with a datacenter to ship records last updated within a specified time window. A rewind ships every matching record in that window, whether or not XDR previously shipped it to the destination. You can use rewind when first associating a namespace or when re-associating a namespace that is already configured for the datacenter.

Use these rules when planning a rewind:

• For a new namespace association, add the namespace with rewind in the same command.
• For an existing namespace association, remove the namespace from the datacenter, then add it back with rewind. Removing the namespace discards the existing XDR queue, so records can be missed if the rewind period is shorter than the current XDR lag.
• To ship records updated within a specific time window, set rewind to the number of seconds before the current time.
• To rewind only specific sets, configure ship-only-specified-sets and ship-set after removing the namespace and before adding it back. See Ship specific sets to a new namespace for the full sequence.
• For large namespaces, using asbackup and asrestore, or a touch UDF that triggers normal XDR shipping, may be more efficient than a set-specific rewind. See How to rewind XDR for a namespace for details.
• When a namespace is added with or without rewind, XDR sets a fresh Last Ship Time (LST) for that namespace and DC pair. The LST is communicated across nodes and persisted to SMD. XDR does not ship records older than the LST.

Tip

By default, a rewind will not ship records that were written into this cluster by XDR. If you want the rewind to ship all records, regardless of whether they were written by XDR or a local client, you will need to dynamically set forward to true in the destination’s dc configuration, for the duration of the rewind. When the rewind is complete, remember to set it back to false.

Rewind and restart examples

1. In the following example, the rewind 600 clause rewinds shipment to 10 minutes before the current time for datacenter DC1 and namespace namespace1:

   asadm

   asadm -e "enable; manage config xdr dc DC1 remove namespace namespace1"
   asadm -e "enable; manage config xdr dc DC1 add namespace namespace1 rewind 600"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC1;namespace=namespace1;action=remove"
   asinfo -v "set-config:context=xdr;dc=DC1;namespace=namespace1;action=add;rewind=600"

   • For small datasets, you can restart the shipment entirely from the beginning with rewind all.

   • For large datasets, starting a complete rewind can increase server loads.

2. The following example completely restarts shipment for datacenter DC2 and namespace namespace2:

   asadm

   asadm -e "enable; manage config xdr dc DC2 remove namespace namespace2"
   asadm -e "enable; manage config xdr dc DC2 add namespace namespace2 rewind all"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC2;namespace=namespace2;action=remove"
   asinfo -v "set-config:context=xdr;dc=DC2;namespace=namespace2;action=add;rewind=all"

Ship specific sets to a new namespace

In some cases, when associating a namespace to a datacenter, you can restrict shipping to only specific sets.

1. Disassociate the namespace from the datacenter.

   asadm

   asadm -e "enable; manage config xdr dc DC_NAME remove namespace NAMESPACE_NAME"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE_NAME;action=remove"

2. Dynamically configure ship-only-specified-sets for that namespace. These parameters must be set while the namespace is disassociated; they cannot be changed after the action=add step.

   asadm

   asadm -e "enable; manage config xdr dc DC_NAME namespace NAMESPACE_NAME param ship-only-specified-sets to true"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE_NAME;ship-only-specified-sets=true"

3. Specify which sets to ship.

   asadm

   asadm -e "enable; manage config xdr dc DC_NAME namespace NAMESPACE_NAME param ship-set to SETNAME"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE_NAME;ship-set=SETNAME"

4. Rewind when associating the namespace to the datacenter.

   asadm

   asadm -e "enable; manage config xdr dc DC_NAME add namespace NAMESPACE_NAME rewind all"

   asinfo

   asinfo -v "set-config:context=xdr;dc=DC_NAME;namespace=NAMESPACE_NAME;action=add;rewind=all"

Recovery modes

When you rewind a namespace, XDR scans the index and ships matching records for that namespace, partition by partition. This is XDR recovery mode. For when recovery mode activates and how it fits the shipment lifecycle, see XDR recovery mode. XDR supports the following recovery modes.

Single-threaded interleaved recovery

Single-threaded interleaved recovery is the default when recovery-threads is 1.

In this mode, one recovery thread processes multiple partitions in an interleaved fashion. The mode is active when max-recoveries-interleaved is greater than or equal to 0.

Use max-recoveries-interleaved to control how many partitions recover concurrently. Set it greater than 0 to limit concurrent recovery.

Recovery jobs are inserted at position mri-1 in the recovery queue rather than at the end. That ordering spreads recovery work across partitions so large partitions do not monopolize recovery time.

If high incoming throughput pushes partitions back into recovery after they finish a round, raise max-recoveries-interleaved to reduce how many partitions recover at once.

Parallel partition-dedicated recovery

Parallel partition-dedicated recovery is active when recovery-threads is greater than 1 and max-recoveries-interleaved is 0 (Database 8.1.1 and later).

When the entire partition is finished, the thread picks up the next partition from the queue. Multiple threads work simultaneously on different partitions, with each thread fully dedicated to one partition at a time.

Dynamic configuration constraints

• When switching from parallel to interleaved mode: To enable interleaving when max-recoveries-interleaved = 0, first set recovery-threads to 1, then change max-recoveries-interleaved to the desired value.
• When switching from interleaved to parallel mode: To change recovery-threads when max-recoveries-interleaved >= 0, first set max-recoveries-interleaved to 0, then change recovery-threads to the desired value.