XDR stack overflow risk (AER-6898)

This page describes the XDR recovery drain defect (AER-6898) and provides safe upgrade instructions for affected Aerospike Database versions. This defect can cause stack overflow crashes or silent data corruption in Enterprise and Federal Edition clusters that use XDR. A standard rolling upgrade on a cluster running an affected version may trigger the crash. Follow the safe upgrade procedure below.

Tip

You do not need to follow the safe upgrade procedure if any of the following are true:

• You are running Aerospike Community Edition.
• You are not using XDR.
• You are currently running a version of Aerospike Database prior to the first affected version or equal to or later than the fixed in version specified in Affected versions below.

Background

When a node transitions from master to prole while XDR recovery is active, the XDR retry queue is not drained. Up to 1,000 in-flight shipment requests can remain outstanding and continue to count against the n_in_progress limit. If that node later becomes master again in regular (non-recovery) mode, the calculation 50 - n_in_progress underflows because the result is unsigned — sizing a stack-allocated variable-length array far beyond what the thread stack can accommodate.

The crash typically manifests as a segfault in process_trans_q, or as silent corruption of the primary index if the overflowed stack lands in mapped memory. Both outcomes can destabilize the cluster.

Affected versions

This defect applies to Enterprise and Federal Edition only, affecting any version that included AER-6795 or AER-6804:

Release line	First affected	Fixed in
6.1.x	6.1.0.47	No patch available
6.2.x	6.2.0.37	No patch available
6.3.x	6.3.0.35	6.3.0.36
6.4.x	6.4.0.30	6.4.0.34
7.0.x	7.0.0.22	No patch available
7.1.x	7.1.0.14	7.1.0.23
7.2.x	7.2.0.8	7.2.0.17
8.0.x	8.0.0.3	8.0.0.15
8.1.x	8.1.0.0	8.1.2.0

Note

The 6.1.x, 6.2.x, and 7.0.x release lines reached end of life before this defect was discovered. Clusters still running these versions should upgrade directly to a patched release in a supported line.

Detecting a vulnerable state

A node is in a vulnerable state if its XDR retry queue exceeds 50 entries and its partition role is prole. Run the following asinfo command for each configured datacenter and namespace:

asinfo -l -v "xdr-dc-state:dc=DC-NAME;namespace=NAMESPACE"

In the output, look for partitions where role is P (prole) and retry_q exceeds 50:

namespace:partition:role:state:safe_lst:first_trans_lut:trans_q:retry_q:windows:in_progress
NAMESPACE:0:P:R:0:0:0:1000:0:1000

If any partition shows retry_q > 50 on a prole node, the cluster is in the vulnerable state.

Caution

Because partition role assignments change during a rolling upgrade, all deployments running an affected version should use the safe upgrade procedure regardless of whether a vulnerable state is currently detected.

Safe upgrade procedure

1. Record the current UTC timestamp. You need to know the time prior to beginning step 2 so you can rewind XDR shipping after the upgrade to avoid missing records written during the upgrade window.

2. Disable XDR shipping across all nodes by removing all namespaces from all datacenter configurations. This drains the retry queue on every node and prevents the poisoned n_in_progress state from persisting across rolling restarts.

   For each namespace in each datacenter, use the set-config info command:

   asinfo -v "set-config:context=xdr;dc=DC-NAME;namespace=NAMESPACE;action=remove"

   Repeat for every namespace and every datacenter.

3. Wait until XDR activity is quiescent on all nodes before continuing. You can verify using:

   asinfo -l -v "xdr-dc-state:dc=DC-NAME;namespace=NAMESPACE"

4. Perform a rolling upgrade to the fixed release across all nodes. See Upgrade or repair an Aerospike server.

5. After all nodes are upgraded, recreate the XDR datacenter namespace configuration. Add each namespace back to its datacenter, rewinding XDR prior to the timestamp recorded in step 1. For example, if the upgrade took 30 minutes, rewind at least 1800 seconds:

   asinfo -v "set-config:context=xdr;dc=DC-NAME;namespace=NAMESPACE;action=add;rewind=1800"