Update replication factor dynamically on AKO

This page explains how to update replication-factor for an AP namespace by editing the AerospikeCluster custom resource (CR). The replication-factor setting controls how many copies of each record Aerospike keeps in a namespace.

How you update replication-factor depends on the namespace’s consistency mode. In AP mode, every group of nodes that can still communicate can continue to serve reads and writes during a network partition. In strong consistency (SC) mode, only the group that holds a quorum, meaning a majority of the configured roster nodes, can continue to accept writes. Because of that difference, AKO can apply replication-factor dynamically only for AP namespaces, without a rolling restart, when dynamic configuration updates are enabled.

Before you update

• You can only update the replication factor of AP namespaces. AKO cannot update replication-factor dynamically for strong consistency (SC) namespaces.
• When you update replication-factor, change only that field. Do not combine it with other Aerospike configuration changes, and do not change more than one namespace in the same CR update.
• You must wait for the update to finish before you change the replication factor in another namespace.
• Before you update, verify that spec.enableDynamicConfigUpdate is true in the AerospikeCluster CR. Run kubectl describe aerospikecluster CLUSTER_NAME -n aerospike and check the spec section for this parameter.

Note

For AP namespaces, Aerospike Database 6.0 and later treat replication-factor as a dynamic namespace parameter. In AKO 4.4.0 and later, AKO can apply that change from the CR when dynamic configuration updates are enabled.

Update the replication factor

1. Update only the replication-factor value for the target namespace in the AerospikeCluster CR.

   spec:
     enableDynamicConfigUpdate: true
     aerospikeConfig:
       namespaces:
         - name: test
           replication-factor: 3

2. Apply the updated CR.

   kubectl apply -f aerospike-cluster.yaml

3. Confirm that AKO applied the new value.

   kubectl -n aerospike describe aerospikecluster AEROSPIKE_CLUSTER_NAME

   Check the target namespace under Status > Aerospike Config > Namespaces and confirm the new replication-factor value.

After the update, AKO applies the new replication factor without a rolling restart when the target namespace is AP and dynamic configuration updates are enabled. The updated value persists after future pod restarts.

Troubleshooting

The following examples use asadm to connect to an Aerospike node. See Connect with asadm first.

Inconsistent replication-factor during node restarts

If a node restarts while a replication-factor (RF) update is in progress, you can see mixed RF values across nodes. The restarted node may report no cluster quorum in info output.

To see the symptoms, run info on an affected pod:

asadm -U USERNAME -P PASSWORD -e "info"

An example output fragment follows (values vary by cluster):

Node|Node|               IP|    Build|Migrations|~~~~~~~~~~~~~~Cluster~~~~~~~~~~~~~~~|Client|  Uptime
    |  ID|                 |         |          |Size|        Key|Integrity|Principal| Conns|
... | 1A0|10.138.0.78:31042|E-8.1.0.0|   0.000  |   0|          0|True     |        0|     7|00:00:24

The restarted node reports Size: 0, Key: 0, and Principal: 0 under the cluster columns, which indicates no quorum. Other nodes may still show the previous or target RF until the update finishes.

Resolution steps

1. Confirm that the desired RF in the AerospikeCluster CR (spec.aerospikeConfig.namespaces) matches what you intend, then allow AKO to converge and propagate RF to all nodes.

2. Verify the current RF and quorum on other pods.

   asadm -U USERNAME -P PASSWORD -e "show config like replication-factor"

   Every node must eventually report the same replication-factor for the namespace.

3. If AKO cannot propagate the desired RF to all nodes, align RF manually on individual nodes.

   asadm -U USERNAME -P PASSWORD --enable -e 'asinfo -v "set-config:context=namespace;namespace=NAMESPACE_NAME;replication-factor=VALUE"'

4. If the reconciler appears stuck during reclustering, trigger a warm restart on a pod.

   kubectl -n NAMESPACE exec -it POD_NAME -- bash -c 'kill -SIGUSR1 1'

5. Monitor recovery.

   • Watch AKO and Aerospike logs for reconciler progress and successful recluster activity.
   • Re-run show config like replication-factor until all nodes show the same RF.
   • Re-run asadm -U USERNAME -P PASSWORD -e "info" until Size, Key, and Principal show consistent values on every node, indicating a complete quorum.