Index checkpoints for planned pod restarts

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

Configure the Aerospike Kubernetes Operator (AKO) to save volatile Aerospike Database shared memory before AKO replaces a pod. The replacement pod restores the checkpoint from its persistent volume instead of rebuilding from the disk, or starting empty when the namespace is in-memory without storage-backed persistence. An index checkpoint is not a database backup or a disaster-recovery copy.

Applies to

Aerospike Kubernetes Operator 4.6.0 and later.
Aerospike Database 8.2.0.0 Enterprise Edition and later.
Planned pod restarts that AKO manages, such as a PodRestart operation, a rolling update, or moving a pod off a Kubernetes node listed in spec.k8sNodeBlockList.
Kubernetes operators and site reliability engineers who manage persistent storage and planned maintenance.

Prerequisites

List index-checkpoint in spec.previewFeatures. The opt-in is required to configure index-checkpoint-path, index-checkpoint-threads, index-checkpoint-compression, or skip-checkpoint. If you deploy the cluster with the aerospike-cluster Helm chart, set the chart’s previewFeatures value instead, and set image.tag to 8.2.0.0 or later.
Back index-checkpoint-path with a volume in spec.storage.volumes whose source is a persistentVolume and whose volumeMode is Filesystem, private to each Aerospike pod.
Use a durable storage class that retains the PVC during pod restarts. For replacement on a different Kubernetes node, use storage that can detach from one node and attach to another.
Mount an existing absolute directory that the Database process owns and can write.
Verify that no namespace name is ., .., contains /, or ends in .tmp or .deleting.
Configure maintenance tooling to wait for AKO instead of force-deleting a pod during checkpoint creation.
Size the PVC for one uncompressed checkpoint copy for all checkpointed namespaces on one Database node, plus operational headroom.
Use a replication-factor of 2 or greater for an in-memory namespace without storage-backed persistence. With a replication-factor of 1, anything that loses the checkpoint permanently loses that namespace’s data.
AKO cannot add a persistent volume to a running cluster in place, so an existing cluster needs the rack revision workflow rather than a direct CR edit.

Configure the index checkpoint

The configuration is the same for a new and an existing cluster, and only how you apply it differs. For an existing cluster, apply it through Storage scaling using Rack Revision, which migrates one pod at a time. Each step notes what changes for an existing cluster.

Estimate the checkpoint PVC size for one pod.

The PVC holds one uncompressed checkpoint for every checkpointed namespace on the node. For the sizing rule, see Size the checkpoint volume. For what each namespace contributes, see What the checkpoint captures.
Add the checkpoint volume and Database configuration to the AerospikeCluster CR file.

The following CR excerpt configures common storage for a new cluster. For an existing cluster, add the equivalent volume and Database configuration as part of the linked rack revision workflow instead of adding the persistent volume in place. The configuration creates one checkpoint PVC per pod and mounts it at /opt/aerospike/checkpoint. Follow Storage provisioning to choose a durable storage class that meets the prerequisites. Replace CHECKPOINT_STORAGE_CLASS with the storage class name. Replace CHECKPOINT_VOLUME_SIZE with a Kubernetes storage quantity such as 20Gi.
AerospikeCluster CR excerpt
```
spec:
  image: aerospike/aerospike-server-enterprise:8.2.0.0

  previewFeatures:
    - index-checkpoint

  storage:
    volumes:
      - name: index-checkpoint
        aerospike:
          path: /opt/aerospike/checkpoint
        source:
          persistentVolume:
            storageClass: CHECKPOINT_STORAGE_CLASS
            volumeMode: Filesystem
            size: CHECKPOINT_VOLUME_SIZE
            accessModes:
              - ReadWriteOnce

  aerospikeConfig:
    service:
      index-checkpoint-path: /opt/aerospike/checkpoint
    namespaces:
      - name: test
        replication-factor: 2
        index-checkpoint-threads: 16
        index-checkpoint-compression: true
        skip-checkpoint: false
        storage-engine:
          type: memory
          data-size: 1073741824
```
index-checkpoint-path defines one directory for the Database node. Aerospike Database creates a subdirectory named for the node ID, in hexadecimal, under that path, and a subdirectory for each checkpointed namespace under the node directory.

index-checkpoint-threads accepts values from 1 through 16 and defaults to 16. Set the value based on the processor and storage bandwidth available to the pod. index-checkpoint-compression defaults to true. Set skip-checkpoint: true on a namespace that must not be checkpointed. Aerospike Database also silently skips a namespace whose primary index, secondary indexes, and data are all on durable storage. If no namespace on the pod is checkpointed, AKO replaces the pod without a checkpoint save.

Danger
An in-memory namespace without storage-backed persistence that skips the checkpoint starts empty after every pod restart. Its data survives only if surviving replicas hold a copy. With a replication-factor of 1, a planned pod restart permanently loses that namespace’s data.

AKO also sets skip-checkpoint=true dynamically, before a save, on any namespace whose checkpoint the replacement pod could not use: the cluster-wide index-checkpoint-path moved, the namespace no longer checkpoints in the CR, or its in-memory layout changed because data-size changed or storage backing was gained or lost. A running value of true therefore does not always correspond to a CR setting.
Merge the excerpt into the existing AerospikeCluster CR file and apply the file.
Terminal
```
kubectl apply -f YOUR_CR_FILE
```
Replace YOUR_CR_FILE with the path to the complete CR file. For an existing cluster, copy each rack’s effective storage configuration into spec.rackConfig.racks[].storage, add the checkpoint volume, and update that rack’s revision. Keep every existing volume in the rack storage configuration. Add index-checkpoint-path to the common Database configuration as shown in the excerpt. Follow the linked Rack Revision workflow to migrate one pod at a time.
Wait for AKO to finish reconciling the cluster.
Terminal
```
kubectl -n NAMESPACE wait \
  --for=jsonpath='{.status.phase}'=Completed \
  aerospikecluster/CLUSTER_NAME \
  --timeout=30m
```
Replace NAMESPACE and CLUSTER_NAME with the Kubernetes namespace and AerospikeCluster name.

Verify the configuration

List the checkpoint PVCs:

kubectl -n NAMESPACE get pvc

Confirm that each Aerospike pod has a bound checkpoint PVC. The PVCs use the configured storage class and requested capacity.

Confirm which namespaces the pod actually checkpoints:

kubectl -n NAMESPACE exec POD_NAME -c aerospike-server -- \
  asinfo -v 'checkpoint-status'

Each checkpointed namespace appears with state=none until the first save, which is the steady-state answer rather than an error:

test:state=none:files_completed=0:files_total=0:is_parked=false:park_ms=0

A namespace missing from this output is not checkpointed. A pod with no checkpointed namespaces returns a single node-global entry instead, with no namespace name. That pod is replaced normally and restores nothing.

is_parked=false:park_ms=0

Planned pod restart and uncontrolled failure behavior

Before a planned pod restart, AKO sends checkpoint-save to every pod in a rolling-update batch before it deletes any of them, then deletes each pod as soon as that pod’s save finishes.

AKO sets skip-checkpoint=true dynamically on any namespace whose checkpoint the replacement pod could not use.
AKO sends checkpoint-save to each pod in the batch.
AKO polls checkpoint-status on each parked pod.
When every checkpointed namespace on a pod reports done or failed, AKO deletes that pod without waiting for the rest of the batch.
The StatefulSet recreates the pod and mounts the same PVC, which keeps the checkpoint associated with the same pod identity.
Aerospike Database restores the checkpoint automatically during startup.

A failed checkpoint does not stop the replacement. AKO records an IndexCheckpointCompleted event listing the namespaces it saved, and an IndexCheckpointFailed warning event for any namespace whose save failed. The replacement pod cold-starts a failed namespace: it rebuilds volatile indexes from durable data, and an in-memory namespace without storage-backed persistence starts empty and heals from surviving replicas.

AKO handles restart types differently:

WarmRestart restarts the Aerospike process in the existing pod. Shared-memory segments remain in the pod’s inter-process communication namespace, so AKO does not create an index checkpoint.
PodRestart and an AKO-managed rolling pod restart create a checkpoint before the pod is deleted when the replacement keeps a compatible shared-memory configuration. The replacement pod restores from the checkpoint volume.
A configuration change that changes shared-memory geometry, such as data-size, cannot restore the earlier checkpoint. A saved segment is compatible only when its shared-memory key and size match the replacement configuration. AKO excludes a namespace from the save when its data-size changes or it gains or loses storage backing, so the replacement pod cold-starts that namespace. If an earlier checkpoint for an in-memory namespace without storage-backed persistence is still on the volume because a previous startup did not consume it, the replacement pod does not start. See Pod stops after a data-size change.
An uncontrolled pod or node failure cannot create a checkpoint before the process stops. If the failure also destroys shared memory, Aerospike Database performs a cold restart and rebuilds volatile indexes from durable data when available. If no checkpoint is on the volume, an in-memory namespace without storage-backed persistence starts empty on that node and heals from surviving replicas.

Troubleshoot checkpoint startup and replacement

Pod does not start after configuration

Verify that you use AKO 4.6.0 and later and Database 8.2.0.0 Enterprise Edition and later. Check the server log for a warning containing checkpoint:. The mounted checkpoint directory must be absolute, must not be a symbolic link, must not be world-writable, and must be owned and writable by the user the Database process runs as.

If the server log reports index checkpoint directory is present but unusable, Aerospike Database found a checkpoint it cannot read because of its ownership, permissions, or a symbolic link. It refuses to start at every replication factor so that the checkpoint is preserved. Fix the ownership or permissions, or remove the foreign entry, and then let the pod restart.

Pod stops after a `data-size` change

An in-memory namespace without storage-backed persistence records its data-size in its checkpoint. If an earlier checkpoint for that namespace is still on the volume after you change data-size, the replacement pod stops during startup at every replication factor and the server log reports the following:

{test} configured data-size no longer matches the on-disk index-checkpoint geometry - revert data-size and restart to recover the data (or remove the checkpoint directory to discard it and start fresh)

AKO does not save a checkpoint for a namespace whose data-size changes, so the checkpoint on the volume is one that an earlier startup did not consume. Choose one of the following:

Revert data-size in the CR to recover the data in the checkpoint.
Remove that namespace’s directory under the node directory in index-checkpoint-path to discard the checkpoint. The namespace then starts empty and receives data from surviving replicas.

PVC remains pending

Check that CHECKPOINT_STORAGE_CLASS exists and supports the requested access mode and capacity. For cross-node rescheduling, verify that the storage class supports detaching and reattaching the volume.

Replacement Pod does not restore from a checkpoint

Check AKO events and operator logs for a failed checkpoint operation. Aerospike Database validates each checkpoint before attaching shared memory. Aerospike server warnings containing checkpoint: identify path, storage, manifest, or checkpoint commit failures. If validation fails, Aerospike Database rebuilds durable indexes. An in-memory namespace without storage-backed persistence starts empty and recovers from surviving replicas when its replication-factor is 2 or greater. Preserve the server log before retrying so you can identify the failed validation. Also verify that skip-checkpoint is not true for the affected namespace and that the same PVC is mounted on the replacement pod.

kubectl -n NAMESPACE get events \
  --field-selector involvedObject.name=CLUSTER_NAME
kubectl -n NAMESPACE logs POD_NAME -c aerospike-server

After a save begins, the Database process cannot return to service. It exits with a nonzero status after a failed save. Correct the storage or configuration problem, wait for the cluster to return to Completed, and use a new operation ID to retry PodRestart.

Next steps

Index checkpoint: The feature, its configuration parameters, and what a checkpoint captures
Checkpoint a node: The checkpoint-save and checkpoint-status operator procedure
Aerospike node maintenance operations
Storage
Operations
Data in memory storage configuration for Aerospike on Kubernetes