--- title: "Index checkpoints for planned pod restarts" description: "Configure Aerospike Kubernetes Operator (AKO) to preserve Aerospike Database shared memory before planned pod restarts." --- # Index checkpoints for planned pod restarts > For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt) > > All documentation pages available in markdown. Configure the Aerospike Kubernetes Operator (AKO) to save volatile Aerospike Database shared memory before AKO replaces a pod. The replacement pod restores the checkpoint from its persistent volume instead of rebuilding from the disk, or starting empty when the namespace is in-memory without storage-backed persistence. An index checkpoint is not a database backup or a disaster-recovery copy. ## Applies to - Aerospike Kubernetes Operator 4.6.0 and later. - Aerospike Database 8.2.0.0 Enterprise Edition and later. - Planned pod restarts that AKO manages, such as a [`PodRestart` operation](https://aerospike.com/docs/kubernetes/reference/config-reference#operations), a rolling update, or moving a pod off a Kubernetes node listed in [`spec.k8sNodeBlockList`](https://aerospike.com/docs/kubernetes/manage/node-maintenance#k8snodeblocklist). - Kubernetes operators and site reliability engineers who manage persistent storage and planned maintenance. ::: caution The index checkpoint is a preview Database feature, and you opt into it explicitly. List `index-checkpoint` in `spec.previewFeatures` in the `AerospikeCluster` custom resource (CR). AKO starts each pod with the required `--preview index-checkpoint` server option only while that entry is present. Adding or removing a `spec.previewFeatures` entry changes the server container arguments, so AKO replaces every pod in a rolling restart. ::: ::: danger AKO issues the `checkpoint-save` service-control command, not you. Any client that can issue it starts an irreversible shutdown of that node. Enable [Database access control](https://aerospike.com/docs/kubernetes/manage/security/access-control) so only AKO and authorized administrators can run service-control commands. If access control is disabled, restrict network access to the Database service port. ::: ## Prerequisites - List `index-checkpoint` in `spec.previewFeatures`. The opt-in is required to configure `index-checkpoint-path`, `index-checkpoint-threads`, `index-checkpoint-compression`, or `skip-checkpoint`. If you deploy the cluster with the `aerospike-cluster` Helm chart, set the chart’s `previewFeatures` value instead, and set `image.tag` to 8.2.0.0 or later. - Back `index-checkpoint-path` with a volume in `spec.storage.volumes` whose source is a `persistentVolume` and whose `volumeMode` is `Filesystem`, private to each Aerospike pod. - Use a durable storage class that retains the PVC during pod restarts. For replacement on a different Kubernetes node, use storage that can detach from one node and attach to another. - Mount an existing absolute directory that the Database process owns and can write. - Verify that no namespace name is `.`, `..`, contains `/`, or ends in .tmp or .deleting. - Configure maintenance tooling to wait for AKO instead of force-deleting a pod during checkpoint creation. - Size the PVC for one uncompressed checkpoint copy for all checkpointed namespaces on one Database node, plus operational headroom. - Use a [`replication-factor`](https://aerospike.com/docs/database/reference/config#namespace__replication-factor) of 2 or greater for an in-memory namespace without storage-backed persistence. With a `replication-factor` of 1, anything that loses the checkpoint permanently loses that namespace’s data. - AKO cannot add a persistent volume to a running cluster in place, so an existing cluster needs the [rack revision workflow](https://aerospike.com/docs/kubernetes/manage/configure/scaling/#storage-scaling-using-rack-revision) rather than a direct CR edit. ::: danger Checkpoint protection starts only after the configuration and checkpoint volume are active on a pod. The rolling restart that first enables the feature replaces every pod, and it cannot use a checkpoint created by the earlier pod configuration. Verify that replica data is available, update one pod at a time, and wait for migrations to finish during the initial rollout. ::: ## Configure the index checkpoint The configuration is the same for a new and an existing cluster, and only how you apply it differs. For an existing cluster, apply it through [Storage scaling using Rack Revision](https://aerospike.com/docs/kubernetes/manage/configure/scaling#storage-scaling-using-rack-revision), which migrates one pod at a time. Each step notes what changes for an existing cluster. 1. Estimate the checkpoint PVC size for one pod. The PVC holds one uncompressed checkpoint for every checkpointed namespace on the node. For the sizing rule, see [Size the checkpoint volume](https://aerospike.com/docs/database/manage/database/index-checkpoint#size-the-checkpoint-volume). For what each namespace contributes, see [What the checkpoint captures](https://aerospike.com/docs/database/manage/database/index-checkpoint#what-the-checkpoint-captures). 2. Add the checkpoint volume and Database configuration to the `AerospikeCluster` CR file. The following CR excerpt configures common storage for a new cluster. For an existing cluster, add the equivalent volume and Database configuration as part of the linked rack revision workflow instead of adding the persistent volume in place. The configuration creates one checkpoint PVC per pod and mounts it at `/opt/aerospike/checkpoint`. Follow [Storage provisioning](https://aerospike.com/docs/kubernetes/manage/storage/storage-provisioning) to choose a durable storage class that meets the prerequisites. Replace _`CHECKPOINT_STORAGE_CLASS`_ with the storage class name. Replace _`CHECKPOINT_VOLUME_SIZE`_ with a Kubernetes storage quantity such as `20Gi`. AerospikeCluster CR excerpt ```yaml spec: image: aerospike/aerospike-server-enterprise:8.2.0.0 previewFeatures: - index-checkpoint storage: volumes: - name: index-checkpoint aerospike: path: /opt/aerospike/checkpoint source: persistentVolume: storageClass: CHECKPOINT_STORAGE_CLASS volumeMode: Filesystem size: CHECKPOINT_VOLUME_SIZE accessModes: - ReadWriteOnce aerospikeConfig: service: index-checkpoint-path: /opt/aerospike/checkpoint namespaces: - name: test replication-factor: 2 index-checkpoint-threads: 16 index-checkpoint-compression: true skip-checkpoint: false storage-engine: type: memory data-size: 1073741824 ``` `index-checkpoint-path` defines one directory for the Database node. Aerospike Database creates a subdirectory named for the node ID, in hexadecimal, under that path, and a subdirectory for each checkpointed namespace under the node directory. `index-checkpoint-threads` accepts values from 1 through 16 and defaults to 16. Set the value based on the processor and storage bandwidth available to the pod. `index-checkpoint-compression` defaults to `true`. Set `skip-checkpoint: true` on a namespace that must not be checkpointed. Aerospike Database also silently skips a namespace whose primary index, secondary indexes, and data are all on durable storage. If no namespace on the pod is checkpointed, AKO replaces the pod without a checkpoint save. ::: danger An in-memory namespace without storage-backed persistence that skips the checkpoint starts empty after every pod restart. Its data survives only if surviving replicas hold a copy. With a `replication-factor` of 1, a planned pod restart permanently loses that namespace’s data. ::: AKO also sets `skip-checkpoint=true` dynamically, before a save, on any namespace whose checkpoint the replacement pod could not use: the cluster-wide `index-checkpoint-path` moved, the namespace no longer checkpoints in the CR, or its in-memory layout changed because `data-size` changed or storage backing was gained or lost. A running value of `true` therefore does not always correspond to a CR setting. 3. Merge the excerpt into the existing `AerospikeCluster` CR file and apply the file. Terminal ```bash kubectl apply -f YOUR_CR_FILE ``` Replace _`YOUR_CR_FILE`_ with the path to the complete CR file. For an existing cluster, copy each rack’s effective storage configuration into `spec.rackConfig.racks[].storage`, add the checkpoint volume, and update that rack’s `revision`. Keep every existing volume in the rack storage configuration. Add `index-checkpoint-path` to the common Database configuration as shown in the excerpt. Follow the linked Rack Revision workflow to migrate one pod at a time. 4. Wait for AKO to finish reconciling the cluster. Terminal ```bash kubectl -n NAMESPACE wait \ --for=jsonpath='{.status.phase}'=Completed \ aerospikecluster/CLUSTER_NAME \ --timeout=30m ``` Replace _`NAMESPACE`_ and _`CLUSTER_NAME`_ with the Kubernetes namespace and `AerospikeCluster` name. ## Verify the configuration List the checkpoint PVCs: Terminal ```bash kubectl -n NAMESPACE get pvc ``` Confirm that each Aerospike pod has a bound checkpoint PVC. The PVCs use the configured storage class and requested capacity. Confirm which namespaces the pod actually checkpoints: Terminal ```bash kubectl -n NAMESPACE exec POD_NAME -c aerospike-server -- \ asinfo -v 'checkpoint-status' ``` Each checkpointed namespace appears with `state=none` until the first save, which is the steady-state answer rather than an error: ```text test:state=none:files_completed=0:files_total=0:is_parked=false:park_ms=0 ``` A namespace missing from this output is not checkpointed. A pod with no checkpointed namespaces returns a single node-global entry instead, with no namespace name. That pod is replaced normally and restores nothing. ```text is_parked=false:park_ms=0 ``` ## Planned pod restart and uncontrolled failure behavior Before a planned pod restart, AKO sends `checkpoint-save` to every pod in a rolling-update batch before it deletes any of them, then deletes each pod as soon as that pod’s save finishes. 1. AKO sets `skip-checkpoint=true` dynamically on any namespace whose checkpoint the replacement pod could not use. 2. AKO sends `checkpoint-save` to each pod in the batch. 3. AKO polls `checkpoint-status` on each parked pod. 4. When every checkpointed namespace on a pod reports `done` or `failed`, AKO deletes that pod without waiting for the rest of the batch. 5. The StatefulSet recreates the pod and mounts the same PVC, which keeps the checkpoint associated with the same pod identity. 6. Aerospike Database restores the checkpoint automatically during startup. A failed checkpoint does not stop the replacement. AKO records an `IndexCheckpointCompleted` event listing the namespaces it saved, and an `IndexCheckpointFailed` warning event for any namespace whose save failed. The replacement pod cold-starts a failed namespace: it rebuilds volatile indexes from durable data, and an in-memory namespace without storage-backed persistence starts empty and heals from surviving replicas. AKO handles restart types differently: - `WarmRestart` restarts the Aerospike process in the existing pod. Shared-memory segments remain in the pod’s inter-process communication namespace, so AKO does not create an index checkpoint. - `PodRestart` and an AKO-managed rolling pod restart create a checkpoint before the pod is deleted when the replacement keeps a compatible shared-memory configuration. The replacement pod restores from the checkpoint volume. - A configuration change that changes shared-memory geometry, such as `data-size`, cannot restore the earlier checkpoint. A saved segment is compatible only when its shared-memory key and size match the replacement configuration. AKO excludes a namespace from the save when its `data-size` changes or it gains or loses storage backing, so the replacement pod cold-starts that namespace. If an earlier checkpoint for an in-memory namespace without storage-backed persistence is still on the volume because a previous startup did not consume it, the replacement pod does not start. See [Pod stops after a `data-size` change](#pod-stops-after-a-data-size-change). - An uncontrolled pod or node failure cannot create a checkpoint before the process stops. If the failure also destroys shared memory, Aerospike Database performs a cold restart and rebuilds volatile indexes from durable data when available. If no checkpoint is on the volume, an in-memory namespace without storage-backed persistence starts empty on that node and heals from surviving replicas. ::: caution AKO uses a Transmission Control Protocol (TCP) readiness probe, and the Database process continues accepting TCP connections while it is parked after `checkpoint-save`. An additional probe that sends an info command can fail during the park because most info commands are unavailable. `checkpoint-status` and a repeated `checkpoint-save` remain available. Do not configure such a probe to restart the pod while a checkpoint is in progress. ::: ## Troubleshoot checkpoint startup and replacement ### Pod does not start after configuration Verify that you use AKO 4.6.0 and later and Database 8.2.0.0 Enterprise Edition and later. Check the server log for a warning containing `checkpoint:`. The mounted checkpoint directory must be absolute, must not be a symbolic link, must not be world-writable, and must be owned and writable by the user the Database process runs as. If the server log reports `index checkpoint directory is present but unusable`, Aerospike Database found a checkpoint it cannot read because of its ownership, permissions, or a symbolic link. It refuses to start at every replication factor so that the checkpoint is preserved. Fix the ownership or permissions, or remove the foreign entry, and then let the pod restart. ### Pod stops after a `data-size` change An in-memory namespace without storage-backed persistence records its `data-size` in its checkpoint. If an earlier checkpoint for that namespace is still on the volume after you change `data-size`, the replacement pod stops during startup at every replication factor and the server log reports the following: ```text {test} configured data-size no longer matches the on-disk index-checkpoint geometry - revert data-size and restart to recover the data (or remove the checkpoint directory to discard it and start fresh) ``` AKO does not save a checkpoint for a namespace whose `data-size` changes, so the checkpoint on the volume is one that an earlier startup did not consume. Choose one of the following: - Revert `data-size` in the CR to recover the data in the checkpoint. - Remove that namespace’s directory under the node directory in `index-checkpoint-path` to discard the checkpoint. The namespace then starts empty and receives data from surviving replicas. ### PVC remains pending Check that _`CHECKPOINT_STORAGE_CLASS`_ exists and supports the requested access mode and capacity. For cross-node rescheduling, verify that the storage class supports detaching and reattaching the volume. ### Replacement Pod does not restore from a checkpoint Check [AKO events](https://aerospike.com/docs/kubernetes/reference/troubleshooting#events) and [operator logs](https://aerospike.com/docs/kubernetes/reference/troubleshooting#operator-logs) for a failed checkpoint operation. Aerospike Database validates each checkpoint before attaching shared memory. Aerospike server warnings containing `checkpoint:` identify path, storage, manifest, or checkpoint commit failures. If validation fails, Aerospike Database rebuilds durable indexes. An in-memory namespace without storage-backed persistence starts empty and recovers from surviving replicas when its `replication-factor` is 2 or greater. Preserve the server log before retrying so you can identify the failed validation. Also verify that `skip-checkpoint` is not `true` for the affected namespace and that the same PVC is mounted on the replacement pod. Terminal ```bash kubectl -n NAMESPACE get events \ --field-selector involvedObject.name=CLUSTER_NAME kubectl -n NAMESPACE logs POD_NAME -c aerospike-server ``` After a save begins, the Database process cannot return to service. It exits with a nonzero status after a failed save. Correct the storage or configuration problem, wait for the cluster to return to `Completed`, and use a new operation ID to retry `PodRestart`. ## Next steps - [Index checkpoint](https://aerospike.com/docs/database/manage/database/index-checkpoint): The feature, its configuration parameters, and what a checkpoint captures - [Checkpoint a node](https://aerospike.com/docs/database/manage/cluster/checkpoint-node): The `checkpoint-save` and `checkpoint-status` operator procedure - [Aerospike node maintenance operations](https://aerospike.com/docs/kubernetes/manage/node-maintenance) - [Storage](https://aerospike.com/docs/kubernetes/reference/config-reference#storage) - [Operations](https://aerospike.com/docs/kubernetes/reference/config-reference#operations) - [Data in memory storage configuration for Aerospike on Kubernetes](https://aerospike.com/docs/kubernetes/manage/storage/data-in-memory)