Index checkpoint

This page describes how to configure the index checkpoint, available in Database 8.2.0 and later as a preview feature, Enterprise Edition only.

The index checkpoint is a durable copy of a node’s shared memory that lets the node warm restart even when its shared memory segments are gone. It needs both the --preview index-checkpoint startup flag and a configured checkpoint directory.

Not for production

The index checkpoint’s configuration parameters, info commands, and output formats can change before it becomes generally available. To run it in production and receive support for it, contact Aerospike Support.

The name is slightly misleading, as this functionality extends well beyond the primary index. Aerospike holds several things in System V (SysV) shared memory. The checkpoint copies what namespace keeps there, the primary index, the secondary index when that is in shared memory, and the records themselves for an in-memory namespace without storage-backed persistence. For the last case, the checkpoint is the only durable copy of the data. See What the checkpoint captures.

When asd restarts in the same inter-process communication (IPC) namespace, it reattaches to those segments and warm restarts. A container replacement or a host reboot gives the node a fresh IPC namespace, so the segments no longer exist. Without a checkpoint the node then cold restarts, and an in-memory namespace without storage-backed persistence comes back empty. The index checkpoint closes that gap. A single operator command gracefully shuts down the node while writing its shared-memory state to disk on the way out. On the next start, Aerospike automatically reads this checkpoint to rebuild the memory segments

Aerospike calls the automatic rebuild hydrating the checkpoint.

These checkpoints are strictly single-use. After asd successfully hydrates the node state, it deletes the file before the node rejoins the cluster, ensuring the checkpoint protects only that specific restart. If the node performs a warm restart instead, reattaching directly to surviving shared-memory segments without hydrating, the checkpoint remains on disk as a fallback until overwritten by the next checkpoint-save. In both cases, the checkpoint serves as a restart accelerator, not a persistent backup.

When to use the index checkpoint

Using an index checkpoint drastically cuts node recovery times after an ungraceful shutdown by trading a small amount of restart I/O for a much faster startup.

Performance benefits and trade-offs

A cold restart rebuilds the primary index by scanning namespace data storage for every record’s metadata, so it takes longer the more records the node holds. Hydrating a checkpoint reads the index back and reattaches it, exactly as a warm restart does, and skips that scan. Startup time then scales with the size of the index and the throughput of the checkpoint volume rather than with the volume of record data on the storage devices, so the saving grows with the data set and is largest for the namespaces that take longest to cold restart today.

Set against that, checkpoint-save adds a copy to the node’s shutdown, sized by the same in-shared-memory footprint and bounded by the same volume’s throughput. Both directions are I/O-bound, which is why compression is on by default.

When it applies

Use it when the node’s shared memory does not survive the restart:

• Kubernetes: a pod is replaced. A rolling upgrade, an image change, or a reschedule onto another node destroys the pod and starts the replacement in a fresh IPC namespace, so the segments are gone. No external tool can copy them in that window, which is why the Aerospike Shared Memory Tool (ASMT) cannot help here. For a comparison of the two, see Warm restart when shared memory is gone.
• Kubernetes: the host reboots. Every pod on it loses its shared memory, whether or not the pod itself is replaced.
• Bare metal or a virtual machine: the host reboots. The checkpoint replaces the multi-step ASMT runbook with a single command before the reboot and an automatic hydrate afterward.

You do not need it when asd restarts inside the same IPC namespace and the segments survive. That case is already an ordinary warm restart, and it reads no checkpoint. It covers a plain systemctl restart aerospike on bare metal, and in Kubernetes a container restart within a pod that keeps its sandbox, such as the Aerospike Kubernetes Operator’s WarmRestart operation. Its PodRestart operation replaces the pod, so that one does need a checkpoint.

For a strong-consistency enabled namespace, the checkpoint avoids more than the rebuild time. Each record’s replication state lives only in the primary index, so a cold restart loses it and every record on that node enters an unreplicated state until the appeal process resolves it against the acting master. Hydrating the checkpoint preserves the primary index, so the replication state survives and no appeals are needed. That matters most when more than one node restarts, because a partition whose roster replicas were all on restarting nodes has no acting master to appeal to, and every read of an unreplicated record must re-replicate it first.

A namespace with index-type pmem or index-type flash already keeps its primary index outside shared memory, so it warm restarts without a checkpoint whenever its index storage follows the node. The index checkpoint addresses the default index-type shmem. Such a namespace is still checkpointed when its secondary index lives in shared memory, in which case the checkpoint carries that index alone.

What the checkpoint captures

The checkpoint captures each class of state independently, and it captures a class when that class lives in shared memory. The namespace’s storage engine decides only the data stripes.

Class	Captured when
Primary index	The namespace uses the default index-type shmem
Secondary index	The secondary index is in shared memory for any storage engine, and its metadata shut down trusted
Data stripes	The namespace is storage-engine memory without storage-backed persistence, so the checkpoint is the only durable copy of the records

A secondary index on pmem or flash is already durable and is skipped. So is one whose metadata is absent or did not shut down trusted; asd rebuilds those secondary indexes from system metadata on the next start.

A device or pmem namespace therefore checkpoints its primary index, and its secondary index too when that lives in shared memory. Its records stay on the device, so a checkpoint that cannot be hydrated costs a slower cold restart and nothing more.

A storage-backed storage-engine memory namespace is the same: the persistence layer holds the records, so the checkpoint carries the index only, and the hydrate is gated on the persistence layer exactly as it is for a device namespace.

A namespace with nothing volatile, with a durable primary index, a durable secondary index, and durable data, is skipped silently. asd starts normally and simply does not checkpoint it, which is what makes one node-wide path safe on a cluster with a mixed set of namespaces. To confirm which namespaces are actually covered, run checkpoint-status on a running node: only checkpointed namespaces appear in its output.

Data loss risk for in-memory namespaces

For an in-memory namespace without storage-backed persistence, the checkpoint is the only durable copy of the records. When no usable checkpoint exists at all, the namespace comes back empty and recovers only by migration from surviving replicas. With replication-factor 1, when simultaneous node losses exceed the replication factor, or when every node in the cluster is restarted this way, the records are permanently lost and must be restored from an asbackup backup.

Prerequisites

• Namespaces using the default index-type shmem.
• A namespace with something left to checkpoint. A storage-backed storage-engine memory namespace qualifies, and is checkpointed index-only. A namespace whose primary index, secondary index, and data are all already durable is skipped silently rather than rejected.
• A way to pass the --preview index-checkpoint startup flag on every node you checkpoint. See Enable the index checkpoint.
• A durable, non-volatile file system for the checkpoint directory, meeting the path requirements. The directory must be private to this node and must be attached on every boot. See Path requirements.
• Capacity on that file system for each checkpointed namespace’s shared memory footprint, measured uncompressed. See Size the checkpoint volume.
• A user holding the sys-admin role, which the checkpoint-save and checkpoint-status commands require. See Pre-defined roles. That role is enforced only on a cluster with security configured; without it, anyone who can reach the info port can halt a node. See Secure the checkpoint directory.
• A stable node-id. By default Aerospike derives the node ID from the MAC address and the fabric port, and a host reboot can change the MAC. Set an explicit node-id so the node rejoins with the same identity.
• A stable --instance value, if you run more than one asd on the host. Checkpoint blocks are keyed by the instance, so it must be identical when you save the checkpoint and when you hydrate it. A mismatch is not tolerated silently: the node rejects every block in the checkpoint. For an in-memory namespace without storage-backed persistence at replication-factor 1 it halts rather than start empty; at 2 or more it discards the checkpoint and heals by migration, which shows up only as an unexplained cold start. Deployments running a single daemon leave it at the default of 0 and need do nothing here.

Kubernetes

Four requirements have Kubernetes-specific answers, and missing any one of them loses the checkpoint:

• Rejecting tmpfs is not sufficient to guarantee durability. An emptyDir volume on the default disk medium and a hostPath on an ephemeral node both pass the startup check and are still destroyed by the replacement the checkpoint is meant to survive. Back the checkpoint directory with a PersistentVolume that is retained and rebound to the replacement pod under the same node-id.
• terminationGracePeriodSeconds must cover the copy plus the window in which the orchestrator confirms the checkpoint and sends SIGTERM. The server does not bound the copy, so this is the setting that decides whether a slow copy survives: when the grace period expires first, the kubelet sends SIGKILL mid-copy and the checkpoint is lost. Set it against the park timeout you actually use rather than a fixed figure, and measure the copy on a representative namespace. The 30-second default is far too short for any real index.
• The volume must be ReadWriteOnce and bound to one pod. Two pods sharing a checkpoint volume make each other’s startup delete fail, and the pods crash-loop.
• Readiness and liveness probes must not restart the pod during the checkpoint. A TCP-socket probe is unaffected, because the process keeps accepting connections. An exec or HTTP probe that issues an info command fails for the whole save, so either suspend it for the duration or scope it to checkpoint-status, which answers throughout.

The Aerospike Kubernetes Operator (AKO) provisions the volume and sets the grace period. See Provisioning storage for Aerospike on Kubernetes and confirm what your own probes issue.

Enable the index checkpoint

Enabling the index checkpoint takes two things. The --preview index-checkpoint flag enables the feature at startup, and index-checkpoint-path tells the node where to write. Setting the path turns the checkpoint on for every eligible namespace on the node; exclude individual namespaces with skip-checkpoint. Without the flag, asd rejects index-checkpoint-path at startup.

index-checkpoint is one of two preview features in Database 8.2.0, alongside yaml-config. To run both, either separate the names with commas or repeat the flag, as in --preview yaml-config,index-checkpoint.

Under systemd, the shipped aerospike unit expands $ASD_OPTIONS into the asd command line, so set the flag there. Add it to /etc/sysconfig/aerospike:

/etc/sysconfig/aerospike

ASD_OPTIONS="--preview index-checkpoint"

You can set it in a drop-in instead, which keeps the setting out of the packaged files. Name the drop-in so that it sorts after aerospike.conf:

/etc/systemd/system/aerospike.service.d/zz-preview.conf

[Service]
Environment="ASD_OPTIONS=--preview index-checkpoint"

asd-coldstart

asd-coldstart overwrites /etc/systemd/system/aerospike.service.d/aerospike.conf with Environment="ASD_OPTIONS=--cold-start" before it starts the service. systemd applies drop-ins in lexical order and the last assignment to a variable wins, so a drop-in sorting before aerospike.conf, or a value in /etc/sysconfig/aerospike, is replaced outright and the preview flag is lost. asd then refuses to start, because index-checkpoint-path is still configured:

'index-checkpoint-path' is a preview feature; enable it with --preview index-checkpoint

Naming the drop-in zz-preview.conf makes it win instead, which keeps the preview flag but drops --cold-start from an asd-coldstart run. To cold start a checkpointed node, set both together in whichever file sorts last: Environment="ASD_OPTIONS=--cold-start --preview index-checkpoint".

A forced cold start skips the hydrate, so it does not consume the checkpoint: the copy is still on disk after the node comes up, and the next checkpoint-save replaces it.

Reload the unit definition if you added a drop-in, then restart the node so it picks up the flag:

systemctl daemon-reload
systemctl restart aerospike

Where you start asd by hand rather than through a service manager, pass the flag on the command line, as the Aerospike server user:

asd --preview index-checkpoint --config-file /etc/aerospike/aerospike.conf

Either way, confirm the running process has the flag before you depend on it:

ps -o args= -C asd

Expected output:

/usr/bin/asd --preview index-checkpoint --config-file /etc/aerospike/aerospike.conf --fgdaemon

Run asd as the Aerospike server user. A daemon started under another user writes checkpoint files that fail the ownership check described in Secure the checkpoint directory.

Configure the checkpoint directory

index-checkpoint-path is a service-level setting: one absolute directory on the durable file system, for the whole node. Each eligible namespace gets its own <namespace> subdirectory beneath it. There is no per-namespace path.

Setting it checkpoints every eligible namespace on the node. Exclude one with skip-checkpoint in its namespace stanza; a namespace with nothing volatile to checkpoint is skipped without it.

service {
    index-checkpoint-path /opt/aerospike/checkpoint
}

namespace test {
    replication-factor 2

    index-checkpoint-threads 8

    storage-engine device {
        device /dev/DEVICE_NAME
    }
}

namespace archive {
    replication-factor 2

    skip-checkpoint true

    storage-engine device {
        device /dev/OTHER_DEVICE_NAME
    }
}

skip-checkpoint defaults to false. It is the only one of these parameters that is dynamic, so you can exclude a namespace without a restart, and the change applies to the next checkpoint-save:

asinfo -v 'set-config:context=namespace;id=NAMESPACE;skip-checkpoint=true'

Set it in aerospike.conf as well, so that it survives a restart.

index-checkpoint-threads sets how many workers copy the checkpoint blocks, from 1 to 16, and defaults to 16. A value outside that range is a configuration error rather than being clamped: asd refuses to start and logs index-checkpoint-threads must be >= 1 and <= 16, not VALUE. Requires a node restart to take effect.

The same thread count governs both directions, so it also sets how many workers hydrate the index at startup. Raising it shortens both the save and the restart, provided the checkpoint storage can feed the extra workers.

Several namespaces on the same node can share one path. Each gets its own <namespace> subdirectory and its blocks are keyed separately, so saving or hydrating one namespace never touches another’s. Two nodes must never share a path. See Path requirements.

index-checkpoint-compression compresses the blocks as they are written and defaults to true. Requires a node restart to take effect. Hydrating a checkpoint is I/O-bound, and bandwidth to the checkpoint device dominates restart time, so compression reduces both the time to save the checkpoint and the time to read it back. Set it to false only when the checkpoint device has bandwidth to spare and you prefer not to spend the CPU.

Compression does not weaken the checkpoint’s integrity check. The checksum covers the uncompressed bytes, so a stream that decompresses to the wrong contents is caught on hydrate exactly as an uncompressed one would be, and both settings round-trip identically.

Path requirements

asd validates the path at startup and refuses to start unless it is:

• An absolute path.
• An existing directory owned by the Aerospike server user.
• Writable by that user. Ownership alone is not sufficient. A directory the server owns but cannot write to fails the check.
• Not world-writable and not a symbolic link. Group-writable is allowed, so the directory works with the Kubernetes fsGroup pattern.
• On a durable, non-volatile file system. A tmpfs, ramfs, or hugetlbfs path is rejected.
• A dedicated directory. asd rejects a path that is, or contains, the work directory or any namespace’s storage devices or files, storage shadows, or primary- and secondary-index shared memory mount points. The go-live delete removes the whole NAMESPACE subdirectory, so a path that overlaps one of those could destroy unrelated data. Directories that only share a prefix, such as /opt/aerospike/checkpoint and /opt/aerospike/checkpoint-data, do not overlap and are both valid.

An overlayfs path is allowed but logs a warning at startup, because the checkpoint survives only if the upper writable layer is durable. A container image layer is not, so treat the warning as a prompt to move the path onto a mounted volume.

That list is complete. asd does not reject the parameter over what a namespace holds: a namespace with nothing volatile to checkpoint is skipped silently, and the node starts.

Two further requirements are operational, and asd cannot check either at startup.

Give each node its own checkpoint directory, always attached

Never point two nodes at the same directory, whether through a shared PersistentVolume, an NFS or EFS export, or a shared host path. As a node starts it removes each <namespace> directory it hydrated from. A <namespace> or <namespace>.deleting left by another node is one it cannot remove, which makes that delete fail, and asd halts rather than serve on top of a checkpoint it could not clear. The node then crash-loops until the directory is cleared by hand. This is deliberate: a shared checkpoint volume is a configuration error, not something the server works around. The volume is sized for one node’s checkpoint anyway, so a second writer breaks the checkpoint regardless.

The volume must be attached before asd starts. A directory that is missing because its volume did not mount looks exactly like a first start, so the node comes up empty and serves. If the volume comes back later and the node then restarts without its shared memory, it can hydrate the checkpoint it skipped and roll the namespace back to that older state. Fail the start rather than let asd run without the volume.

Choosing and creating this directory has security consequences. See Secure the checkpoint directory before you create it.

Save a checkpoint and stop the node

checkpoint-save writes a checkpoint for every checkpointed namespace as part of the node’s own graceful shutdown. The node then leaves the cluster and the process parks, staying alive and serving checkpoint-status and a re-issued checkpoint-save, so that an operator or an orchestrator can confirm the checkpoint is complete before stopping the process. A re-issued checkpoint-save is idempotent: it reports state and never re-runs the save, returning checkpoint-save already in progress or checkpoint-save already complete as plain success strings, so a retrying script is safe.

The graceful shutdown is what the command does; the checkpoint is what it carries out with it. The two are configured separately, and the shutdown does not depend on the checkpoint. Once index-checkpoint-path is set, a node whose namespaces are all durable or all set skip-checkpoint still accepts checkpoint-save: it returns ok, departs the cluster, shuts down gracefully, parks, and writes nothing. The command is rejected only when index-checkpoint-path itself is unset. This is what makes one uniform command safe to issue across a cluster whose nodes do not all have something to checkpoint, but it also means ok is not evidence that anything was saved. Confirm coverage with checkpoint-status first.

Writing the checkpoint is not time-bounded. The park timeout begins only after every targeted namespace finishes writing its state. By default, the node stays parked for 300 seconds (configurable from 1 to 3,600 seconds using checkpoint-save:timeout=SECONDS). When the timeout elapses, the process exits automatically to prevent an unresponsive orchestrator from holding the node open indefinitely.

The node stops answering monitoring for the whole save

From the moment the node accepts checkpoint-save until the process exits, including the copy and the park, it answers only checkpoint-status and a re-issued checkpoint-save. Every other info command, including statistics, build, partitions, replicas, and peers, returns:

ERROR:22:checkpoint-save in progress - only checkpoint-status and checkpoint-save are available

This is a monitoring problem, not a data-path one. After a quiesce handoff completes the node owns no partitions, so no client transaction reaches it and the park has no effect on clients. If you skip the quiesce, transactions that land on the node during the save fail with a retryable UNAVAILABLE rather than hanging, and clients retry them against the node that now holds the partition.

The process stays alive and keeps accepting connections throughout, so a socket-level health check still passes while every info-level check fails, and monitoring shows the node as down.

Before you issue checkpoint-save, suppress alerting for the node and disable or extend any probe that could restart, evict, or kill it. A probe-driven restart during the copy destroys the checkpoint in progress, and there is no earlier copy to fall back on, so the node comes back cold, which is the outcome the checkpoint exists to prevent.

checkpoint-status answering while everything else refuses is how you tell an intentionally parked node from a dead one. Use it as the health check for the duration.

checkpoint-status is not limited to the parked window. It answers on a normally running node as well, where each checkpointed namespace reports state=none until the first checkpoint-save. That makes it a quick way to confirm which namespaces the feature is actually enabled for, before you depend on it during a maintenance window.

checkpoint-save cannot be undone

Saving a checkpoint cannot be undone. No command checkpoints a running node, nothing cancels a checkpoint once issued, and there is no unquiesce-style reversal. Once you issue checkpoint-save, the node is going down, and the only path forward is to stop the process and restart it. Confirm you intend to take the node out of service before you run it.

Restart one node at a time

checkpoint-save shuts the node down. On a multi-node cluster, restart one node at a time and let migrations finish before you move to the next. For namespaces with strong-consistency, follow the planned maintenance procedure for strong consistency. Restarting more than one node simultaneously without following the SC planned maintenance procedure can drop the cluster below write quorum for some partitions, blocking all writes to those partitions until the node recovers.

Only an explicit checkpoint-save writes a checkpoint. An unplanned stop of any kind, including SIGKILL, an out-of-memory event, a Kubernetes eviction, or an asd crash, writes none, and because the last checkpoint was deleted when the node started, there is nothing on disk to fall back to either. A node lost that way restarts cold, or comes up empty for a storage-engine memory namespace without storage-backed persistence. Treat the index checkpoint as a planned-maintenance tool rather than a durability mechanism: run storage-engine memory namespaces at replication-factor 2 or higher and keep a regular backup schedule regardless of checkpointing.

1. Confirm the checkpoint volume has room for the checkpoint.

   df -h /opt/aerospike/checkpoint

   Free space must be at least the shared memory footprint of every namespace the node checkpoints, measured uncompressed. See Size the checkpoint volume.

2. Quiesce the node and recluster, so that it stops acting as a replica.

   We recommend quiescing, though it is not required. checkpoint-save does not quiesce for you, and without a quiesce clients hold references to a node that is about to depart, which causes timeouts that quiescing avoids. Skip this step and step 3 only when those timeouts are acceptable.

3. Confirm the quiesce handoff is complete before you issue checkpoint-save. Ops/sec must drop to zero on the quiesced node, and proxy counters must stop incrementing.

4. Save the checkpoint and watch it complete.

   manage checkpoint save issues the save and polls the node until every namespace finishes. It requires Tools 13.1.0 (asadm 5.1.0) or later. It prompts for confirmation first, because a checkpoint cannot be undone. manage checkpoint without save only displays status.

   Admin+> manage checkpoint save with NODE

   The node returns ok, leaves the cluster, writes its checkpoint, and parks. It is out of the cluster before the copy starts, so plan capacity for the whole save, not just the park. manage checkpoint save keeps polling the node in the same session after it leaves.

   To hold the park longer than the default 300 seconds, pass --timeout. The range is 1 to 3600 seconds. The value is fixed by the call that starts the checkpoint, so re-issuing does not change it. --poll-interval sets the seconds between polls, default 2. Flags go before with.

   Admin+> manage checkpoint save --timeout 1200 --poll-interval 5 with NODE

   To issue the save without waiting, pass --no-wait and check progress separately with manage checkpoint status. To skip the confirmation prompt in a script, pass --no-warn. See manage checkpoint for every flag and message.

5. Confirm every namespace reached done.

   manage checkpoint save returns once every namespace is done or failed. To read the status again, or after a --no-wait save:

   Admin+> manage checkpoint status with NODE

   ~~~~Shared-Memory Checkpoint (2026-09-24 21:25:15 UTC)~~~
   Namespace|      Node|State|Files|Progress|Parked|    Park
            |          |     |     |        |      |    Time
   test     |node2:3000|done |11/11| 100.0 %|True  |00:00:03
   Number of rows: 1

   A new asadm session reaches a parked node only when seeded at it, for example asadm -h NODE_IP --enable, because the node has left the cluster.

   From a script or an orchestrator, issue the info command directly instead. The response is a semicolon-delimited list with one entry per checkpointed namespace, each of the form NAMESPACE:state=STATE:files_completed=N:files_total=N:is_parked=BOOL:park_ms=N:

   asinfo -h HOST -p PORT -v 'checkpoint-status'

   test:state=copying:files_completed=3:files_total=11:is_parked=false:park_ms=0;analytics:state=done:files_completed=6:files_total=6:is_parked=false:park_ms=0

   state is none when no checkpoint has run, copying while one is in progress, done when it is complete, and failed if it did not finish. files_completed and files_total count the shared memory segment files copied and the total to copy, including both index and data segments.

   is_parked turns true once every namespace has finished and the node has entered the park, and park_ms then climbs on each poll. Together they answer the question a poller actually has: a node with park_ms near your terminationGracePeriodSeconds is one nothing has come to stop.

   A failed state means that namespace has no checkpoint at all once the copy has begun. checkpoint-save deletes the previous checkpoint just before it starts writing the new one, so a save that fails during the copy leaves nothing to fall back on. A save rejected by an earlier check, before the copy starts, leaves the previous checkpoint intact.

   Do not reboot or replace the container on a

   The node is already out of the cluster and cannot be returned to service in place, so the only way forward is to stop the process and start it again. Start it normally, so the node rejoins and re-replicates, then read the log for the reason the save failed and retry it before you perform the maintenance.

   Rebooting or replacing the container instead restarts the node with no checkpoint. A in-memory namespace without storage-backed persistence comes up empty, and every record it held that is not on a surviving replica is gone.

   Do not poll with

   The node has left the cluster at this point, so it has to be reached directly. manage checkpoint status recognizes a parked node and does that, and so does asinfo with an explicit -h. Running the info command through asadm does not:

   # Does not work for polling a checkpointing node
   asadm -h HOST --enable -e 'asinfo -v "checkpoint-status"'

   That form re-checks the cluster before each command, and a node that has departed for a checkpoint fails the check and is dropped. It can return a poll or two first, so one successful result is not evidence the next will work.

   Poll every few seconds rather than continuously: on a cluster with security configured, each asinfo call authenticates from scratch.

   Because the park begins only after the copy finishes, every namespace already reports done or failed by the time the node parks. What bounds the copy is the shutdown timeout of whatever stops the node: TimeoutSec in the systemd unit, or terminationGracePeriodSeconds in Kubernetes. If either cuts the copy short, raise it, then retry with a higher index-checkpoint-threads or faster checkpoint storage.

6. Stop the parked process.

   systemctl stop aerospike

   The unit sets TimeoutSec to 600 seconds, which exceeds the park, so systemd waits for the process to exit rather than stopping it partway through. In a container, let the orchestrator deliver the SIGTERM.

   Where you started asd by hand, signal that process by its own process ID. List the daemons with their command lines, so that you can tell them apart on a host running more than one:

   pgrep -a -x asd

   Expected output:

   4821 /usr/bin/asd --preview index-checkpoint --instance 0 --config-file /etc/aerospike/instance0.conf
   4977 /usr/bin/asd --preview index-checkpoint --instance 1 --config-file /etc/aerospike/instance1.conf

   Match on the --instance value or the --config-file path of the daemon you checkpointed, then signal that process ID:

   kill -TERM ASD_PID

   Replace ASD_PID with the process ID of the parked daemon. Under a service manager, get it from the unit instead:

   systemctl show -p MainPID --value aerospike

   Signalling the wrong daemon shuts down a serving node with no checkpoint, and nothing in the output distinguishes that from success.

   Do not use SIGKILL, which bypasses the park. Letting the park time out is safe once checkpoint-status reports done for every namespace. The node parks on failed too, so a timed-out park is not by itself evidence that a checkpoint was written.

   Check the exit status. The process exits 0 only when every namespace’s checkpoint succeeded and storage shut down cleanly, and non-zero if either failed. The status is the same however the process is reaped, whether by SIGTERM, by SIGINT, or by the park timing out, so a script or an orchestrator can gate the reboot or the container replacement on it.

   systemctl stop aerospike ; echo "exit=$?"

   A non-zero exit means the node will not come up warm

   A failed save leaves no checkpoint. Do not replace the container or reboot the host on a non-zero exit. See state=failed in step 5. For an in-memory namespace without storage-backed persistence, restarting at this point loses every record this node holds that is not on a surviving replica.

7. Replace the container, or reboot the host. Then follow Verify how each namespace started.

   Checkpoint every instance on the host before rebooting

   A host reboot destroys the shared memory of every asd instance on that host, not only the one you checkpointed. Repeat steps 2 through 6 for each instance before you reboot. An instance stopped without checkpoint-save cold restarts, and a storage-engine memory namespace without storage-backed persistence on that instance comes up empty.

The checkpoint reflects the moment checkpoint-save ran, and the node serves no writes after that, so issue it as the last step before you stop the process.

What happens on the next start

There is no hydrate flag. During namespace setup asd reattaches surviving shared memory segments if it finds them, and otherwise hydrates the committed checkpoint. It validates the integrity of each class of checkpoint block before it attaches anything.

Survival is judged per class, and as a set rather than segment by segment. Each index class carries an anchor segment that asd writes last when it refills that class, so a valid anchor is proof that the whole set landed. Where only some of a class’s segments survived, asd adopts the ones that are present as they are and refills only the missing ones from the checkpoint, verifying each against the class manifest before it attaches anything. A class whose anchor does not verify is refilled in full, or falls back to the outcome in the table below.

asd then deletes the checkpoint as the index goes live, before the node rejoins the cluster, but only the checkpoint it actually hydrated from. That is what makes a hydrated checkpoint single-use: from that moment the live index begins to diverge from the copy on disk, so keeping it would let a later restart roll the namespace back.

If a node performs a warm restart by reattaching directly to surviving shared-memory segments without hydrating from disk, the existing checkpoint is preserved. It remains on disk as a recovery fallback until the next checkpoint-save replaces it, ensuring the target directory never holds more than one copy.

Two consequences worth planning around:

• A checkpoint-save is not required before a restart that keeps its shared memory, such as an asd restart on the same host. The node reattaches, and the existing checkpoint survives.
• It is required before anything that destroys shared memory, such as a host reboot, a container replacement, or a move to another host, because that is the case where the node has to hydrate, and hydrating consumes the copy.

A device or pmem namespace hydrates the index checkpoint only when the checkpoint still matches the device it was taken from, so one belonging to different data is rejected rather than trusted. A storage-backed in-memory namespace is gated the same way against its persistence layer, and the rest of this section groups it with device and pmem for that reason.

A checkpoint carries exactly the same cross-version reach as a warm restart, no more and no less, because hydrating one recreates the same shared memory segments and puts them through the same compatibility gate. The binary that hydrates a checkpoint does not have to be the one that saved it.

Cross-version compatibility

A rolling container image upgrade comes up warm wherever a warm restart between the same two builds would. A checkpoint the server cannot use is refused rather than partially read, and the namespace falls back to a cold restart.

Upgrade rollback may cold restart

The rollback leg of a failed upgrade is not guaranteed warm. An older build cannot always read a checkpoint written by a newer one, in which case the namespace cold starts: a device, pmem, or storage-backed namespace rebuilds its index from its persistence layer, and a memory namespace without storage-backed persistence rebuilds its index from the data stripes the checkpoint restored. Plan the rollback leg on that basis.

Most checkpoint problems need no operator action. When the node cannot use a checkpoint but the records exist elsewhere, it discards the checkpoint, logs the reason, and recovers on its own: a namespace with storage-backed persistence rebuilds from its persistence layer, and an in-memory namespace without storage-backed persistence starts empty and is refilled by migration from the other nodes. An invalid index-checkpoint-path is a separate startup error, described in Path requirements.

When the node halts and waits for you

An in-memory namespace without storage-backed persistence is the exception. Its checkpoint, or the stripes still in shared memory, can be the only copy of its records. Rather than start empty over a copy that might be perfectly good, the node stops. It halts before it reads, deletes, or changes anything, so a halted node has lost nothing and its data stays recoverable.

asd halts when:

• The checkpoint directory is unreadable, for example wrong ownership or permissions, a symlink or a non-directory at the path, or an entry left there by another node.
• A transient I/O error left the checkpoint’s validity undetermined, so the node cannot tell whether the copy is good.
• The checkpoint’s data is corrupt and no other node holds those records.
• The restore was rejected because the checkpoint’s blocks carry a different --instance than the node was started with, or belong to another node, and no other node holds those records.
• data-size no longer matches the geometry recorded in the checkpoint, or the geometry of stripes that survived in shared memory.

Each message names the namespace and ends with the action to take. For a data-size change against surviving stripes:

{NAMESPACE} configured data-size no longer matches the surviving in-memory stripe geometry -
revert data-size and restart to recover the data (or remove its shmem segments to discard it
and start fresh)

and against the checkpoint on disk:

{NAMESPACE} configured data-size no longer matches the on-disk index-checkpoint geometry -
revert data-size and restart to recover the data (or remove the checkpoint directory to
discard it and start fresh)

Recover from a halt

1. Restart the node once. A transient storage condition may have cleared, in which case the checkpoint is read normally and the namespace comes up with its records intact. Nothing is consumed by a halted start, so this costs nothing to try.

2. Fix the cause, then restart. Restore the directory’s ownership and permissions, remove the foreign entry or the symlink, or set data-size back to its previous value. The checkpoint then hydrates and the records come back. This is the only recovery that keeps the data.

3. Check --instance before you discard. Checkpoint blocks are keyed by the instance, so a node started with a different value than the one that saved the checkpoint rejects all of them. Compare the --instance on the running command line against the one used for the checkpoint-save, correct it, and restart.

4. If the cause cannot be fixed, discard and heal. Remove the namespace’s checkpoint directory, or start the node with --cold-start. Both have the same effect: the namespace comes up empty and the other nodes refill it by migration.

Try the fix before you discard

Discarding is not a repair. The namespace comes up empty and recovers only because other nodes still hold the records, so it depends on the rest of the cluster being healthy and current. Fixing the directory or reverting data-size recovers the data itself, and a halted node stays recoverable for as long as you leave it down. There is no window to race.

The recommended procedure quiesces a node before checkpoint-save, which hands its partitions to the other nodes first. A node checkpointed that way has nothing left to lose by starting empty.

Setting index-checkpoint-path also constrains namespace names, because the checkpoint directory layout is PATH/NAMESPACE plus .tmp and .deleting siblings. While the path is set, asd refuses to start on a namespace named . or .., one containing /, or one ending in .tmp or .deleting, any of which would collide with that layout.

Situation	device, pmem, or storage-backed in-memory	memory without storage-backed persistence
Shared memory segments survive	Warm restart by reattaching, reading no checkpoint	Warm restart by reattaching, reading no checkpoint
Segments gone, checkpoint valid	Hydrate the index, then warm restart	Hydrate the index, data stripes, and secondary index, then warm restart
Segments gone, only an earlier unused checkpoint on disk	Hydrate it, then warm restart	Hydrate it, then warm restart, and the namespace returns to that checkpoint’s contents
Segments gone, checkpoint missing	Cold restart and rebuild the index from its persistence layer	Namespace starts empty and heals by migration
Segments gone, checkpoint present but unusable	Cold restart and rebuild the index from its persistence layer	The node halts. See When the node halts and waits for you
First start, no checkpoint yet	Ordinary cold restart	Ordinary empty start

The third row is the fallback case. Because a checkpoint the node never hydrated from is kept, a node that later loses its shared memory has that earlier copy to come back on instead of starting empty. At replication-factor 2 or more the cluster reconciles it forward through ordinary partition versions and migration; at replication-factor 1 it is the difference between the last saved state and nothing.

A stale checkpoint is not a correctness problem. Partition versions and migration supersede it, exactly as they do for any node that returns with older data, and deleted records are not resurrected.

A rejected checkpoint needs no cleanup. Rejecting it is not an error and does not halt the start. Because the node did not hydrate from it, the go-live delete leaves it in place, and the next checkpoint-save clears it before writing the replacement.

An in-memory namespace without storage-backed persistence comes up empty only when there is no usable committed checkpoint and no surviving stripes. Where the records exist elsewhere, on a persistence layer or on the other nodes, the node resolves the situation itself, and the only cost is the cold restart duration plus the risk of zombie records. Where they do not, it halts and waits rather than decide for you.

The save is atomic: a partial copy never becomes the committed checkpoint, and the volume holds one checkpoint at a time.

Verify how each namespace started

Each namespace logs how it started. The log holds one of these lines per namespace per boot, so match them against the timestamp of the restart you just performed rather than taking the first hit:

grep 'beginning ' /var/log/aerospike/aerospike.log | tail -n 20

Expected output:

INFO (namespace): {test} beginning warm restart

Where asd logs to the journal instead, use:

journalctl -u aerospike --since "10 min ago" | grep 'beginning '

Two outcomes matter after a container replacement or a host reboot:

• beginning warm restart means the namespace came up with its primary index intact. The replacement or reboot destroyed the shared memory segments, so on this path the index can only have come from the checkpoint.
• beginning cold start means the namespace rebuilt its index by scanning its persistence layer, so no usable checkpoint was found. The log gives the reason. See What happens on the next start for what each reason implies.

Then confirm the node rejoined and the data is back:

asinfo -h HOST -p PORT -v 'namespace/NAMESPACE'

The namespace’s objects count returns to its value from before the shutdown. Confirm migrations finish before you restart the next node:

asinfo -h HOST -p PORT -v 'cluster-stable:size=CLUSTER_SIZE;ignore-migrations=false'

Size the checkpoint volume

Provision each checkpointed namespace’s shared memory footprint, measured uncompressed, plus file system headroom. The volume holds one checkpoint at a time.

Measure the current footprint with ipcs -m and total the Aerospike segments. See Can I monitor system shared memory used by Aerospike? for how to identify them.

If several namespaces share one path, sum the footprint for each, then add file system headroom. Size against the uncompressed footprint. Compression shrinks what lands on disk, but treat that as a speed benefit rather than a capacity budget.

The volume is written only during checkpoint-save and read only at startup, never in the request path, so it can use slower and cheaper durable storage than the data devices.

Secure the checkpoint directory

Without security, any client that reaches the info port can halt a node

checkpoint-save and checkpoint-status require the sys-admin role, but Aerospike enforces that role only when a security stanza is configured. On a cluster with no security stanza, every info command is authorized, so anyone who can reach the info port can issue checkpoint-save and take a node out of service.

That matters more here than for other privileged info commands, which are reversible. This one is not: the node shuts down, leaves the cluster, and parks. Because it parks rather than exits, a service manager does not restart it into a serving state, so the node stays out until an operator intervenes. Repeated across the cluster it is a full outage, and for a storage-engine memory namespace without storage-backed persistence the records go with it.

Before you configure index-checkpoint-path, either enable access control and grant sys-admin only to the operators who need it, or restrict network access to the info port to trusted hosts. In a shared environment, do both.

Write access equals code execution

Write access to index-checkpoint-path is trust-equivalent to running code as the Aerospike server user. When a node hydrates a checkpoint, the contents of that directory become the live primary index and, for an in-memory namespace without storage-backed persistence, the record data. The checkpoint is validated by an unkeyed checksum, which detects accidental corruption but does not detect tampering. An attacker with write access can supply a crafted primary index for any checkpointed namespace, whatever its storage engine, and so hide or redirect its records. For a storage-engine memory namespace without storage-backed persistence the directory also holds the records themselves, so they can inject or modify record contents directly. Either way the result becomes authoritative on restart and propagates cluster-wide through migration.

Directory permissions are the protection. Run the following command as root to create the directory owned by the server user, with mode 0700 where you do not need the Kubernetes fsGroup share:

install -d -m 0700 -o AEROSPIKE_USER -g AEROSPIKE_GROUP /opt/aerospike/checkpoint

Replace AEROSPIKE_USER and AEROSPIKE_GROUP with the user and group asd runs as, which are the values of service.user and service.group. asd refuses to start when the directory is not owned by the user it runs as. See Run Aerospike as a non-root user.

Create the parent directory first if it does not exist, so that install does not create it as 0700 as well. Run this only against a path you control: install -d applies the mode and ownership to an existing directory as well as a new one.

Verify it:

stat -c '%U %G %a' /opt/aerospike/checkpoint

Expected output:

AEROSPIKE_USER AEROSPIKE_GROUP 700

Group-writable is permitted for the fsGroup pattern, which puts every member of that group inside the trust boundary. It is not permission to share the volume between nodes; see Path requirements.

The checkpoint holds the full primary index and, for an in-memory namespace without storage-backed persistence, all record data. Get encryption at rest from the storage layer, such as an encrypted PersistentVolume or block device. The checkpoint does not encrypt its own contents. When you decommission a node or stop checkpointing a namespace, erase the directory with the same care you apply to a data device.

Troubleshooting

Symptom	Cause	Fix
asd rejects index-checkpoint-path at startup	The preview flag is missing	Add --preview index-checkpoint to the startup arguments
checkpoint-save returns ERROR:4:'index-checkpoint-path' is not configured	The service-level index-checkpoint-path is unset, so the feature is off for this node	Set index-checkpoint-path and restart the node with --preview index-checkpoint
checkpoint-save returned ok but no checkpoint was written	The path is set, but every namespace is already fully durable or sets skip-checkpoint, so there was nothing to copy. The node still shut down gracefully and parked. This is not an error	Expected. If you meant a namespace to be covered, clear skip-checkpoint on it, then confirm with checkpoint-status before the next maintenance window
checkpoint-save returns ERROR:22:server is still starting; retry checkpoint-save once ready	The node has not finished starting	Wait for startup to complete, then reissue
checkpoint-save returns ERROR:25:enterprise only	The node is a Community Edition build	Use Enterprise Edition
Re-issuing checkpoint-save returns checkpoint-save already in progress or checkpoint-save already complete	The command is idempotent, so a repeat reports state instead of restarting the save	Expected. These are success responses, not errors, and are safe for a script or orchestrator to retry
checkpoint-save returns ERROR::checkpoint-save failed - see checkpoint-status	A save already ran on this node and failed. The error code is suppressed, so the string carries two colons	Do not replace the container or reboot. Run checkpoint-status, then follow the state=failed guidance
checkpoint-save returns ERROR::checkpoint-save raced a shutdown already in progress - check checkpoint-status	A signal claimed the shutdown first, so no checkpoint is written on this path	Run checkpoint-status to confirm, then restart the node and reissue
A namespace came up cold or empty on a host running several daemons	Another asd instance on the host was stopped without checkpoint-save, or the wrong process was signalled. A host reboot wipes the shared memory of every instance	Checkpoint each instance on the host before rebooting. Identify daemons with pgrep -a -x asd and match on --instance or --config-file
Monitoring reports the node down while a checkpoint is running	Every info command except checkpoint-status and checkpoint-save is refused for the whole save, so info-level scrapes and probes fail while socket-level ones pass	Expected during maintenance. Poll checkpoint-status to confirm the node is parked rather than dead, and suppress alerting for the duration
Another info command returns ERROR:22:checkpoint-save in progress - only checkpoint-status and checkpoint-save are available	The node is parked and serves only checkpoint-status and a re-issued checkpoint-save	Expected. Poll checkpoint-status, then stop the process
checkpoint-status reports state=failed	The save did not finish for that namespace, which leaves it with no checkpoint	Do not reboot or replace the container. Check the log, restart asd so the node rejoins, then retry the save
asd refuses to start after you set index-checkpoint-path	The path is relative, missing, not owned by the server user, world-writable, a symbolic link, or on a volatile file system	Correct the path against the path requirements
asd starts, then halts, and repeats on every restart	The checkpoint directory holds a <namespace> or <namespace>.deleting left by another node, which this node cannot remove, so its startup delete cannot complete	Give each node its own checkpoint directory, then clear the foreign directories by hand. See Path requirements
A namespace came up empty although a checkpoint existed	The checkpoint volume was not attached when asd started, so the directory looked empty	Attach the volume before starting asd, and fail the start rather than run without it. Do not restart again before the volume is back, or the node can hydrate the skipped checkpoint and roll the namespace back
asd refuses to start with configured data-size no longer matches the surviving in-memory stripe geometry	An in-memory namespace without storage-backed persistence kept its shared memory segments across the restart, but data-size changed, so the surviving stripes no longer match the configuration	Revert data-size to its previous value and restart to recover the records. Removing the namespace’s shared memory segments also starts the node but discards every record it holds
asd refuses to start with index checkpoint directory is present but unusable	The checkpoint directory has the wrong ownership or permissions, or a symlink or non-directory is squatting the path, or an entry belongs to another node. The checkpoint may be the namespace’s only copy, so the node halts rather than start empty over it	Fix the ownership and permissions, or remove the foreign entry, then restart. See Recover from a halt
asd refuses to start with index checkpoint could not be read (transient I/O error)	A storage error left the checkpoint’s validity undetermined, so the node cannot tell whether its only copy is good	Restart once. A transient condition may have cleared. If it repeats, resolve the storage condition, then restart. See Recover from a halt
asd refuses to start with configured data-size no longer matches the on-disk index-checkpoint geometry	data-size changed, so the checkpoint records a geometry the configuration no longer describes. This is the on-disk counterpart of the surviving-stripes message above	Revert data-size to its previous value and restart to recover the records, or remove the checkpoint directory to discard it and start fresh
asd refuses to start with index checkpoint DATA manifest is corrupt and is the only copy	The checkpoint’s data is corrupt and no other node holds those records, so starting empty would lose them silently	Restore a good checkpoint if you have one. Otherwise remove the checkpoint directory to accept the loss and start fresh
asd refuses to start with namespace name is unusable with 'index-checkpoint-path'	The namespace is named . or .., contains /, or ends in .tmp or .deleting, which collides with the checkpoint directory layout	Rename the namespace, or unset index-checkpoint-path
A namespace is missing from checkpoint-status output	It has nothing volatile to checkpoint, so it is skipped silently. asd does not report this as an error	Expected. Confirm with checkpoint-status which namespaces are covered before you rely on the feature
checkpoint-status cannot reach the node	The node left the cluster when it parked, and the tool discovered it through a cluster seed	Connect directly to the node’s own address
The namespace cold restarted although a checkpoint existed	The checkpoint did not match the current persistence layer, or it failed its integrity check	No action needed where the records persist outside the checkpoint. Check the log for the reason
An in-memory namespace without storage-backed persistence started empty	No usable committed checkpoint	Wait for migration from surviving replicas. Restore from a backup if the records do not return
The checkpoint save ran out of space	The volume is smaller than the shared memory footprint of the namespaces this node checkpoints, or it holds files that are not this save’s: a checkpoint left by a namespace that stopped checkpointing, a .tmp from an interrupted save, or another node’s data on a shared volume	Check the directory contents, then size the volume to the uncompressed shared memory footprint plus headroom and retry. For an in-memory namespace without storage-backed persistence, do not reboot or replace the container before the retry succeeds. With no committed checkpoint, the namespace starts empty on the next restart
The process exited before checkpoint-status reported done	The copy outran the shutdown timeout of whatever stopped the node, not the park. The park starts only after the copy finishes	Raise TimeoutSec in the systemd unit, or terminationGracePeriodSeconds in Kubernetes, then retry with a higher index-checkpoint-threads or faster checkpoint storage
You want to cancel a checkpoint-save	Not possible. The command has no undo	Let it finish, then stop and restart the node

Disable the index checkpoint

Exclude one namespace

Set skip-checkpoint on it and leave the path in place. This is the safe way to stop checkpointing a namespace: no further checkpoint is written for it, and every other namespace is unaffected.

Any checkpoint already on disk for that namespace stays there. The node only removes a directory it hydrated from, and a skipped namespace never hydrates, so nothing clears it and no later checkpoint-save replaces it. Remove PATH/NAMESPACE by hand once the node is running if you want the space back.

asinfo -v 'set-config:context=namespace;id=NAMESPACE;skip-checkpoint=true'

Set it in aerospike.conf as well, so that it survives a restart.

Turn the feature off for the node

Removing index-checkpoint-path is different. It is the only handle asd has on the checkpoint directory, so once it is gone the startup sweep no longer runs and anything left on disk is stranded.

Delete the directory contents before you remove the path

A stranded checkpoint is not only wasted space. If you later set index-checkpoint-path back to the same directory and the node’s shared memory is gone, asd can hydrate that old checkpoint and roll the namespace back to its state at the last save, resurrecting records deleted since. The startup delete is what prevents this, and removing the path removes it.

For an in-memory namespace without storage-backed persistence, the stranded files are also a full copy of that node’s records at rest, in a directory nothing manages.

1. Stop the node.

2. Delete the contents of the index-checkpoint-path directory. After a normal restart it is already empty; it holds a checkpoint only if the node was last stopped with checkpoint-save.

3. Remove index-checkpoint-path from the service stanza.

4. Start the node.

Each namespace then behaves as it did before, and cold restarts unless its shared memory segments survive the restart.

The --preview index-checkpoint flag only enables the feature. With no index-checkpoint-path set, the flag checkpoints nothing, so it is safe to leave in place. Remove it from ASD_OPTIONS when you want the node back on a stock startup line.

Next steps

• Quiesce a node for the full planned-maintenance procedure this fits into.
• Warm restart when shared memory is gone to compare the index checkpoint with ASMT.