--- title: "Cluster consistency" description: "Manage Aerospike Strong Consistency clusters: add/remove nodes, update rosters, and perform planned maintenance." --- # Cluster consistency > For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt) > > All documentation pages available in markdown. This page describes how to manage the cluster nodes of a [strong consistency](https://aerospike.com/docs/database/learn/strong-consistency) (SC) namespace in the Aerospike Database. ## Add nodes to the cluster and roster Use the `asadm` [manage roster & show roster](https://aerospike.com/docs/database/tools/asadm/live-mode#roster) commands. Otherwise, use the equivalent [asinfo - roster](https://aerospike.com/docs/database/reference/info#roster) and [asinfo - roster-set](https://aerospike.com/docs/database/reference/info#roster-set) commands. 1. Install and configure Aerospike on the new nodes as described in [Configure strong consistency](https://aerospike.com/docs/database/manage/namespace/consistency). 2. When the nodes have joined the cluster, use the following command to verify that the result in [`cluster_size`](https://aerospike.com/docs/database/reference/metrics#node_stats__cluster_size) is greater than the result in `ns_cluster_size`. In the following output, all 4 nodes report `cluster_size: 4`, but only the 3 existing roster members report `ns_cluster_size: 3`. The newly added node (node2) reports `ns_cluster_size: 0` because it is not yet on the roster. Terminal window ```bash Admin> show stat -flip like cluster_size ~Service Statistics (2026-04-14 00:32:38 UTC)~ Node|cluster_size node1:3000 | 4 node2:3000 | 4 node3:3000 | 4 node4:3000 | 4 Number of rows: 4 ~test Namespace Statistics (2026-04-14 00:32:38 UTC)~ Node|ns_cluster_size node1:3000 | 3 node2:3000 | 0 node3:3000 | 3 node4:3000 | 3 Number of rows: 4 ``` 3. Use `show roster` to see the newly observed nodes in its Observed Nodes section. 4. Use the following command to copy the `Observed Nodes` list into the `Pending Roster`. Terminal window ```bash Admin> enable Admin+> manage roster stage observed ns test Pending roster now contains observed nodes. Run "manage recluster" for your changes to take affect. ``` 5. Activate the new roster with the `manage recluster` command. Terminal window ```bash Admin+> manage recluster Successfully started recluster ``` 6. Run `show roster` to confirm that the roster has been updated on all nodes. Verify that the service’s `cluster_size` matches the namespace’s `ns_cluster_size`. ## Remove nodes and update the roster This section describes how to remove a node from an existing namespace configured with SC. ::: caution Do not simultaneously remove a number of nodes equal to or greater than your [replication-factor](https://aerospike.com/docs/database/reference/config#namespace__replication-factor) (RF). Removing too many nodes at the same time causes [unavailable partitions](https://aerospike.com/docs/database/learn/strong-consistency#dead-partitions). If you observe unavailable partitions, re-add the nodes and wait for the cluster to synchronize before proceeding. With RF=1, some partitions become unavailable whenever **any** node leaves the cluster, making rolling operations impractical. With RF=N on an N-node cluster, all partitions become unavailable during a single node shutdown. ::: ### Remove node from the cluster 1. Run `show roster`. Verify all roster nodes are present in the cluster. Terminal window ```bash Admin+> show roster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2026-04-14 00:32:55 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Node| Node ID|Namespace| Current Roster| Pending Roster| Observed Nodes node1:3000|*BB978F2CCF9F18A|test |BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA node2:3000| BB9547249D8CE66|test |BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA node3:3000| BB919C6ABA304FA|test |BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA node4:3000| BB957376A7ADD8E|test |BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA Number of rows: 4 ``` ::: note In [rack-aware](https://aerospike.com/docs/database/manage/namespace/rack-aware) clusters, node IDs in the roster include an `@rack-id` suffix (for example, `BB978F2CCF9F18A@1`). This suffix indicates which rack each node belongs to. ::: 2. Safely shut down the nodes to be removed with the following command (in this example, on `node2:3000` with node id `BB9547249D8CE66`). Verify you are removing fewer nodes than your configured RF. ::: note Any operational procedure that executes a `SIGTERM` is safe. After startup, a `SIGTERM` flushes data to disk and properly signal other servers. ::: Terminal window ```bash systemctl stop aerospike ``` 3. When migrations are complete, run the `stat` command on `partitions_remaining` until the `migrate_partitions_remaining` stat becomes zero on all nodes. Terminal window ```bash Admin> show stat service like partitions_remaining -flip ~~~~~Service Statistics (2026-04-14 00:33:10 UTC)~~~~ Node|migrate_partitions_remaining node1:3000 | 0 node3:3000 | 0 node4:3000 | 0 Number of rows: 3 ``` ### Remove node from the roster 1. Run `show roster` to verify that BB9547249D8CE66 is no longer in the `Observed Nodes`. In the following example there is one fewer node in the `Observed Nodes` column than in `Current Roster` and `Pending Roster`. Terminal window ```bash Admin+> show roster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2026-04-14 00:34:56 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Node| Node ID|Namespace| Current Roster| Pending Roster| Observed Nodes node1:3000|*BB978F2CCF9F18A|test |BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB919C6ABA304FA node3:3000| BB919C6ABA304FA|test |BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB919C6ABA304FA node4:3000| BB957376A7ADD8E|test |BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB9547249D8CE66,BB919C6ABA304FA|BB978F2CCF9F18A,BB957376A7ADD8E,BB919C6ABA304FA Number of rows: 3 ``` 2. Copy the `Observed Nodes` list into the `Pending Roster`. Terminal window ```bash Admin+> manage roster stage observed ns test Pending roster now contains observed nodes. Run "manage recluster" for your changes to take affect. ``` 3. Run `manage recluster` to apply the change. Terminal window ```bash Admin+> manage recluster Successfully started recluster ``` 4. Check if startup is complete. Run the following command periodically until it returns `ok`. Terminal window ```bash asinfo -h [ip of host] -v 'status' ``` ::: caution Avoid executing a process shut down such as restarting during startup. For more information, see [Dead partitions](https://aerospike.com/docs/database/learn/strong-consistency#dead-partitions). ::: ## Planned maintenance For rolling upgrades and planned maintenance, such as OS patching or host reboots on SC namespaces, process **one node at a time**. In multi-AZ rack-aware deployments, you can process [one rack at a time](#rack-at-a-time-multi-az-rack-aware-deployments). During planned maintenance, the same nodes return to the cluster after the upgrade or reboot, so the [roster](https://aerospike.com/docs/database/learn/architecture/clustering/consistency-modes#roster-of-nodes) does not need to be updated. The roster only changes when you permanently [add nodes](#add-nodes-to-the-cluster-and-roster) or [remove nodes](#remove-nodes-and-update-the-roster). If a node fails to rejoin after maintenance and must be replaced, see [Remove nodes and update the roster](#remove-nodes-and-update-the-roster) for how to update the roster, and [Revive dead partitions](#revive-dead-partitions) if dead partitions result. If a procedure is stuck on a verification step or you encounter unexpected behavior, see [Troubleshooting](https://aerospike.com/docs/database/advanced/troubleshoot/), search the [Support Knowledge Base](https://support.aerospike.com/hc/en-us), or open a support case. ::: note Processing one node at a time is important in SC namespaces. When a node cold restarts, its records enter an **unreplicated** state until the [appeal process](https://aerospike.com/docs/database/reference/metrics#namespace__appeals_tx_remaining) completes against the other nodes. If multiple nodes cold restart simultaneously, partitions where all roster replicas are on the restarting nodes have no node to appeal to, causing [dead partitions](https://aerospike.com/docs/database/learn/strong-consistency#dead-partitions). Even with warm restart using ASMT, processing one node at a time keeps the cluster fully available. You do **not** need to wait for migrations to complete before proceeding to the next node. However, the quiesce handoff must complete on the next node before shutting it down. ::: ### Before you begin Configure [`migrate-fill-delay`](https://aerospike.com/docs/database/reference/config#service__migrate-fill-delay) **on every node** to a value that exceeds the expected time for a single node to complete maintenance and rejoin. This suppresses unnecessary “fill” migrations to stand-in (non-roster) replicas while a node is temporarily out of the cluster. In SC namespaces, `migrate-fill-delay` only affects non-roster replicas; roster replica migrations proceed immediately. See [Delay migrations](https://aerospike.com/docs/database/manage/cluster/delay-migrations) for details. ::: caution If you set `migrate-fill-delay` dynamically, the value reverts to the static configuration on node restart. Since planned maintenance involves restarting nodes, set this value in the configuration file (`aerospike.conf`) so it persists across restarts. ::: ### Upgrading Aerospike (asd restart only, no host reboot) When only the Aerospike daemon is restarted, for example, during a rolling upgrade, the node can [warm restart](https://aerospike.com/docs/database/manage/database/fast-start) because shared memory segments holding the primary index and [secondary indexes in shared memory](https://aerospike.com/docs/database/manage/namespace/secondary-index#default-secondary-index-storage) survive. Process one node at a time: 1. [Quiesce](https://aerospike.com/docs/database/manage/cluster/quiesce-node#quiesce-a-node) the node, then trigger a recluster. Terminal window ```bash Admin+> manage quiesce with Admin+> manage recluster ``` Verify: `show statistics like pending_quiesce` shows `true` on the target: Terminal window ```bash Admin+> show statistics like pending_quiesce ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~test Namespace Statistics (2026-04-14 00:28:10 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Node |node1:3000|node2:3000|node3:3000|node4:3000 pending_quiesce|false |true |false |false Number of rows: 2 ``` After recluster, `show statistics like quiesce` shows `effective_is_quiesced: true` on the target and `nodes_quiesced: 1` on all nodes: Terminal window ```bash Admin+> show statistics like quiesce ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~test Namespace Statistics (2026-04-14 00:28:20 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Node |node1:3000|node2:3000|node3:3000|node4:3000 effective_is_quiesced|false |true |false |false nodes_quiesced |1 |1 |1 |1 pending_quiesce |false |true |false |false Number of rows: 4 ``` 2. Wait for the quiesce handoff to complete. The quiesced node hands off master status. Wait until no active traffic or proxies are hitting the quiesced node. Terminal window ```bash Admin+> show latencies ``` Verify: ops/sec drops to zero on the quiesced node, then confirm `client_proxy_*` and `batch_sub_proxy_*` counters stop incrementing on the quiesced node, and `from_proxy_*` counters stop on the remaining nodes. See the [quiesce verification reference](https://aerospike.com/docs/database/manage/cluster/quiesce-node#verify-quiesce-handoff) for full output examples. 3. Shut down `asd`, perform the upgrade, and restart `asd`. The node warm restarts. Terminal window ```bash sudo systemctl stop aerospike # ... perform upgrade ... sudo systemctl start aerospike ``` Verify: `info network` shows the node has rejoined at the expected cluster size. Terminal window ```bash Admin> info network ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information (2026-04-14 00:29:18 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Node| Node ID| IP| Build|Migrations|~~~~~~~~~~~~~~~~~~Cluster~~~~~~~~~~~~~~~~~~|Client| Uptime | | | | |Size| Key|Integrity| Principal| Conns| node1:3000 | BB94B7AEB45DB52|10.0.3.1:3000|E-8.1.1.2| 0.000 | 4|AA5AF50552AF|True |BB9D787F6BAF3D6| 5|00:20:34 node2:3000 |*BB9D787F6BAF3D6|10.0.3.2:3000|E-8.1.1.2| 0.000 | 4|AA5AF50552AF|True |BB9D787F6BAF3D6| 5|00:00:14 node3:3000 | BB989A1BF1D8116|10.0.3.3:3000|E-8.1.1.2| 0.000 | 4|AA5AF50552AF|True |BB9D787F6BAF3D6| 5|00:20:34 node4:3000 | BB9D48DF5A70CEE|10.0.3.4:3000|E-8.1.1.2| 0.000 | 4|AA5AF50552AF|True |BB9D787F6BAF3D6| 5|00:20:34 Number of rows: 4 ``` 4. [Validate](#validate-partitions) that the cluster has no unavailable or dead partitions. 5. Repeat from step 1 for the next node. ### Planned maintenance with host reboot When the host itself is rebooted, shared memory is wiped and the node [cold restarts](https://aerospike.com/docs/database/manage/database/cold-start) unless the primary index is persisted beforehand. Use [ASMT](https://aerospike.com/docs/database/tools/asmt) to avoid a cold restart and its side effects, such as potential unreplicated records and zombie records. Process one node at a time: 1. [Quiesce](https://aerospike.com/docs/database/manage/cluster/quiesce-node#quiesce-a-node) the node, then trigger a recluster. Terminal window ```bash Admin+> manage quiesce with Admin+> manage recluster ``` Verify: same as [rolling upgrade step 1](#upgrading-aerospike-asd-restart-only-no-host-reboot). 2. Wait for the quiesce handoff to complete. Terminal window ```bash Admin+> show latencies ``` Verify: same as [rolling upgrade step 2](#upgrading-aerospike-asd-restart-only-no-host-reboot). 3. Shut down `asd`. Terminal window ```bash sudo systemctl stop aerospike ``` 4. Back up the indexes of each namespace with [`asmt`](https://aerospike.com/docs/database/tools/asmt/backup). The `-z` option enables [compression](https://aerospike.com/docs/database/tools/asmt/options), which is recommended for planned maintenance. Terminal window ```bash sudo asmt -b -v -z -p -n ``` See [Back up indexes with ASMT](https://aerospike.com/docs/database/tools/asmt/backup) for full output details. ::: note ASMT is not needed for namespaces whose primary index is configured on Persistent Memory or Flash (SSDs), as those persist across reboots. Similarly, secondary indexes configured with [`sindex-type flash`](https://aerospike.com/docs/database/manage/namespace/secondary-index#secondary-index-on-flash) or [`sindex-type pmem`](https://aerospike.com/docs/database/manage/namespace/secondary-index#secondary-index-in-pmem) persist across reboots and do not require ASMT. See [When not to use ASMT](https://aerospike.com/docs/database/tools/asmt#when-not-to-use-asmt). ::: 5. Reboot the host and perform OS or hardware maintenance. 6. After the host is back, restore the indexes of each namespace with [`asmt`](https://aerospike.com/docs/database/tools/asmt/restore). The `-z` option is not needed; ASMT auto-detects compressed files. Terminal window ```bash sudo asmt -r -v -p -n ``` See [Restore indexes with ASMT](https://aerospike.com/docs/database/tools/asmt/restore) for full output details. 7. Start `asd`. The node warm restarts from the restored index instead of cold restarting. The Aerospike log confirms this with `beginning warm restart` for each namespace (instead of `beginning cold start`). Terminal window ```bash sudo systemctl start aerospike ``` Verify: `info network` shows the node has rejoined at the expected cluster size. Terminal window ```bash Admin> info network ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information (2026-04-14 00:35:42 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Node| Node ID| IP| Build|Migrations|~~~~~~~~~~~~~~~~~~Cluster~~~~~~~~~~~~~~~~~~|Client| Uptime | | | | |Size| Key|Integrity| Principal| Conns| node1:3000 | BB94B7AEB45DB52|10.0.3.1:3000|E-8.1.1.2| 0.000 | 4|3866DA39491B|True |BB9D48DF5A70CEE| 5|00:23:37 node2:3000 | BB90B0CA8BD688A|10.0.3.2:3000|E-8.1.1.2| 0.000 | 4|3866DA39491B|True |BB9D48DF5A70CEE| 5|00:00:14 node3:3000 | BB989A1BF1D8116|10.0.3.3:3000|E-8.1.1.2| 0.000 | 4|3866DA39491B|True |BB9D48DF5A70CEE| 5|00:23:37 node4:3000 |*BB9D48DF5A70CEE|10.0.3.4:3000|E-8.1.1.2| 0.000 | 4|3866DA39491B|True |BB9D48DF5A70CEE| 5|00:23:37 Number of rows: 4 ``` 8. [Validate](#validate-partitions) that the cluster has no unavailable or dead partitions. 9. Repeat from step 1 for the next node. ::: containerized and ephemeral environments By default, the Aerospike [node ID](https://aerospike.com/docs/database/reference/config#service__node-id) is derived from the node’s MAC address and fabric port. In containerized or cloud environments where a host reboot can change the MAC address, the node rejoins with a different node ID. This causes the cluster to treat it as a new node, disrupting migrations, ASMT warm restart, and roster membership. To prevent this, configure an explicit [`node-id`](https://aerospike.com/docs/database/reference/config#service__node-id) for each node. See [Assign node IDs](https://aerospike.com/docs/database/manage/namespace/consistency#assign-node-ids-optional) for details. ::: ### Rack at a time (multi-AZ rack-aware deployments) In a [rack-aware](https://aerospike.com/docs/database/manage/namespace/rack-aware) cluster deployed across multiple availability zones, you can take down a full rack at a time instead of one node at a time, provided the remaining racks can maintain partition availability. #### Availability requirements The remaining racks must hold enough roster replicas to keep all partitions available while the target rack is down. In SC, a partition is available when the surviving nodes form a [majority](https://aerospike.com/docs/database/learn/architecture/clustering/consistency-modes) of the roster **and** at least one roster replica for that partition is among them. This depends on the number of racks and the [`replication-factor`](https://aerospike.com/docs/database/reference/config#namespace__replication-factor) (RF): - **3+ racks, RF >= number of racks** (for example, 3 AZs with RF=3): Every partition has a roster replica on every rack. One rack down leaves a clear majority of roster nodes, each holding a replica. All partitions remain available. No special configuration is needed. - **3+ racks, RF < number of racks** (for example, 3 AZs with RF=2): Each partition has replicas on RF distinct racks. One rack down still leaves a majority of roster nodes, and because rack-aware placement spreads each partition’s replicas across different racks, at least one roster replica for every partition survives. All partitions remain available. Do not take down more than one rack at a time. Losing two of three racks removes the majority and causes unavailable partitions. - **2 racks** (any RF): Taking down one rack leaves exactly half the roster, so there is no majority. Partitions whose roster-master is on the downed rack become unavailable (~50%). Use the [`active-rack` optimization](#optimization-with-active-rack) in the following section to maintain full availability during planned maintenance. ::: tie-breaker node deployments Some deployments add a single permanently quiesced “tie-breaker” node on a third rack to provide a majority vote for two equally sized data racks. While this topology can maintain availability when one data rack is down, it introduces significant operational complexity: the tie-breaker has a divergent configuration, can interfere with `migrate-fill-delay` behavior, may unexpectedly accept writes in small clusters, and adds network connectivity requirements across three zones instead of two. The rack-by-rack procedures on this page apply to the two data racks, but do not address tie-breaker-specific considerations. If you operate a tie-breaker deployment, see [How to ensure availability in Strong Consistency with equally sized racks?](https://support.aerospike.com/hc/en-us/articles/49938446932251-How-to-ensure-availability-in-Strong-Consistency-with-equally-sized-racks) for detailed caveats. For new deployments, prefer either three equal racks with RF=3 or two equal racks with the [`active-rack`](#optimization-with-active-rack) feature instead. ::: #### Procedure 1. [Quiesce](https://aerospike.com/docs/database/manage/cluster/quiesce-node#quiesce-a-node) all nodes in the target rack and trigger a recluster. Terminal window ```bash Admin+> manage quiesce with Admin+> manage quiesce with ... Admin+> manage recluster ``` Verify: `show statistics like quiesce` shows `effective_is_quiesced: true` on the quiesced nodes and `nodes_quiesced` equals the number of quiesced nodes on all nodes. Terminal window ```bash Admin+> show statistics like quiesce ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~test Namespace Statistics (2026-04-14 00:41:02 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Node |node1:3000|node2:3000|node3:3000|node4:3000|node5:3000|node6:3000 effective_is_quiesced|false |false |true |true |false |false nodes_quiesced |2 |2 |2 |2 |2 |2 pending_quiesce |false |false |true |true |false |false Number of rows: 4 ``` 2. Wait for the quiesce handoff. Verify no traffic or proxies reach the quiesced nodes. Terminal window ```bash Admin+> show latencies ``` Verify: ops/sec drops to zero on all quiesced nodes, `client_proxy_*` and `batch_sub_proxy_*` counters stop incrementing, and `from_proxy_*` counters stop on the remaining nodes. See the [quiesce verification reference](https://aerospike.com/docs/database/manage/cluster/quiesce-node#verify-quiesce-handoff) for full output examples. 3. **Recommended:** Dynamically change [`cluster-name`](https://aerospike.com/docs/database/reference/config#service__cluster-name) on the nodes in that rack to a different value. This ejects them from the cluster cleanly. The nodes depart on their own rather than being detected as failed, which avoids the [evade flag](https://aerospike.com/docs/database/learn/strong-consistency#dead-partitions) that would otherwise exclude them from super-majority calculations on rejoin. The static configuration file (`aerospike.conf`) retains the original `cluster-name`, so no file edits are needed. Terminal window ```bash asinfo -v 'set-config:context=service;cluster-name=maintenance-temp' -h ``` ::: tip A clean `asd` shutdown via `systemctl stop` (SIGTERM) does not set the evade flag, so this step is not strictly required when you can guarantee a clean shutdown. It is defense-in-depth: it ensures a clean departure even if the shutdown is interrupted (for example, by an OOM kill or crash). It is essential if you need to eject nodes while they are still running. ::: 4. Shut down `asd` on each node. If hosts will be rebooted, use [ASMT](https://aerospike.com/docs/database/tools/asmt) to back up the indexes of each namespace before rebooting and restore them afterward. Terminal window ```bash sudo systemctl stop aerospike # If rebooting: sudo asmt -b -v -z -p -n # ... reboot and perform maintenance ... sudo asmt -r -v -p -n ``` Verify: [Validate](#validate-partitions) that the cluster reports zero unavailable and zero dead partitions with the rack down. This confirms the remaining racks hold a majority of the roster with at least one replica for every partition. 5. Perform maintenance on the rack’s hosts. 6. Start `asd`. On startup, the node reads the original `cluster-name` from the static configuration and automatically rejoins the main cluster. Terminal window ```bash sudo systemctl start aerospike ``` Verify: `info network` shows all nodes have rejoined at the expected cluster size. Terminal window ```bash Admin> info network ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information (2026-04-14 00:48:17 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Node| Node ID| IP| Build|Migrations|~~~~~~~~~~~~~~~~~~Cluster~~~~~~~~~~~~~~~~~~|Client| Uptime | | | | |Size| Key|Integrity| Principal| Conns| node1:3000 | BB94B7AEB45DB52|10.0.3.1:3000|E-8.1.1.2| 0.000 | 6|A3F7C912DE4B|True |BB9E312FA6C9B28| 5|01:11:25 node2:3000 |*BB9D787F6BAF3D6|10.0.3.2:3000|E-8.1.1.2| 0.000 | 6|A3F7C912DE4B|True |BB9E312FA6C9B28| 5|01:11:25 node3:3000 | BB989A1BF1D8116|10.0.3.3:3000|E-8.1.1.2| 0.000 | 6|A3F7C912DE4B|True |BB9E312FA6C9B28| 5|00:00:14 node4:3000 | BB9D48DF5A70CEE|10.0.3.4:3000|E-8.1.1.2| 0.000 | 6|A3F7C912DE4B|True |BB9E312FA6C9B28| 5|00:00:14 node5:3000 | BB9A63D2E17F490|10.0.3.5:3000|E-8.1.1.2| 0.000 | 6|A3F7C912DE4B|True |BB9E312FA6C9B28| 5|01:11:25 node6:3000 | BB9E312FA6C9B28|10.0.3.6:3000|E-8.1.1.2| 0.000 | 6|A3F7C912DE4B|True |BB9E312FA6C9B28| 5|01:11:25 Number of rows: 6 ``` 7. [Validate](#validate-partitions) that the cluster has no unavailable or dead partitions and that all nodes have rejoined. 8. Wait for migrations to complete. Use [`cluster-stable`](https://aerospike.com/docs/database/reference/info#cluster-stable) to check. It returns `ERROR` while migrations are in progress and the cluster key when they are done. Run it periodically until all nodes return the same key: Terminal window ```bash Admin+> asinfo -v 'cluster-stable:size=6;ignore-migrations=false' node1:3000 (10.0.3.1) returned: A3F7C912DE4B node2:3000 (10.0.3.2) returned: A3F7C912DE4B node3:3000 (10.0.3.3) returned: A3F7C912DE4B node4:3000 (10.0.3.4) returned: A3F7C912DE4B node5:3000 (10.0.3.5) returned: A3F7C912DE4B node6:3000 (10.0.3.6) returned: A3F7C912DE4B ``` 9. Repeat for the next rack. ### Optimization with `active-rack` In Database 7.2.0 and later, the [`active-rack`](https://aerospike.com/docs/database/manage/namespace/consistency#designate-an-active-rack) feature can both ensure full availability for two equally sized racks and shorten the procedure. When `active-rack` is configured, the designated active rack holds **all** master partitions. The passive rack holds only secondaries. This means: - The quiesce step can be skipped for the passive rack. Since the passive rack has no masters, there is no master handoff needed, so taking it down does not cause a master gap. - The active rack remains fully available while the passive rack is down. ::: caution With RF=2 and two racks, the active rack has no redundancy while the passive rack is down: every partition has exactly one roster replica (the master on the active rack). If any node in the active rack fails during this window, partitions mastered on that node become unavailable until it returns. Minimize the maintenance window and monitor the active rack closely. ::: Procedure with `active-rack`: 1. Designate the rack that will stay up as [`active-rack`](https://aerospike.com/docs/database/reference/config#namespace__active-rack). In SC mode, this requires setting the config, reclustering, re-rostering (with `manage roster stage observed`), and reclustering again. See [Change active rack for SC dynamically](https://aerospike.com/docs/database/manage/namespace/consistency#change-active-rack-for-sc-dynamically). Wait for migrations to complete (masters move to the active rack). Terminal window ```bash Admin+> manage config namespace param active-rack to Admin+> manage recluster Admin+> manage roster stage observed ns Admin+> manage recluster ``` The `manage roster stage observed` command prompts for interactive confirmation because the active-rack change modifies the roster prefix (for example, from no marker to `M1`). For scripted or non-interactive use, set the roster directly with [`roster-set`](https://aerospike.com/docs/database/reference/info#roster-set), including the `M` prefix that appears in the Observed Nodes list: Terminal window ```bash asinfo -v 'roster-set:namespace=;nodes=M|@,@,...' asadm ... -e "manage recluster" ``` Verify: `asinfo -v 'cluster-stable:size=N;ignore-migrations=false'` returns the same cluster key on all nodes (migrations complete). Use `show pmap` to confirm all Primary partitions are on the active rack’s nodes. 2. **Recommended:** Dynamically change [`cluster-name`](https://aerospike.com/docs/database/reference/config#service__cluster-name) on the passive rack’s nodes to eject them from the cluster. The static config retains the original `cluster-name`. See the tip about when this step is essential in the [rack procedure](#procedure). Terminal window ```bash asinfo -v 'set-config:context=service;cluster-name=maintenance-temp' -h ``` 3. Shut down `asd`. If hosts will be rebooted, use [ASMT](https://aerospike.com/docs/database/tools/asmt) to back up and later restore the indexes of each namespace. Terminal window ```bash sudo systemctl stop aerospike # If rebooting: sudo asmt -b -v -z -p -n # ... reboot and perform maintenance ... sudo asmt -r -v -p -n ``` 4. Perform maintenance on the passive rack’s hosts. 5. Start `asd`. The node reads the original `cluster-name` from static config and rejoins automatically. Terminal window ```bash sudo systemctl start aerospike ``` Verify: `info network` shows all nodes have rejoined at the expected cluster size. Terminal window ```bash Admin> info network ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information (2026-04-14 01:12:45 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Node| Node ID| IP| Build|Migrations|~~~~~~~~~~~~~~~~~~Cluster~~~~~~~~~~~~~~~~~~|Client| Uptime | | | | |Size| Key|Integrity| Principal| Conns| node1:3000 | BB94B7AEB45DB52|10.0.3.1:3000|E-8.1.1.2| 0.000 | 4|D74E19A3B82C|True |BB9D48DF5A70CEE| 5|01:35:53 node2:3000 |*BB9D787F6BAF3D6|10.0.3.2:3000|E-8.1.1.2| 0.000 | 4|D74E19A3B82C|True |BB9D48DF5A70CEE| 5|01:35:53 node3:3000 | BB989A1BF1D8116|10.0.3.3:3000|E-8.1.1.2| 0.000 | 4|D74E19A3B82C|True |BB9D48DF5A70CEE| 5|00:00:14 node4:3000 | BB9D48DF5A70CEE|10.0.3.4:3000|E-8.1.1.2| 0.000 | 4|D74E19A3B82C|True |BB9D48DF5A70CEE| 5|00:00:14 Number of rows: 4 ``` 6. [Validate](#validate-partitions) that the cluster has no unavailable or dead partitions and that all nodes have rejoined. 7. Wait for migrations to complete before switching `active-rack`. Use [`cluster-stable`](https://aerospike.com/docs/database/reference/info#cluster-stable) to check. It returns `ERROR` while migrations are in progress and the cluster key when they are done. Run it periodically until all nodes return the same key: Terminal window ```bash Admin+> asinfo -v 'cluster-stable:size=4;ignore-migrations=false' node1:3000 (10.0.3.1) returned: D74E19A3B82C node2:3000 (10.0.3.2) returned: D74E19A3B82C node3:3000 (10.0.3.3) returned: D74E19A3B82C node4:3000 (10.0.3.4) returned: D74E19A3B82C ``` 8. Switch `active-rack` to point to the now-maintained rack (repeat the set config / recluster / re-roster / recluster sequence). Wait for migrations to complete. 9. Repeat steps 2-8 for the other rack. 10. After both racks are maintained, disable `active-rack` to restore normal balanced partition distribution. Terminal window ```bash Admin+> manage config namespace param active-rack to 0 Admin+> manage recluster Admin+> manage roster stage observed ns Admin+> manage recluster ``` Verify: `show pmap` shows Primary and Secondary partitions evenly distributed across all nodes. ::: note `active-rack` only helps with **planned** maintenance of the passive rack. If the designated active rack itself goes down unexpectedly, the surviving passive rack becomes unavailable until `active-rack` is dynamically switched and the roster is updated to use the remaining nodes. ::: ## Validate partitions When you validate partitions, each node reports the global number of dead or unavailable partitions. For example, if the entire cluster has determined that 100 partitions are unavailable, all of the current nodes report 100 unavailable partitions. Use `show pmap` to display the partition map. The `Unavailable` and `Dead` columns should be 0 for each node. Terminal window ```bash Admin> show pmap ~~~~~~~~~~~~~~~~~~~Partition Map Analysis~~~~~~~~~~~~~~~~~~ Namespace| Node| Cluster Key|~~~~~~~~~~~~Partitions~~~~~~~~~~~~ | | |Primary|Secondary|Unavailable|Dead test |node1:3000|A1D7BBA0D9EF| 1024| 1024| 0| 0 test |node2:3000|A1D7BBA0D9EF| 1024| 1024| 0| 0 test |node3:3000|A1D7BBA0D9EF| 1024| 1024| 0| 0 test |node4:3000|A1D7BBA0D9EF| 1024| 1024| 0| 0 test | | | 4096| 4096| 0| 0 Number of rows: 4 ``` ### Revive dead partitions You may wish to use your namespace in spite of potentially missing data. For example, you may have entered a maintenance state where you have disabled application use, and are preparing to reapply data from a reliable message queue or other source. 1. Identify dead partitions with `show pmap`. Terminal window ```bash Admin> show pmap ~~~~~~~~~~~~~~~~~~~Partition Map Analysis~~~~~~~~~~~~~~~~~~ Namespace| Node| Cluster Key|~~~~~~~~~~~~Partitions~~~~~~~~~~~~ | | |Primary|Secondary|Unavailable|Dead test |node1:3000|A1D7BBA0D9EF| 915| 915| 0| 264 test |node2:3000|A1D7BBA0D9EF| 915| 915| 0| 264 test |node3:3000|A1D7BBA0D9EF| 915| 915| 0| 264 test |node4:3000|A1D7BBA0D9EF| 915| 915| 0| 264 test | | | 3660| 3660| 0| 264 Number of rows: 4 ``` 2. Run `revive` to acknowledge the potential data loss on each server. Terminal window ```bash Admin+> manage revive ns test ~~~Revive Namespace Partitions~~~ Node|Response node1:3000 |ok node2:3000 |ok node3:3000 |ok node4:3000 |ok Number of rows: 4 ``` 3. Run `recluster` to revive the dead partitions. Terminal window ```bash Admin+> manage recluster Successfully started recluster ``` 4. Verify that there are no longer any dead partitions with `show pmap`. Terminal window ```bash Admin> show pmap ~~~~~~~~~~~~~~~~~~~Partition Map Analysis~~~~~~~~~~~~~~~~~~ Namespace| Node| Cluster Key|~~~~~~~~~~~~Partitions~~~~~~~~~~~~ | | |Primary|Secondary|Unavailable|Dead test |node1:3000|A1D7BBA0D9EF| 1024| 1024| 0| 0 test |node2:3000|A1D7BBA0D9EF| 1024| 1024| 0| 0 test |node3:3000|A1D7BBA0D9EF| 1024| 1024| 0| 0 test |node4:3000|A1D7BBA0D9EF| 1024| 1024| 0| 0 test | | | 4096| 4096| 0| 0 Number of rows: 4 ```