# Cluster consistency This page describes how to manage the cluster nodes of a [strong consistency](https://aerospike.com/docs/database/learn/strong-consistency) (SC) namespace in the Aerospike Database. ## Add nodes to the cluster and roster \[Use the `asadm` [manage roster & show roster](https://aerospike.com/docs/database/tools/asadm/live-mode#roster) commands. Otherwise, use the equivalent [asinfo - roster](https://aerospike.com/docs/database/reference/info#roster) and [asinfo - roster-set](https://aerospike.com/docs/database/reference/info#roster-set) commands. 1. Install and configure Aerospike on the new nodes as described in [Configure strong consistency](https://aerospike.com/docs/database/manage/namespace/consistency). 2. When the nodes have joined the cluster, use the following command to verify that the result in [`cluster_size`](https://aerospike.com/docs/database/reference/metrics#node_stats__cluster_size) is greater than the result in `ns_cluster_size`. This is visible in the following output where `cluster_size` shows 6 rows, while ns\_cluster\_size shows 5 rows. Terminal window ```bash Admin> show stat -flip like cluster_size ~Service Statistics (2021-10-22 23:43:35 UTC)~ Node|cluster_size node1.aerospike.com:3000| 6 node2.aerospike.com:3000| 6 node4.aerospike.com:3000| 6 node5.aerospike.com:3000| 6 node6.aerospike.com:3000| 6 node7.aerospike.com:3000| 6 Number of rows: 6 ~test Namespace Statistics (2021-10-22 23:43:35 UTC)~ Node|ns_cluster_size node1.aerospike.com:3000| 5 node2.aerospike.com:3000| 5 node4.aerospike.com:3000| 5 node5.aerospike.com:3000| 5 node6.aerospike.com:3000| 5 node7.aerospike.com:3000| 5 Number of rows: 6 ``` 3. Use `show roster` to see the newly observed nodes in its Observed Nodes section. 4. Use the following command to copy the `Observed Nodes` list into the `Pending Roster`. Terminal window ```bash Admin> enable Admin+> manage roster stage observed ns test Pending roster now contains observed nodes. Run "manage recluster" for your changes to take affect. ``` 5. Activate the new roster with the `manage recluster` command. Terminal window ```bash Admin+> manage recluster Successfully started recluster ``` 6. Run `show roster` to confirm that the roster has been updated on all nodes. Verify that the service’s `cluster_size` matches the namespace’s `ns_cluster_size`. ## Remove nodes and update the roster This section describes how to remove a node from an existing namespace configured with SC. ::: caution Do not simultaneously remove a number of nodes equal to or greater than your [replication-factor](https://aerospike.com/docs/database/reference/config#namespace__replication-factor) (RF). Removing too many nodes simultaneously may result in [unavailable partitions](https://aerospike.com/docs/database/learn/strong-consistency#dead-partitions), meaning some data is unavailable to the application. If you observe unavailable partitions, re-add the nodes and wait for the cluster to synchronize before proceeding. Namespaces with `replication-factor` set to 1 (RF1) have some partitions unavailable whenever **any** node leaves the cluster, making it impractical to perform a rolling restart or upgrade. Strong consistency implies the guarantee that with RF N, N copies of data are written to the cluster. A fully formed cluster must contain X nodes where X >= N to satisfy this. Creating a cluster where X = N means that all partitions become unavailable during a single node shut down. ::: ### Remove node from the cluster 1. Run `show roster`. Verify all roster nodes are present in the cluster. Terminal window ```bash Admin+> pager on Admin+> show roster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2021-10-23 00:08:54 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Node | Node ID | Namespace| Current Roster | Pending Roster | Observed Nodes node1.aerospike.com:3000 |BB9070016AE4202 | test |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 node3.aerospike.com:3000 |BB9060016AE4202 | test |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 node4.aerospike.com:3000 |BB9050016AE4202 | test |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 node5.aerospike.com:3000 |BB9040016AE4202 | test |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 node6.aerospike.com:3000 |BB9010016AE4202 | test |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 node7.aerospike.com:3000 |*BB9020016AE4202| test |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 Number of rows: 6 ``` 2. Shut down safely the nodes to be removed with the following command (in this example on `node1.aerospike.com:3000` with node id `BB9070016AE4202`). Verify you are removing fewer nodes that your configured RF. ::: note Any operational procedure that executes a `SIGTERM` is safe. After startup, a `SIGTERM` flushes data to disk and properly signal other servers. ::: Terminal window ```shell systemctl stop aerospike ``` 3. When migrations are complete, run the `stat` command on `partitions_remaining` until the `migrate_partitions_remaining` stat becomes zero on all nodes. Terminal window ```bash Admin> show stat service like partitions_remaining -flip ~~~~~Service Statistics (2021-10-23 00:25:10 UTC)~~~~ Node|migrate_partitions_remaining node2.aerospike.com:3000| 0 node4.aerospike.com:3000| 0 node5.aerospike.com:3000| 0 node6.aerospike.com:3000| 0 node7.aerospike.com:3000| 0 Number of rows: 5 ``` ### Remove node from the roster 1. Run `show roster` to verify that BB9070016AE4202 is removed from the pending roster. In the following example there is one fewer node in the `Observed Nodes` column than in `Current Roster` and `Pending Roster`. Terminal window ```bash Admin+> pager on Admin+> show roster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2021-10-23 00:26:56 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Node| Node ID|Namespace| Current Roster| Pending Roster| Observed Nodes node1.aerospike.com:3000|BB9070016AE4202 |test |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 node3.aerospike.com:3000|BB9060016AE4202 |test |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 node4.aerospike.com:3000|BB9050016AE4202 |test |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 node5.aerospike.com:3000|BB9040016AE4202 |test |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 node6.aerospike.com:3000|BB9010016AE4202 |test |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 node7.aerospike.com:3000|*BB9020016AE4202|test |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 Number of rows: 6 ``` 2. Copy the `Observed Nodes` list into the `Pending Roster`. Terminal window ```bash Admin+> manage roster stage observed ns test Pending roster now contains observed nodes. Run "manage recluster" for your changes to take affect. ``` 3. Run `manage recluster` to apply the change. Terminal window ```bash Admin+> manage recluster Successfully started recluster ``` 4. Check if startup is complete. Run the following command periodically until it returns `ok`. Terminal window ```bash asinfo -h [ip of host] -v 'status' ``` ::: caution Avoid executing a process shut down such as restarting during startup. For more information, see [Dead partitions](https://aerospike.com/docs/database/learn/strong-consistency#dead-partitions). ::: ## Validate partitions When you validate partitions, each node reports the global number of dead or unavailable partitions. For example, if the entire cluster has determined that 100 partitions are unavailable, all of the current nodes report 100 unavailable partitions. 1. Use the following command to display which nodes report dead or unavailable partitions. ```plaintext show stat namespace for like 'unavailable|dead' -flip ``` 1. In the output of this command, the columns `unavailable` and `dead` should be 0 for each node. Terminal window ```bash Admin> show stat namespace for test like 'unavailable|dead' -flip ~~~~~~test Namespace Statistics (2021-10-23 00:36:43 UTC)~~~~~~~ Node|dead_partitions|unavailable_partitions node1.aerospike.com:30000| 0| 0 node2.aerospike.com:30000| 0| 0 node4.aerospike.com:30000| 0| 0 node5.aerospike.com:30000| 0| 0 node6.aerospike.com:30000| 0| 0 Number of rows: 5 ``` ### Revive dead partitions You may wish to use your namespace in spite of potentially missing data. For example, you may have entered a maintenance state where you have disabled application use, and are preparing to reapply data from a reliable message queue or other source. 1. Identify dead partitions. Terminal window ```bash Admin> show stat namespace for test like dead -flip ~test Namespace Statistics (2021-10-23 00:38:41 UTC)~ Node|dead_partitions node1.aerospike.com:3000| 264 node2.aerospike.com:3000| 264 node4.aerospike.com:3000| 264 node5.aerospike.com:3000| 264 node6.aerospike.com:3000| 264 Number of rows: 5 ``` 2. Run `revive` to acknowledge the potential data loss on each server. Terminal window ```bash Admin+> manage revive ns test ~~~Revive Namespace Partitions~~~ Node|Response node1.aerospike.com:3000|ok node2.aerospike.com:3000|ok node4.aerospike.com:3000|ok node5.aerospike.com:3000|ok node6.aerospike.com:3000|ok Number of rows: 5 ``` 3. Run `recluster` to revive the dead partitions. Terminal window ```bash Admin+> manage recluster Successfully started recluster ``` 4. Verify that there are no longer any dead partitions with the `dead_partitions` metric. Terminal window ```bash Admin> show stat namespace for test like dead -flip ~test Namespace Statistics (2021-10-23 00:40:41 UTC)~ Node|dead_partitions node1.aerospike.com:3000| 0 node2.aerospike.com:3000| 0 node4.aerospike.com:3000| 0 node5.aerospike.com:3000| 0 node6.aerospike.com:3000| 0 Number of rows: 5 ```