Cluster consistency

This page describes how to manage the cluster nodes of a strong consistency (SC) namespace in the Aerospike Database.

Add nodes to the cluster and roster

[Use the asadm manage roster & show roster commands. Otherwise, use the equivalent asinfo - roster and asinfo - roster-set commands.

1. Install and configure Aerospike on the new nodes as described in Configure strong consistency.

2. When the nodes have joined the cluster, use the following command to verify that the result in cluster_size is greater than the result in ns_cluster_size. This is visible in the following output where cluster_size shows 6 rows, while ns_cluster_size shows 5 rows.

   Admin> show stat -flip like cluster_size
   ~Service Statistics (2021-10-22 23:43:35 UTC)~
                       Node|cluster_size
   node1.aerospike.com:3000|           6
   node2.aerospike.com:3000|           6
   node4.aerospike.com:3000|           6
   node5.aerospike.com:3000|           6
   node6.aerospike.com:3000|           6
   node7.aerospike.com:3000|           6
   Number of rows: 6
   
   ~test Namespace Statistics (2021-10-22 23:43:35 UTC)~
                       Node|ns_cluster_size
   node1.aerospike.com:3000|              5
   node2.aerospike.com:3000|              5
   node4.aerospike.com:3000|              5
   node5.aerospike.com:3000|              5
   node6.aerospike.com:3000|              5
   node7.aerospike.com:3000|              5
   Number of rows: 6

3. Use show roster to see the newly observed nodes in its Observed Nodes section.

4. Use the following command to copy the Observed Nodes list into the Pending Roster.

   Admin> enable
   Admin+> manage roster stage observed ns test
   Pending roster now contains observed nodes.
   Run "manage recluster" for your changes to take affect.

5. Activate the new roster with the manage recluster command.

   Admin+> manage recluster
   Successfully started recluster

6. Run show roster to confirm that the roster has been updated on all nodes. Verify that the service’s cluster_size matches the namespace’s ns_cluster_size.

Remove nodes and update the roster

This section describes how to remove a node from an existing namespace configured with SC.

Caution

Do not simultaneously remove a number of nodes equal to or greater than your replication-factor (RF). Removing too many nodes simultaneously may result in unavailable partitions, meaning some data is unavailable to the application. If you observe unavailable partitions, re-add the nodes and wait for the cluster to synchronize before proceeding.

Namespaces with replication-factor set to 1 (RF1) have some partitions unavailable whenever any node leaves the cluster, making it impractical to perform a rolling restart or upgrade.

Strong consistency implies the guarantee that with RF N, N copies of data are written to the cluster. A fully formed cluster must contain X nodes where X >= N to satisfy this. Creating a cluster where X = N means that all partitions become unavailable during a single node shut down.

Remove node from the cluster

1. Run show roster. Verify all roster nodes are present in the cluster.

   Admin+> pager on
   Admin+> show roster
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2021-10-23 00:08:54 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            Node            |  Node ID       | Namespace|     Current Roster                                                                           |    Pending Roster                                                                             |                Observed Nodes
   node1.aerospike.com:3000 |BB9070016AE4202 |  test  |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
   node3.aerospike.com:3000 |BB9060016AE4202 |  test  |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
   node4.aerospike.com:3000 |BB9050016AE4202 |  test  |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
   node5.aerospike.com:3000 |BB9040016AE4202 |  test  |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
   node6.aerospike.com:3000 |BB9010016AE4202 |  test  |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
   node7.aerospike.com:3000 |*BB9020016AE4202|  test  |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
   Number of rows: 6

2. Shut down safely the nodes to be removed with the following command (in this example on node1.aerospike.com:3000 with node id BB9070016AE4202). Verify you are removing fewer nodes that your configured RF.

   Note

   Any operational procedure that executes a SIGTERM is safe. After startup, a SIGTERM flushes data to disk and properly signal other servers.

   systemctl stop aerospike

3. When migrations are complete, run the stat command on partitions_remaining until the migrate_partitions_remaining stat becomes zero on all nodes.

   Admin> show stat service like partitions_remaining -flip
   ~~~~~Service Statistics (2021-10-23 00:25:10 UTC)~~~~
                       Node|migrate_partitions_remaining
   node2.aerospike.com:3000|                           0
   node4.aerospike.com:3000|                           0
   node5.aerospike.com:3000|                           0
   node6.aerospike.com:3000|                           0
   node7.aerospike.com:3000|                           0
   Number of rows: 5

Remove node from the roster

1. Run show roster to verify that BB9070016AE4202 is removed from the pending roster. In the following example there is one fewer node in the Observed Nodes column than in Current Roster and Pending Roster.

   Admin+> pager on
   Admin+> show roster
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2021-10-23 00:26:56 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                       Node|         Node ID|Namespace|                                                                                 Current Roster|                                                                                 Pending Roster|                                                                 Observed Nodes
   node1.aerospike.com:3000|BB9070016AE4202 |test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
   node3.aerospike.com:3000|BB9060016AE4202 |test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
   node4.aerospike.com:3000|BB9050016AE4202 |test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
   node5.aerospike.com:3000|BB9040016AE4202 |test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
   node6.aerospike.com:3000|BB9010016AE4202 |test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
   node7.aerospike.com:3000|*BB9020016AE4202|test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
   Number of rows: 6

2. Copy the Observed Nodes list into the Pending Roster.

   Admin+> manage roster stage observed ns test
   Pending roster now contains observed nodes.
   Run "manage recluster" for your changes to take affect.

3. Run manage recluster to apply the change.

   Admin+> manage recluster
   Successfully started recluster

4. Check if startup is complete. Run the following command periodically until it returns ok.

   asinfo -h [ip of host] -v 'status'

Caution

Avoid executing a process shut down such as restarting during startup. For more information, see Dead partitions.

Validate partitions

When you validate partitions, each node reports the global number of dead or unavailable partitions. For example, if the entire cluster has determined that 100 partitions are unavailable, all of the current nodes report 100 unavailable partitions.

1. Use the following command to display which nodes report dead or unavailable partitions.

show stat namespace for <ns> like 'unavailable|dead' -flip

1. In the output of this command, the columns unavailable and dead should be 0 for each node.

   Admin> show stat namespace for test like 'unavailable|dead' -flip
   ~~~~~~test Namespace Statistics (2021-10-23 00:36:43 UTC)~~~~~~~
                       Node|dead_partitions|unavailable_partitions
   node1.aerospike.com:30000|              0|                     0
   node2.aerospike.com:30000|              0|                     0
   node4.aerospike.com:30000|              0|                     0
   node5.aerospike.com:30000|              0|                     0
   node6.aerospike.com:30000|              0|                     0
   Number of rows: 5

Revive dead partitions

You may wish to use your namespace in spite of potentially missing data. For example, you may have entered a maintenance state where you have disabled application use, and are preparing to reapply data from a reliable message queue or other source.

1. Identify dead partitions.

   Admin> show stat namespace for test like dead -flip
   ~test Namespace Statistics (2021-10-23 00:38:41 UTC)~
                       Node|dead_partitions
   node1.aerospike.com:3000|            264
   node2.aerospike.com:3000|            264
   node4.aerospike.com:3000|            264
   node5.aerospike.com:3000|            264
   node6.aerospike.com:3000|            264
   Number of rows: 5

2. Run revive to acknowledge the potential data loss on each server.

   Admin+> manage revive ns test
   ~~~Revive Namespace Partitions~~~
                       Node|Response
   node1.aerospike.com:3000|ok
   node2.aerospike.com:3000|ok
   node4.aerospike.com:3000|ok
   node5.aerospike.com:3000|ok
   node6.aerospike.com:3000|ok
   Number of rows: 5

3. Run recluster to revive the dead partitions.

   Admin+> manage recluster
   Successfully started recluster

4. Verify that there are no longer any dead partitions with the dead_partitions metric.

   Admin> show stat namespace for test like dead -flip
   ~test Namespace Statistics (2021-10-23 00:40:41 UTC)~
                       Node|dead_partitions
   node1.aerospike.com:3000|            0
   node2.aerospike.com:3000|            0
   node4.aerospike.com:3000|            0
   node5.aerospike.com:3000|            0
   node6.aerospike.com:3000|            0
   Number of rows: 5