Skip to main content
We are excited to be a part of AWS re:Invent 2024. Visit us at booth #1844 in Las Vegas.More info
Loading

Upgrading to Aerospike Database 6.4 or later

Overview​

This page describes special considerations when you upgrade to Database 6.4.

caution

A Database 5.x instance cannot upgrade directly to 6.4.

Important pre-upgrade considerations​

tip

Always review the special upgrade instructions of the interim database releases between the current version of your cluster and the target version.

  • Before you upgrade from Database 4.9 or 5.x, follow the instructions in Storage Format Upgrade in 6.0 Release.
  • Back up /opt/aerospike/smd/security.smd in your system metadata (SMD) Directory Structure and aerospike.conf, the Aerospike configuration file.
  • Find and fix any secondary index (sindex) names that are longer than 64 characters. See Upgrade to Database 6.1.
  • If you're an Enterprise Edition customer, ensure that your feature-key file has the explicit feature keys needed in Database 6.3 and later.

Upgrade path from Database 5.x​

A cluster of Aerospike Database 5 serving secondary index queries must pass through one of versions 6.0, 6.1, 6.2, or 6.3 before upgrading to Database 6.4 or later.

See the Minimum Usable Client Version section of the Client Matrix to verify that the client you use is compatible with the database version you are targeting.

Special upgrade instructions for Aerospike Database 6.3​

  • If you do not have single-bin namespaces in your cluster, you can upgrade to Database 6.4 with a regular rolling upgrade.

  • If you have single-bin name spaces, use the following steps to move them from single-bin to multi-bin by doing a special rolling upgrade. This applies to systems moving from Database 6.3 or earlier to Database 6.4.0.18 or later.

  • You can upgrade while single-bin apps continue to run.

caution

You must follow these instructions exactly as written, in the order described below. Aerospike customers should not hesitate to contact Aerospike support for a review of these upgrade instructions.

Step 1: Perform a rolling upgrade of the cluster to version 6.4.0.18 or later​

On each cluster node:

  1. Take down the Aerospike daemon (asd) on the node as you usually would (quiesce, etc).

  2. Install Database 6.4 from RPM or Debian package. Verify that you’re using release 6.4.0.18 or later.

  3. Before starting up asd, modify the single-bin namespace configuration as follows:

  • remove single-bin true
  • add convert-single-bin true

  1. Wipe any storage devices and files associated with the single-bin namespaces you are converting. This is enforced by asd, where convert-single-bin true prevents asd from starting unless namespace storage is wiped. For storage-engine memory, of course, this does not apply.

  2. Verify that all nodes in the cluster are running 6.4.0.18 before advancing to the next step.

Step 2: Dynamically set convert-single-bin to false on all the nodes.​

  1. In asadm:

    Admin> enable --warn
    Admin+> manage config namespace nssinglebin param convert-single-bin to false
    ~Set Namespace Param convert-single-bin to false~
            Node|Response
    10.0.0.1:3000|ok      
    10.0.0.2:3000|ok      
    10.0.0.3:3000|ok          
    Number of rows: 3

  2. Verify that the convert-single-bin config is false on all nodes.

    Admin> show config namespace for nssinglebin like convert-single-bin
    ~~~~~~~~~~~~~~nssinglebin Namespace Configuration (2024-04-27 00:39:22 UTC)~~~~~~~~~~~~~~
    Node              |10.0.0.1:3000
    convert-single-bin|false      
    Node              |10.0.0.2:3000
    convert-single-bin|false
    Node              |10.0.0.3:3000
    convert-single-bin|false

Step 3: Wait at least 60 seconds​

Wait at least 60 seconds before advancing to the next step. This is enforced to ensure that Step 4 is not inadvertently started before Step 2 is complete.

Step 4: Dynamically set check-single-bin to false​

check-single-bin is a dynamic-only config that starts as true when convert-single-bin is true. At the end of this step:

  • both check-single-bin and convert-single-bin will be false

  • an internal flag named was_single_bin, which also starts as true if convert-single-bin is true, will still be true.

    In asadm:

    Admin> enable --warn
    Admin+> manage config namespace nssinglebin param check-single-bin to false
    ~Set Namespace Param check-single-bin to false~
          Node|Response
    10.0.0.1:3000|ok      
    10.0.0.2:3000|ok      
    10.0.0.3:3000|ok          
    Number of rows: 3

  1. Verify that the configuration is false on all nodes.

    Admin> show config namespace for nssinglebin like check-single-bin
    ~~~~~~~~~~~~~~nssinglebin Namespace Configuration (2024-04-27 00:43:01 UTC)~~~~~~~~~~~~~~
    Node              |10.0.0.1:3000
    check-single-bin|false      
    Node              |10.0.0.2:3000
    check-single-bin|false
    Node              |10.0.0.3:3000
    check-single-bin|false

    Both check-single-bin and convert-single-bin are now false. However, an internal flag named was_single_bin, which starts as true if convert-single-bin is true, remains true.

Step 5: Perform a rolling restart​

Perform a rolling restart of the cluster without convert-single-bin true.

On each cluster node:

  1. Shut down asd on the node as usual (quiesce, etc).

  2. Before restarting asd, modify the namespace configuration and remove convert-single-bin true.

  3. After restarting the cluster nodes, verify that the was_single_bin flag is false on all nodes.

tip

You are now done with converting your single-bin namespace to multi-bin.

Various node states during upgrade​

As an upgrade progresses, nodes go through the following sequence of states.

danger

A node in any of these states can only “mix” with nodes in an adjacent state.

State (a)​

  • All nodes start here before Step 1.
  • Database 6.3 or previous, with the single-bin namespace.
  • Can only mix with other nodes in State (a) or State (b).

State (b)​

Environment​

  • Database 6.4 with all namespaces defined as multi-bin.
  • convert-single-bin configuration parameter set to true
  • check-single-bin configuration parameter set to true
  • was_single_bin field is true

Results​

  • Converts all outbound fabric messages to single-bin.
  • Converts inbound fabric messages from single-bin to multi-bin unless they have a special extra ‘multi-bin’ message field, indicating they do not need conversion.
  • Can only mix with other nodes in State (a), State (b), or State (c).

All nodes should be here after Step 1.

State (c)​

Environment​

  • convert-single-bin configuration parameter set to false
  • check-single-bin configuration parameter set to true
  • was_single_bin field is true

Results​

  • Database 6.4, stores data as multi-bin.
  • Sends all outbound fabric messages as multi-bin with a special extra ‘multi-bin’ message field.
  • Converts inbound fabric messages from single-bin to multi-bin unless they have a special extra ‘multi-bin’ message field, indicating they do not need conversion.
  • Can only mix with other nodes in State (b), State (c), or State (d).

All nodes should be here after Step 2.

State (d)​

Environment​

  • convert-single-bin configuration parameter set to false
  • check-single-bin configuration parameter set to false
  • was_single_bin field is true

Results​

  • Database 6.4, stores data as multi-bin.
  • Sends all outbound fabric messages as multi-bin with a special extra ‘multi-bin’ message field.
  • Assumes all inbound fabric messages are multi-bin.
  • Can only mix with other nodes in this State (c), State (d), or State (e).

All nodes should be here after Step 4.

State (e)​

Environment​

  • convert-single-bin configuration parameter set to false
  • check-single-bin configuration parameter set to false
  • was_single_bin field is false

Results​

  • Database 6.4 shows ordinary multi-bin behavior.
  • Stores data as multi-bin.
  • Sends and receives everything as multi-bin. No special extra ‘multi-bin’ message field sent, and the field is ignored if received.
  • Can only mix with other nodes in State (d) or State (e).

All nodes should be here after Step 5.

Troubleshooting​

Database 6.4 nodes can only start up in state (b) – with convert-single-bin true, or state (e) – without convert-single-bin. As a result:

  • If a node running Database 6.4 goes down while other nodes are in states (a), (b), and/or (c), it must be restarted in state (b). In other words, start this node over at Step 1.

  • If a node running Database 6.4 goes down after any other node has been moved to state (d), it must be restarted in state (e), but only after all the other nodes have also been moved to state (d). In other words, start this node up directly into Step 5 after the other nodes have finished Step 4.

caution

If nodes in non-adjacent states are “mixed” in a cluster, the resulting behavior is undefined. Most fabric messages exchanged between such nodes will generate a variety of log warnings and be rejected by the receiver, but it is not impossible that incompatible-format records get through and are stored, later causing issues when they are read or operated on.

Characteristics of a migrated, previously single-bin namespace​

Once a previously single-bin namespace has been migrated to a new Aerospike cluster with nodes at Database 6.4 or later, it has the following characteristics.

  • The migrated records will have an empty string bin name. The bin can be accessed using the bin name "".
  • Applications that worked with single-bin continue to work.
  • New bins can be added to the records in these namespaces, with a unique bin name.
  • Depending on the data type of the single bin, the converted namespace may consume more storage. See the capacity planning guide for more information.
  • Secondary indexes cannot be created on the empty string bin name.
  • You cannot choose XDR bin shipping while the namespace records have the empty-string bin name. The bin-policy must be all (the default setting).

Changes and updates​

For information about new features, breaking changes, and late breaking news, see the Database 6.4 release notes.