Upgrade to Database 6.0.0

This guide describes how to upgrade to Aerospike Database 6.0.0. It covers the new internal storage format introduced in this version, record size validation required before upgrading, and the per-node upgrade procedure.

Applies to

• Aerospike Database 6.0.0, when upgrading from Database 4.9.0 or 5.x.y.z

Prerequisites

• A running Aerospike cluster on Database 4.9.0 or 5.x.y.z
• We recommend that you take a backup of your data if you are upgrading the cluster nodes in place through a rolling upgrade.
• A copy of your current aerospike.conf configuration file

In this guide

• New storage format — what changed in the internal storage format and why devices must be wiped before upgrading
• Validate record sizes before upgrading — how to identify and fix records that may exceed write-block-size after the 4-byte overhead increase
• Upgrade steps — the per-node rolling upgrade procedure
• Post upgrade — privilege changes and post-upgrade configuration tuning
• Configuration parameter changes — parameters renamed or removed in Database 6.0.0

New storage format

In Database 6.0.0, the Aerospike Database internal storage format changed to include a four-byte record end mark. This change addresses a potential, but unlikely, data-loss condition where a partial write block persisted, causing a record to be corrupted during an unclean shutdown.

When upgrading from Database 5.x.y.z or 4.9.0, each persisted namespace storage device, except PMem-backed namespaces, must be erased and its Aerospike header wiped (zeroized). The header is stored in the first 8MiB of the device.

If you are upgrading from Database 5.x.y.z to 6.x.y.z and there is already data on the device:

• If the device supports RZAT (Read Zero After Trim), use blkdiscard -z or a similar command on the 8MiB header to wipe it, then perform a TRIM of the entire device with the appropriate command for your device and operating system.
• If the device does not support RZAT, you must wipe the entire device.

If you are upgrading from a version earlier than Database 5.7.0 you must upgrade the cluster to Database 5.7.0 before proceeding to Database 6.x.

For details on RZAT and TRIM hardware support, see the Wikipedia article on TRIM implementation.

PMem-backed namespaces do not need to be zeroized because they do not require the use of an end mark.

Note

Database 6.0.0 and pre-6.0.0 are cluster-compatible, so a rolling restart upgrade with mixed versions is supported. Keep a copy of your original aerospike.conf configuration file in case you need to downgrade. When downgrading, the extra four bytes are retained and have no effect other than the slightly larger record size.

Caution

The record overhead increased by 4 bytes in Database 6.0.0, so records that previously met the write-block-size or max-record-size limit might exceed that limit after upgrading. Exceeding write-block-size causes replication and migration issues. Exceeding max-record-size causes writes to the primary to fail that would have previously succeeded. Check your object size histogram to identify records near the write-block-size limit.

See Validate record sizes before upgrading for details.

Validate record sizes before upgrading

Database 6.0.0 adds a 4-byte per-record end mark to the internal storage format. This overhead increase can cause records near the configured write-block-size limit to exceed it after upgrade, leading to write failures and stuck migrations. Follow these steps to assess your risk and take corrective action if needed.

Review record size distribution

Run the following command in asadm to view the bytewise object size distribution for each namespace:

asadm bytewise object size distribution

Admin> show distribution object_size -b

Compare the reported record sizes against your configured write-block-size.

• If most records are well below write-block-size, skip to Proceed with the upgrade.
• If any records are close to or equal to write-block-size, continue to the following section.

You can also query the object-size and object-size-linear histograms directly for finer-grained analysis.

Note

Object-size histograms report compressed sizes when compression is enabled. Records that appear safe in the histogram can still exceed write-block-size after the 4-byte overhead increase if compression ratios vary. Account for compression when evaluating record sizes.

Upgrade to Database 5.7.0 first (if applicable)

If you are upgrading from a version earlier than Database 5.7.0 you must upgrade the cluster to Database 5.7.0 before proceeding to Database 6.x.

Database 5.7.0 introduced max-record-size, which provides a configurable threshold to reject writes that would exceed a specified size. This configuration and the associated scan tooling are required for thorough record-size validation.

If you are already running Database 5.7.0 or later, skip this section.

Identify records near the size limit

On Database 5.7.0 or later, use record size scan tools to find records that will exceed the effective write-block-size limit after the 4-byte overhead increase.

An example Python script using the Aerospike Python client is available at aerospike-examples/6.0-record-size-checker. This script handles uncompressed records and includes a configurable compression ratio variance threshold to identify compressed records that will exceed the write-block-size.

Run the record size checker in dry-run mode

python3 main.py

Example output

2023-08-02 17:44:53,097 [INFO]: Scanning node: BB9C00F800A0142
2023-08-02 17:44:53,098 [INFO]: Node BB9C00F800A0142 does not have compression enabled.
2023-08-02 17:44:53,098 [INFO]: Checking for records of compressed size larger than 1048560 bytes
2023-08-02 17:46:59,366 [INFO]: Namespace: bar, Set: testset, Primary Key: None, Digest: fe0f17700e1b7fcc82401b535f7933667634f8bf
2023-08-02 17:46:59,366 [INFO]: Namespace: bar, Set: testset, Primary Key: None, Digest: fe0f14b652ecbaf4afc46de605d7a4a0b6452f3a
...
2023-08-02 17:46:59,366 [INFO]: Node: BB9C00F800A0142 Returned Record Count: 328633
2023-08-02 17:46:59,366 [INFO]: Node: BB93E00800A0142 Returned Record Count: 330107
2023-08-02 17:46:59,366 [INFO]: Node: BB93F00800A0142 Returned Record Count: 329541

• If no records are near the write-block-size limit, skip to Proceed with the upgrade.
• If problematic records are found, continue to the following section.

Adjust configuration

If records are close to or at the write-block-size limit, adjust your configuration before upgrading.

Increase write-block-size

write-block-size accepts the following discrete values: 128K, 256K, 512K, 1M, 2M, 4M, 8M.

Select the smallest value that accommodates your largest observed record sizes plus the 4-byte overhead. Base this decision on actual data, not a fixed formula.

Caution

write-block-size is not dynamic and requires a node restart to change. Increasing write-block-size increases memory used by the post-write-queue proportionally (memory = write-block-size × device count × queue depth). To offset this, you can reduce the post-write-queue depth by a corresponding factor.

If your use case allows it, you can also break up records that would not fit with the overhead and delete the original records.

Set max-record-size (Database 5.7.0 and later)

max-record-size prevents writes that exceed the configured size. Set it to write-block-size minus 16 bytes to account for the overhead and provide a safety margin.

max-record-size is dynamic and you can change it without a restart:

Set max-record-size using asinfo

asinfo -v "set-config:context=namespace;id=NAMESPACE_NAME;max-record-size=VALUE"

Caution

During a rolling upgrade or any mixed-version state, all nodes must enforce compatible record size limits. If nodes run different versions with different write-block-size or max-record-size values, records valid on one node can be rejected on another. This causes write failures or stuck migrations.

Align max-record-size across all nodes before proceeding with the upgrade. Temporarily setting max-record-size on Database 5.7+ nodes to match the constraints of nodes not yet upgraded can help maintain compatibility.

Validate changes on a single node

Before applying configuration changes cluster-wide:

1. Apply the changes to a single node.

2. Monitor memory usage, write latency, and system stability.

3. If the node remains stable, apply the changes to the remaining nodes one at a time.

Proceed with the upgrade

After verifying that no records exceed the effective write-block-size limit, or after adjusting your configuration to accommodate such records, proceed with the upgrade steps.

Monitor for write failures or migration issues during the upgrade.

Fix a stuck migration

If migrations get stuck after upgrading because records exceed write-block-size, use the following procedures to identify and resolve the affected records.

As of Database 6.0.0, Aerospike logs the digests of records that prevent migrations for some failure types. Failures to migrate because of excessive record size are logged under the drv_ssd context at the DETAIL level. See Log management for details on how to change the log level dynamically.

Caution

Revert to the default INFO level after a few seconds to avoid excessive log volume and running out of log disk space.

When upgrading from Database 5.7.0

1. Set max-record-size to write-block-size minus 16 bytes.

2. Identify the records causing the issue by enabling DETAIL log level:

   Enable DETAIL log level for drv_ssd

   asadm -e "asinfo -v 'log-set:id=0;drv_ssd=detail'"

   The server logs a message similar to the following:

   DETAIL (drv_ssd): (drv_ssd.c:1550) {namespace} write: size 1048577 - rejecting <digest id>

3. Delete the record, or shorten it, using the printed digest.

4. Issue the recluster command to force migrations to reprocess the updated or deleted records.

When upgrading from a version earlier than Database 5.7.0

1. Identify the records causing the issue by enabling DETAIL log level:

   Enable DETAIL log level for drv_ssd

   asadm -e "asinfo -v 'log-set:id=0;drv_ssd=detail'"

   The server logs a message similar to the following:

   DETAIL (drv_ssd): (drv_ssd.c:1550) {namespace} write: size 1048577 - rejecting <digest id>

2. Delete the record, or shorten it, using the printed digest.

3. Issue the recluster command to force migrations to reprocess the updated or deleted records.

   For more information, contact Aerospike Support.

Upgrade steps

See the standard upgrade guide for the common steps in an Aerospike cluster upgrade.

Namespaces with replication-factor 1 require that the node be quiesced and that migrations complete before stopping and upgrading the node. Alternatively, these namespaces may have their data restored from a backup or through XDR or other clients.

Caution

You must delete persisted data before starting a node with Database 6.0.0. Back up your data, or confirm a redundant cluster is in place, before proceeding.

For each node in the cluster:

1. (Optional) Quiesce the node and wait for migrations to complete. This step protects against the unlikely event of an irrecoverable node crash while a node has been taken out to be upgraded.

2. Stop the Aerospike daemon.

3. Delete the stored data for storage-engine device-configured namespaces. Delete the file if using a file. For raw devices, see the process in Initializing SSDs.

4. If there is already data on the device and it supports RZAT (Read Zero After Trim), use blkdiscard -z or a similar command on the 8MiB Aerospike header to wipe it, then perform a TRIM of the entire device with the appropriate command for your device and operating system. If the device does not support RZAT, you must wipe the entire device before continuing to verify no Aerospike data remains.

5. Start the Aerospike daemon.

6. Wait for migrations to complete after the node joins the cluster to allow all the data to be repopulated from other nodes (requires replication factor 2 or greater). See Monitoring migrations for more information.

7. Proceed to the next node.

Post upgrade

The truncate privilege is now a standalone granular permission, and no longer part of the write privilege. Users representing applications that perform truncates must be granted the truncate privilege on one of their roles.

After the upgrade is complete, re-evaluate your max-record-size and write-block-size settings. If you increased write-block-size or set a temporary max-record-size constraint during upgrade preparation, adjust these values based on actual workload behavior. Temporary constraints that are no longer needed can be relaxed.

Configuration parameter changes

The following configuration parameters changed in Database 6.0.0:

• scan-max-done has moved to query-max-done.
• scan-threads-limit has moved to query-threads-limit.
• background-scan-max-rps has moved to background-query-max-rps.
• single-scan-threads has moved to single-query-threads.

The following configuration parameters were removed in Database 6.0.0:

• query-threads
• query-worker-threads
• query-microbenchmark
• query-batch-size
• query-in-transaction-thread
• query-long-q-max-size
• query-priority
• query-priority-sleep-us
• query-rec-count-bound
• query-req-in-query-thread
• query-short-q-max-size
• query-threshold
• query-untracked-time-ms
• batch-without-digests

Next steps

• Review special upgrade instructions for other releases if you plan to upgrade further.
• Verify cluster health using asadm after all nodes complete migration.