Manage solid state storage devices

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

This page describes how to manage solid state storage devices (SSDs) in your Aerospike nodes.

The Aerospike database recovers quickly after adding, removing, or replacing storage devices. You may use any type of storage device in your Aerospike nodes, including SSDs and hard disk drives (HDDs). Aerospike is optimized for SSDs.

SSD changes require a restart. You can use a Fast Start or a Cold Start if your devices are hot-swappable, and your storage-engine type as summarized in the following table:

Storage type	Hot-swappable	Not hot-swappable
pmem	fast start	cold start
memory	cold start	cold start
device	fast start	cold start
Index stored on flash or pmem	fast start	fast start

Applies to

Aerospike Database nodes that use raw SSD or device storage (storage-engine device)
Linux hosts with root or sudo access for device operations

Prerequisites

asd stopped when you replace or reconfigure devices (except hot-swap steps where noted)
util-linux package installed (blkdiscard for zeroization)
Privileged shell access for blkdiscard, udev, and scheduler configuration

After the procedures on this page, devices are zeroized where required, referenced by persistent /dev/disk/by-id/ paths in aerospike.conf, and configured with the correct OS-level disk scheduler.

Initialize SSDs

You must initialize SSDs before using them in an Aerospike deployment. Initializing SSDs ensures that old data, partitioning, and master boot records (MBRs) do not interfere with Aerospike operations. Older devices containing Aerospike data must be entirely zeroized before reuse, but to save time, newly-provisioned devices can have their headers zeroized and the rest of the write blocks trimmed.

To learn more about initializing multiple SSDs in a single operation, see How to zeroize multiple storage devices simultaneously.

New SSDs

Newly-provisioned devices from cloud service providers are new partitions of existing drives that have not been zeroized, meaning they may still contain some data or random noise instead of zeros. Because of the way Aerospike works, accessing raw blocks on the device without a storage system, any non-zero blocks on the drive may get reinterpreted by Aerospike and affect your operations.

For new SSDs, zeroize the 8 MiB device header using blkdiscard -z, included on most Linux distributions in the util-linux package.

blkdiscard -z --length 8MiB /dev/DEVICE_ID

Verify zeroization by confirming the command exits with status 0 and the device is ready to add to aerospike.conf.

Used SSDs

If the drive was previously part of an Aerospike deployment, you need to fully zeroize it before use.

First, check whether your drive supports RZAT (Read Zero After Trim). See this Wikipedia article for a non-exhaustive list.

If the drive supports RZAT, perform a trim on the entire device.
Terminal window
```
blkdiscard /dev/DEVICE_ID
```
Then zeroize the 8 MiB header.
Terminal window
```
blkdiscard -z --length 8MiB /dev/DEVICE_ID
```
If the drive does not support RZAT, you must zeroize the entire device, not just the header.
Terminal window
```
blkdiscard -z /dev/DEVICE_ID
```

Verify zeroization by confirming each blkdiscard command exits with status 0 before you add the device to aerospike.conf.

Configure disk scheduler at the operating system level

Complete this for each raw namespace device before you start asd, especially when running non-root.

The scheduler-mode configuration parameter was removed in Aerospike Database 6.4.0. Set the I/O scheduler for Aerospike Database storage devices at the operating system level.

Run these steps as root. DEVICE_PATH is the persistent path from aerospike.conf (for example /dev/disk/by-id/DISK_BY_ID_PATH).

Identify the disk that backs your storage device.

The I/O scheduler is a property of the whole physical disk. Resolve DEVICE_PATH to its parent disk name with lsblk. When DEVICE_PATH is already a whole disk, lsblk returns no parent name—use the name from readlink -f DEVICE_PATH as BLOCK_DEVICE in the steps that follow.
Terminal window
```
lsblk -no pkname "$(readlink -f DEVICE_PATH)"
```
Replace BLOCK_DEVICE in the following steps with that disk name (for example, nvme0n1, not nvme0n1p1), or with the name from readlink -f DEVICE_PATH if lsblk printed nothing.
Check the current scheduler.
Terminal window
```
cat /sys/block/BLOCK_DEVICE/queue/scheduler
```
The active scheduler appears in square brackets, for example [none] or [mq-deadline].

Set the scheduler.

Use noop on older kernels or none on systems that use the multi-queue block layer (common for NVMe). Use whichever value appears in the scheduler list for your device.

# Older kernels (for example, RHEL 7 and earlier)
echo noop | tee /sys/block/BLOCK_DEVICE/queue/scheduler

# blk-mq systems (for example, RHEL 8 and later, many NVMe devices)
echo none | tee /sys/block/BLOCK_DEVICE/queue/scheduler

Persist the scheduler with udev.

Add a rule to /etc/udev/rules.d/60-aerospike-scheduler.rules that matches your devices by persistent identifier, using the same identifiers as your device ownership rules. Match ENV{DEVTYPE}=="disk" even when DEVICE_PATH is a partition—the queue/scheduler attribute exists only on the whole-disk device, and the scheduler applies to the entire disk.

On cloud VMs, run udevadm info --query=property --name=DEVICE_PATH and match on the property present for that device (for example ID_PATH or ID_SERIAL for Google disks), not only ID_SERIAL_SHORT or ID_WWN.
```
SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL_SHORT}=="DEVICE_SERIAL_1", ATTR{queue/scheduler}="none"
SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_WWN}=="DEVICE_WWID_2", ATTR{queue/scheduler}="noop"
```
Set none or noop according to what your kernel supports. Reload udev rules and trigger the rule for the disk:
Terminal window
```
udevadm control --reload-rules
udevadm trigger --type=devices --subsystem-match=block --action=change --name-match="/dev/BLOCK_DEVICE" --settle
```
Verify the active scheduler appears in square brackets as none or noop:
Terminal window
```
cat /sys/block/BLOCK_DEVICE/queue/scheduler
```

Adding a hot-swap SSD

After you install and initialize a hot-swap SSD, update /etc/aerospike/aerospike.conf with the name of the SSD. We recommend using the World Wide Identifier (WWID), rather than the /dev name, such as /dev/sda or /dev/nvme0n1p1. WWIDs are unique and do not change, while /dev names can change after a restart.

Follow Configure disk scheduler at the operating system level for each device you add. Use the ls command to find WWIDs:
Terminal window
```
ls -l /dev/disk/by-id/
```
The following is an example of the output from a system with an NVMe drive controller. The long output shows that the WWIDs are symlinked to /dev/nvme0n1XX:
Terminal window
```
lrwxrwxrwx 1 root root 13 Mar  6 07:34 nvme-eui.8ce38e100077d4b6 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Mar  6 07:34 nvme-eui.8ce38e100077d4b6-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Mar  6 07:34 nvme-eui.8ce38e100077d4b6-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 Mar  6 07:34 nvme-eui.8ce38e100077d4b6-part3 -> ../../nvme0n1p3
```

Enter your SSD partition names in the storage-engine device section of the namespace configuration:

namespace NAMESPACE-NAME {

    storage-engine device {
        device /dev/disk/by-id/nvme-eui.8ce38e100077d4b6-part1
        device /dev/disk/by-id/nvme-eui.8ce38e100077d4b6-part2
        device /dev/disk/by-id/nvme-eui.8ce38e100077d4b6-part3
        flush-size 128K
        max-record-size 1M
    }
}

To optimize SSD performance, investigate flush-size. For most devices 128KiB seems to be the optimal flush-size.
Verify device paths and scheduler before restart.

Confirm each configured path exists:
Terminal window
```
ls -l /dev/disk/by-id/nvme-eui.8ce38e100077d4b6-part1
```
Confirm the disk scheduler is set per Configure disk scheduler at the operating system level.

Restart Aerospike

Stop and then Fast Start the server if your data is stored on disk (pmem or flash).

Cold Start the server if your data is in memory (shmem).

See index-type for more information about storage types.

Verify the node is ready:

asinfo -v status

Expected output: OK

Non-hot-swappable devices

Stop asd on the node.
Replace the SSD hardware.
Initialize the new SSDs per Initialize SSDs.
Update aerospike.conf with the new /dev/disk/by-id/ paths.
Follow Configure disk scheduler at the operating system level for each device.
Cold Start the server if your storage type is shmem. Fast Start when your storage types are pmem or flash.
Verify the node returns OK from asinfo -v status.

Managing SSD replacements on multiple nodes

If multiple Aerospike nodes require SSD replacements, perform the same procedure on each node one at a time. Wait for data migrations to complete before continuing to the next node.

Next steps

Configure namespace storage in namespace storage configuration.
For non-root deployments, see Configure Aerospike Database to run as non-root.
After device changes, use Fast Start or Cold Start as appropriate for your storage type.