Best practices for Aerospike and Linux

This page describes stability and performance best practices for Aerospike and the Linux operating system. It applies to all supported versions of Aerospike Database on Linux. The Namespace memory budget check requires Database 8.2.0 or later.

Overview

When Aerospike Database starts, it verifies certain best practices and logs a warning for each violation it finds.

Best practices checks

• For production environments, set enforce-best-practices to true so that the server shuts down if any best practices are violated during startup.

Note

This parameter is false by default, however not following the best practices on this page can lead to degraded performance and crashes on some systems.

• When enforce-best-practices is set to false, you can still monitor violations with the failed_best_practices Boolean statistic, or the best-practices info command.

• The failed_best_practices statistic reports true while any best-practice violation is active on the node, including after qualifying dynamic configuration changes. The best-practices info command returns the list of violated checks.

Best practices checked at startup

• Aerospike database best practices
  • service-threads
  • Namespace memory budget
  • Namespace device size
• Linux best practices
  • All-Flash deployment
  • RAM reserved for Linux operating system resources
  • min_free_kbytes
  • swappiness
  • THP - Transparent Huge Pages
  • Zone reclaim mode
  • NVMe partitioning
  • vm.max_map_count
  • Containers - networks
  • Maximum open file limits
    • Non-systemd
    • systemd
  • somaxconn
  • rmem-max
  • wmem-max
  • shmall
  • shmmax

Aerospike Database best practices

The best practices listed in this section are specific to the Aerospike Database.

Port scans

We recommend against scanning ports associated with the service, heartbeat, fabric, and admin port (by default ports 3000-3003). At a high enough throughput this can overwhelm the heartbeat thread resulting in the cluster breaking apart. Additionally it can generate a lot of error messages for other ports.

service-threads

The recommended value depends on the configuration of the namespaces in the aerospike.conf file.

The service-threads best practice is checked at server startup.

• We suggest and default to 5 per CPU/vCPU in the following configuration. If any namespace has:
  • storage-engine set to device and data-in-memory is set to false or
  • data-in-memory is false and commit-to-device is true
  • then the recommended value for service-threads is at least 3 per CPU/vCPU.
• Otherwise:
  • storage-engine is either set to pmem or memory or
    • storage-engine is device with data-in-memory set to true and
    • commit-to-device set to false
    • then the recommended and suggested value for service-threads is at least 1 per CPU/vCPU which is also the default for such configurations.

Namespace memory budget

Starting with Database 8.2.0, the memory best-practice check warns when the configured maximum RAM footprint across all namespaces exceeds the memory budget implied by stop-writes-sys-memory-pct. The goal is to catch over-configuration before host memory pressure forces operational intervention, even when namespace-level stop-writes thresholds have not yet engaged.

The check runs at server startup and re-evaluates when you dynamically change any parameter that affects the budget (see When the memory budget check runs).

Caution

Before upgrading to Database 8.2.0 on a cluster where enforce-best-practices is true, validate the new memory check using one of these approaches:

• Sum each namespace’s configured RAM footprint on the node (see What the server sums) and compare the total to (minimum nonzero stop-writes-sys-memory-pct across all namespaces × system memory limit) / 100. The system memory limit is the finite control group (cgroup) limit when cgroup-mem-tracking is enabled and the server can derive control-group-aware free memory; otherwise, it is host RAM.
• Upgrade a non-production node running Database 8.2.0 and run asinfo -v 'best-practices' there.

asinfo -v 'best-practices' on a node still running a release earlier than 8.2.0 uses the old total-RAM comparison and cannot predict failure under the new budget check. If the check fails after upgrade, the node will refuse to start when enforce-best-practices is true. Reduce namespace memory budgets or adjust stop-writes-sys-memory-pct before upgrading production.

What the server sums

For each namespace on the node, the server adds:

Configuration	Counted when
indexes-memory-budget	Always (0 if unset)
data-size, filesize × file count, or smallest device size × device count	storage-engine is memory only
max-write-cache × device count	storage-engine is device, or storage-engine is memory with an encryption key configured. Not counted for pmem.
post-write-cache × device count	storage-engine is device (SSD) only

Notes:

• The server does not count device or pmem record storage. Only index and cache budgets that consume RAM are included.
• max-write-cache and post-write-cache are counted at their full configured size per device or file, even though unused portions are returned to the system at runtime. This conservative total avoids understating risk.
• Device count is the number of configured device or file entries for the namespace on the node. For storage-engine memory, those entries are the persistence files or devices.
• For storage-engine memory, data-size takes precedence over filesize and device sizes. When the sum uses filesize, the server multiplies filesize by the number of configured files.

Leave headroom beyond this sum for the OS, fragmentation, and unbudgeted index growth. See Capacity planning and In-memory data storage and stripes.

When indexes-memory-budget is 0 (default), the check adds 0 for that parameter. Actual primary and secondary index RAM is not included in the sum. Set a nonzero budget that reflects your expected index footprint to include it. To estimate primary index RAM, see Calculating primary index storage. In production, a nonzero indexes-memory-budget also establishes an index-specific stop-writes threshold and enables utilization alerting through indexes_memory_used_pct. When the budget is 0, monitor the applicable *_used_bytes metrics and overall memory pressure through stop-writes-sys-memory-pct. See Index memory monitoring for monitoring guidance.

Stop-writes budget

The server compares the sum across all namespaces on the node against:

• stop-writes-sys-memory-pct: Uses the minimum nonzero value configured across all namespaces. If none is set, the default is 90. If every namespace explicitly sets stop-writes-sys-memory-pct to 0, the memory best-practice check is disabled entirely.
• system memory limit: Host total RAM from /proc/meminfo, unless cgroup-mem-tracking is enabled and a control group memory limit is detected. In that case the control group limit is used (consistent with stop-writes evaluation since Database 8.1.2).

The check fails when configured maximum memory is greater than the stop-writes budget (exactly at the limit passes).

Warning and monitoring

On violation, the server logs a warning and reports memory through the best-practices info command and failed_best_practices statistic:

failed memory check - data & indexes memory spec for all namespaces (NUM_BYTES) exceeds system memory (NUM_BYTES)

The second byte count is the stop-writes budget: the memory limit × stop-writes-sys-memory-pct / 100.

If enforce-best-practices is true, startup fails when this check (or any other best-practice check) fails. A dynamic configuration change that pushes the node over budget logs a warning and updates best-practices, but does not shut down a running node. However, if enforce-best-practices is true, the node will not restart while the check remains violated.

If left unresolved, memory over-configuration can progress to namespace stop-writes and eventually to OOM process termination. See Stop-writes thresholds and data-size for the full progression.

Caution

Raising stop-writes-sys-memory-pct to clear this warning without reducing configured memory only postpones stop-writes. It does not add physical RAM. See Stop-writes thresholds and data-size.

To bring a node back within budget:

1. If the setting is dynamic (indexes-memory-budget, max-write-cache, post-write-cache, stop-writes-sys-memory-pct): change it with asadm or asinfo, confirm that asinfo -v 'best-practices' no longer reports memory, then put the same value in aerospike.conf.
2. If the setting is static (data-size, filesize): correct aerospike.conf and restart the node. The startup check re-evaluates.

If the node cannot start because enforce-best-practices is true, correct aerospike.conf before starting it again.

When the memory budget check runs

• Startup: The check runs with other best-practice checks.
• Dynamic configuration: The check re-evaluates when you change indexes-memory-budget, stop-writes-sys-memory-pct, max-write-cache, post-write-cache, or cgroup-mem-tracking through asinfo or equivalent tooling.

data-size and filesize are not dynamically configurable. Changes to these parameters take effect after a node restart, when the startup check re-evaluates.

When a dynamic change brings the configuration back within budget, the memory token is removed from best-practices without restarting the node.

Note

Prior to Database 8.2.0, this check compared in-memory data and index budgets to total system RAM. For the history of the deprecated memory-size parameter, see Aerospike Database 7.0.0 Release Notes.

Namespace device size

All the namespace storage devices should be the same size within an 8 MiB range of tolerance.

The namespace device size best practice is checked at server startup.

Initialize new cluster nodes with SMD

When adding a new node to an existing cluster, it is a best practice to initialize it by copying the cluster’s shared metadata (SMD) files from an active cluster node.

See Directory structure - Run time directories for more information about the SMD directory.

When every node in the cluster runs Database 8.2.0 or later, a joining node also waits for initial SMD sync before it answers client or admin connections. We still recommend copying SMD files to seed a node. See System metadata readiness on node join.

Linux best practices

The best practices listed in this section are specific to the Linux operating system.

All-Flash deployment

In an All-Flash deployment, the following kernel parameters are required. enforce-best-practices verifies that these kernel parameters are at least expected values. The All-Flash deployment best practice is checked at server startup.

/proc/sys/vm/dirty_bytes = 16777216
/proc/sys/vm/dirty_background_bytes = 1
/proc/sys/vm/dirty_expire_centisecs = 1
/proc/sys/vm/dirty_writeback_centisecs = 10

• When running as non-root, you must set these values before running the Aerospike server.
• When running as root, the server configures them automatically.

Either way, the node will not start if these parameters can’t be correctly set manually or automatically by the server.

RAM reserved for Linux operating system resources

To help prevent out-of-memory issues with host hardware, keep 10-15% of total physical memory reserved for Linux system resources. The RAM reserved for Linux operating system resources best practice is checked at server startup.

The following may influence memory usage:

• Overhead from the Linux OS and services.
• Overhead caused by memory fragmentation.
• Overhead from primary and secondary Aerospike indexes.
• Namespace data for in-memory namespaces. For more information, see Capacity planning.
• Overhead from cache and queue-related configurations, including max-write-cache (per device) and post-write-cache (per device). The Namespace memory budget best-practice check counts these at full configured size. See Flush size and cache sizes for more information.
• Overhead from the Aerospike process.

min_free_kbytes

Themin_free_kbytes kernel parameter controls how much memory to keep free from filesystem caches. Normally, the kernel occupies almost all free RAM with filesystem caches and frees up memory for allocation by processes as required. As Aerospike performs large allocations in shared memory (1 GiB chunks), the default kernel value may result in an unexpected OOM (out-of-memory kill).

The min_free_kbytes best practice is checked at server startup.

We recommend that you configure the parameter to a minimum of 1.1 GiB, preferably 1.25 GiB if using cloud vendor drivers as these can make large allocations. This ensures that Linux always keeps enough memory available and free for large allocations.

Tip

If min_free_kbytes is set too low, the kernel may use most of the free RAM for filesystem caches, leaving insufficient memory for Aerospike’s large allocations and triggering an out-of-memory kill.

1. Check the parameter value.

   cat /proc/sys/vm/min_free_kbytes

2. If the value is lower, adjust it accordingly to the running kernel and persist across reboots.

   echo 3 > /proc/sys/vm/drop_caches
   echo 1310720 > /proc/sys/vm/min_free_kbytes
   echo "vm.min_free_kbytes=1310720" >> /etc/sysctl.conf

Swappiness

For low-latency operations, using swap to any extent drastically slows down performance. We recommend that you disable swap with swapoff -a and remove the swap partition from /etc/fstab.

If that’s not possible for operational reasons, set the swappiness to 0:

echo 0 > /proc/sys/vm/swappiness
echo "vm.swappiness=0" >> /etc/sysctl.conf

The swappiness best practice is checked at server startup.

THP - Transparent Huge Pages

The best practices startup check permits thp-enabled and thp-defrag to be set to either madvise or never.

Aerospike strongly recommends disabling Transparent Huge Pages (THP) before the Aerospike service starts. The Linux kernel uses THP to improve overall system responsiveness and allocation speed, however it can be counterproductive for high-throughput and low-latency databases when they perform multiple small allocations. THP can cause the system to run out of RAM, with symptoms similar to a memory leak. THP also causes latency when the defragmentation page locks.

Zone reclaim mode

For NUMA architectures,zone_reclaim_mode causes aggressive reclaims and memory scans when enabled.

We recommend that you disable zone_reclaim_mode by setting /proc/sys/vm/zone_reclaim_mode to 0.

The zone_reclaim_mode best practice is checked at server startup.

NVMe partitioning

NVMe devices are normally capable of 4 simultaneous I/O operations.

Due to their connection design, these operations occupy 4 PCIe I/O lanes. On raw devices, Aerospike suggests that you partition each NVMe device to at least 4 partitions to allow 4 write threads to operate in Aerospike and greatly improves the disk speed.

If using a single partition with Aerospike as raw device, iostat may show 100% disk utilization (%util), while the await operation queuing statistic may be showing no queueing (await <1 means no queueing is happening). This indicates that the disk itself can do more, while the PCIe lanes that are used are already saturated.

The NVMe partitioning best practice is checked at server startup.

See Partition your flash devices for details on device partitioning.

vm.max_map_count

If you use Kubernetes or Docker, we recommend that you raise the max_map_count parameter, which controls the maximum number of memory map commands that can be performed by a process. If max_map_count is too low, it may result in memory allocation issues during normal operation.

To change this parameter:

echo "vm.max_map_count=262144" >> /etc/sysctl.conf
echo 262144 > /proc/sys/vm/max_map_count

The vm.max_map_count best practice is checked at server startup.

Note

After modifying max_map_count may need to restart the Docker daemon and all its containers for the changes to take effect.

Containers - networks

When using Kubernetes or Docker, the default behavior is to use EXPOSE and PUBLISH features to publish ports from a container through the host to the outside world. This causes the Docker process to monitor a given port on the host and forward all packets to the container itself. This is highly inefficient and may cause latencies, packet drops and other crashes within the containers under heavy loads.

If using containers, we recommend that you configure those containers to either:

• Use bridged networking, rather than Docker-only NAT, or
• Use iptables to forward packets to the NAT network Aerospike containers, rather than the Docker EXPOSE port feature.
• If using a Docker container, run it with the --net=host flag to inherit /proc/sys/net/core/*mem_max files. Without that flag, maximums cannot be modified from within that environment.

The containers - network best practice is checked at server startup.

See the Docker configuration manuals for details.

Maximum open file limits

Aerospike clients perform dynamic connections to the database nodes as required. This may result in many active connections. These connections, on a Linux system, hold a file descriptor and are treated as open files.

The Aerospike configuration parameter proto-fd-max specifies the maximum number of allowed client connections. The Aerospike server does not start if proto-fd-max is higher than the Linux system’s maximum open files configuration for the process.

After installing Aerospike, verify that the maximum open files for the asd process is configured to have a higher maximum open file value than proto-fd-max to allow for fabric and heartbeat connections as well as any open files.

• Mesh heartbeat and fabric should run on the same NIC.

The maximum open file limits best practice is checked at server startup.

Non-systemd

Edit /etc/init.d/aerospike.conf and change the value of the following line.

ulimit -n 100000

The non-systemd best practice is checked at server startup.

systemd

The systemd best practices is checked at server startup.

1. Create an override.conf file to control this.

   cat <<EOF > /etc/systemd/system/aerospike.service.d/override.conf
   [Service]
   LimitNOFILE=<MAX NUMBER OF FILE DESCRIPTORS>
   EOF

2. Reload the systemd daemon.

   systemctl daemon-reload

3. Restart the Aerospike server to apply the new value.

4. (Optional) You can apply this change dynamically to the asd process if prlimit is available:

   prlimit --pid $(pgrep asd) --nofile=200000

somaxconn

Limit of socket listen() backlog, known in userspace as SOMAXCONN. Defaults to 4096. (Prior to Linux kernel 5.4.0, the default was 128. See tcp_max_syn_backlog for additional tuning for TCP sockets.

echo 4096 > /proc/sys/net/core/somaxconn

The somaxconn best practice is checked at startup.

rmem-max

The maximum receive socket buffer size in bytes.

echo 15728640 > /proc/sys/net/core/rmem_max

The rmem-max best practice is checked at startup only in EE.

wmem-max

The maximum send socket buffer size in bytes.

echo 5242880 > /proc/sys/net/core/wmem_max

The wmem-max best practice is checked at startup only in EE.

shmall

The sum of all shared memory segments on the whole system.

The shmall best practice is checked at startup only in EE.

shmmax

The maximum size of a single shared memory segment.

The shmmax best practice is checked at startup only in EE.