Health check

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

Use Aerospike’s health check to detect health outliers in a cluster, enable and verify the feature, interpret health-outliers and health-stats output, and recognize common false positives.

Health outliers are nodes or devices that deviate from the normal or expected pattern of database activity or performance metrics. Outliers are detected when a node or device value is statistically high compared with the same metric on other peers or devices in the comparison set.

Outliers are important in database monitoring because they can indicate hardware failures, network issues, misconfigurations, or resource exhaustion that may be affecting cluster performance.

Applies to

Aerospike Database 4.3.1.3 and later. Minimum threshold values changed in Database 8.1.0.

Prerequisites

A running Aerospike Database cluster with at least four nodes. Health check needs multiple peers for statistical comparison.
Shell access to each node to run asinfo, or equivalent remote access.
enable-health-check enabled on all nodes before you run health-outliers or health-stats.

How health check works

Aerospike’s health check runs independently on every node. Each node maintains a sliding window of the previous 30 minutes of statistics, compares them once per minute, and reports anomalies when detected. At least one minute of accumulated statistics is required before outlier detection can run.

The Aerospike health check tracks two categories of statistics:

Cluster statistics track the health of peer nodes in the cluster (four statistics).
Local statistics track the health of the node itself (one statistic).

How outliers are detected

Health check compares each value for a statistic against the same statistic on the other peers or devices. To find values that sit well above the rest, it sorts those values and looks at the middle 50%. The width of that middle band is the Interquartile Range (IQR), and the top of the band is the third quartile (Q3).

Health check computes an upper boundary by adding a multiple of the IQR to Q3. That multiplier is the k-factor. Health check uses a k-factor of 3, so the upper boundary is Q3 + 3×IQR.

A node or device is reported as an outlier only when all of the following are true:

The value exceeds that upper boundary (Q3 + 3×IQR).
The value exceeds the minimum threshold for that statistic.
Confidence is at least 50%.

Minimum data points

Outlier detection needs at least three values for a statistic so it can tell a majority trend from a single high value. With only one or two values, no outlier can be declared.

This requirement maps differently to each statistic category:

Cluster statistics (fabric connections opened, node arrivals, proxy requests, replica latency) are tracked per peer, so each node compares itself against its peers. A node has one fewer peer than the cluster size, so a minimum of four nodes is required to reach three comparable values.
Device read latency (a local statistic) is compared across the devices attached to a namespace, so it requires at least three devices in that namespace independent of cluster size.

Minimum thresholds

Thresholds apply to values accumulated over the health check’s 30-minute sliding window. For example, a fabric-connections threshold of 60 means a peer must have opened 60 connections to a single peer within that window. The value must still exceed the upper boundary before the peer is reported.

The minimum thresholds for different statistics are listed in the following table.

Statistic	Min threshold (Database 8.1.0 and later)	Min threshold (prior to Database 8.1.0)
Number of fabric connections opened	60	5
Number of node arrivals	10	1
Number of proxy requests	20	2
Replica latency	0	0
Device read latency	0	0

Tracked statistics

Each statistic is tracked per peer node (cluster statistics) or per device (local statistics). Health check compares each peer’s value against the same statistic on other peers.

Statistic	Category	What it indicates
Number of fabric connections opened	Cluster	Many connections attempted when earlier connections failed or timed out. May indicate network problems, fabric misconfiguration, firewall restrictions, or TLS misconfiguration. At cluster startup or after a topology change, peers typically open similar numbers of connections.
Number of node arrivals	Cluster	How many times a node successfully reconnected after instability. May increase rapidly when the network is intermittent and the heartbeat uses multicast.
Number of proxy requests	Cluster	Clients routing around a node with poor communication or slow responses.
Replica latency	Cluster	The replica is under resource pressure or has a poor connection to the primary. Does not necessarily indicate a storage device issue. Write operations are buffered in memory by default unless commit-to-device is `true`.
Device read latency	Local	One device is much slower than others in the same namespace. Requires at least three devices in the namespace for outlier detection.

Configuration

Health check is a service-level configuration and is disabled by default. You can enable it statically in aerospike.conf or dynamically with the asinfo command enable-health-check.

Static configuration

To enable the health check, add enable-health-check true to the service section of /etc/aerospike/aerospike.conf. To disable the health check, change true to false.

service {
  ...
  enable-health-check true
  ...
}

Dynamic configuration

Use the following commands to enable or disable health check dynamically on a running node:

asinfo -v "set-config:context=service;enable-health-check=true"

asinfo -v "set-config:context=service;enable-health-check=false"

Verify

Confirm that health check is enabled on each node:

asinfo -v "get-config:context=service" | tr ';' '\n' | grep enable-health-check

Expected output:

enable-health-check=true

After enabling health check, allow at least one minute for statistics to accumulate. Outlier detection requires a cluster with at least four nodes. Run health-outliers to confirm the command returns results. An empty list indicates that no outliers were detected.

Info commands

The health-outliers command

health-outliers compares metrics for each host to the same metrics on other nodes within the cluster and returns a list of outliers and their information.

health-outliers output format

Returns a list of tuples:

<id=. . . :confidence_pct=. . . :reason=. . . >

Where:

id= is either the node ID or the device ID which is declared as an outlier.
confidence_pct= is a metric that, when high, denotes that the statistic is highly skewed.
reason= is the description of the statistic that is responsible for declaring the ID as an outlier.

Command

asinfo -v health-outliers -l

Example output

id=bb9040011ac4202:confidence_pct=100:reason=fabric_connections_opened
id=bb9040011ac4202:confidence_pct=100:reason=proxies
id=bb9040011ac4202:confidence_pct=100:reason=node_arrivals
id=/opt/aerospike/data/bar2.dat:namespace=test:confidence_pct=100:reason=device_read_latency

The health-stats command

health-stats returns a list of statistics maintained by the health check for reference. It is useful as a filter when you need the results as the input array elements of a custom detector.

health-stats output format

Returns a list of tuples:

<stat=. . . :value=. . . :[node=. . .:device=. . .:namespace=. . .]>

Where:

stat= identifies the statistic type and defines the stat group. See Customize your outlier detector.
value= is the current window value: a cumulative count over the sliding window for counter statistics, or a moving average for latency statistics.

The stat entry serves as a group ID. Filter on stat and collect the corresponding value entries to form an input array for a custom detector.

Other entries in the tuple provide additional information about the stat:

node= is the node ID.
device= is the device name. Only one of node or device appears per tuple, depending on the stat type.
namespace= is an optional entry that specifies the namespace corresponding to the stat.

Command

asinfo -v health-stats -l

Example output

stat=fabric_connections_opened:value=0:node=BB9070011AC4202
...
stat=fabric_connections_opened:value=0:node=BB9030011AC4202
...
stat=fabric_connections_opened:value=0:node=BB9050011AC4202
...
stat=fabric_connections_opened:value=153:node=BB9040011AC4202
...
stat=test_device_read_latency:value=1408:device=/opt/aerospike/data/bar1.dat:namespace=test
stat=test_device_read_latency:value=306132:device=/opt/aerospike/data/bar2.dat:namespace=test
stat=test_device_read_latency:value=11630:device=/opt/aerospike/data/bar3.dat:namespace=test

Troubleshoot

False positives

During testing and production use, several scenarios can lead to false positives in the outlier detection system.

False positives in low-throughput environments

In a low-throughput setup, a single write operation can trigger a false positive. When a write is sent to one replica, its latency will be non-zero, while all other nodes show zero latency due to inactivity. This discrepancy can cause the system to incorrectly flag the replica with the non-zero latency as an outlier.

False positives after topology changes prior to Database 8.1.0

Prior to Database 8.1.0, minimum thresholds for fabric connections opened, node arrivals, and proxy requests were low enough that expected topology events could trigger false positives. When a new node joined the cluster, existing nodes legitimately opened fresh fabric connections and related counters spiked for one window. Because the new or rejoining peer’s counts were much higher than stable peers, health check could report a false outlier even though the cluster was healthy.

Database 8.1.0 raised these thresholds by roughly 10×. For example, fabric connections opened increased from 5 to 60. Brief spikes from join, restart, and topology changes stay under the reporting threshold while sustained misbehavior still surfaces. See Minimum thresholds for the current values.

Customize your outlier detector

An outlier detector accepts an array of values as input and returns a list of outliers. Use health-stats output to write a custom detector.

Filter on health-stats and use the results as the input array elements of the custom detector. Treat other details in the statistics as attributes. See the health-stats example output.

Filtering on statistic values forms the following groups:

Group 1: stat=fabric_connections_opened

stat=fabric_connections_opened:value=0:node=BB9070011AC4202
stat=fabric_connections_opened:value=0:node=BB9030011AC4202
stat=fabric_connections_opened:value=0:node=BB9050011AC4202
stat=fabric_connections_opened:value=153:node=BB9040011AC4202

The input to the outlier detector will be [0, 0, 0, 153]

Group 2: stat=test_device_read_latency

stat=test_device_read_latency:value=1408:device=/opt/aerospike/data/bar1.dat:namespace=test
stat=test_device_read_latency:value=306132:device=/opt/aerospike/data/bar2.dat:namespace=test
stat=test_device_read_latency:value=11630:device=/opt/aerospike/data/bar3.dat:namespace=test

The input to the outlier detector will be [1408, 11630, 306132]

Pass each value array to your detector and map outlier indices back to node= or device= attributes from the filtered health-stats output. Compare your detector’s results with health-outliers to validate behavior.

Next steps

Monitor essential cluster metrics with Key metrics to monitor.
Review the full enable-health-check reference and health-outliers / health-stats info command reference.