Skip to content

Diagnose hot keys

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

This page explains how to find a hot key and reduce the load it puts on one node. A hot key is a single record that receives a disproportionate share of requests.

Applies to: Aerospike Database Community Edition and Enterprise Edition, in Available and Partition-tolerant (AP) and strong consistency (SC) namespaces. SC namespaces require Enterprise Edition.

Audience: Operators and site reliability engineers who troubleshoot uneven node load or KEY_BUSY errors in a production cluster.

Outcome: You know the namespace, digest, and serving node of the hot key, or you have ruled out a hot key as the cause of the imbalance.

Prerequisites

  • asadm and asinfo access to every node in the cluster.

  • If security is enabled, a user with the sys-admin role, which is required for log-set and configuration changes.

  • Read access to the server log on each node. In Database 8.1.0 and later, the packaged configuration file logs to the console, and systemd stores the log in the journal. Read it with journalctl -u aerospike -a -o cat. If the node runs in a container and logs to the console, read the log through your container runtime.

  • For packet capture only: tcpdump on the node and packet-capture privileges.

How hot keys cause node imbalance

  • Each record belongs to one of 4,096 partitions, and each partition has one master node.
  • All writes to a record go to the master node for its partition. A write hot key also adds replica-write load to the nodes that hold replicas of the record’s partition.
  • With the default replica policy, SEQUENCE, each read also goes to the master first.
  • A hot key therefore concentrates load on one node, while the load on nodes that don’t hold the record stays normal.

For example, suppose an application counts views for a whole site by incrementing one site-views record on every page request. Every increment goes to the master node for that record’s partition. That node’s write latency and throughput rise, while nodes with no copy of the partition stay level. At peak traffic, more than 20 increments are pending for the record at once. Clients therefore start to receive KEY_BUSY errors, and fail_key_busy grows on that node. If you enable key-busy detail logging on the node, the log includes the digest of the site-views record. Responses from that node take longer, so more requests are in flight and clients open more connections to the node, which makes the imbalance worse.

Which transactions can fail with KEY_BUSY

The server tracks in-progress transactions in the rw-hash. When a transaction for a record is already in the rw-hash, the next transaction for the same record is pending until the first one completes. When more than transaction-pending-limit transactions are already pending, the server does the following:

  • Fails the new transaction with KEY_BUSY (error code 14).
  • Increments fail_key_busy.

Whether a transaction can fail this way depends on its type and the namespace mode:

TransactionCan fail with KEY_BUSY
Writes, deletes, and User-Defined Function (UDF) calls, including batch writes and background query operations, in any namespaceYes
Reads in an AP namespaceOnly during migrations, when the read consistency level is all
Linearized reads and multi-record transaction (MRT) read verification in an SC namespace, with replication factor 2 or moreYes
Other reads in an SC namespaceOnly during migrations, or while a write to the same record is still replicating

A read hot key in a stable AP namespace therefore never fails with KEY_BUSY and leaves fail_key_busy flat. Raising transaction-pending-limit doesn’t change that. For the complete rules, see fail_key_busy.

Indicators of a hot key

Compare each node with its peers. The indicators of a hot key usually appear on one node.

  • Higher latency on one node.
  • Higher throughput, in operations per second, on one node.
  • More client connections on one node, in the client_connections statistic. Slower responses mean more requests in flight on that node, so clients open more connections to it.
  • Higher device or network utilization on one node, for example in iostat or sar output.
  • The imbalance moves to another node after you quiesce or restart the node, because the master role for the hot key’s partition moves with it.

Use the following commands to compare latency, throughput, and connections across nodes. Replace NAMESPACE with the namespace name.

Terminal window
asadm -e "show latencies for NAMESPACE"
asadm -e "show statistics service like client_connections"

A slow node, such as one with a failing device or a network problem, has similar indicators. The difference is that a slow node’s indicators remain on that node after its partitions move to other nodes. For slow-node causes, see Node problems.

Identify the hot key

Run the following steps in order, and stop at the first step that gives you a digest. Steps 3 and 4 add load to the node. Run them only on the affected node and only for a few seconds.

In the commands, replace NODE_IP with the address of the affected node. Replace SINK_ID with the ID of the log sink to change. Add your usual asadm and asinfo connection and authentication options. This page uses the log-set info command. set-log is a synonym that some Support articles use.

  1. Check whether fail_key_busy is growing.

    Terminal window
    asadm -e "watch 10 6 show statistics namespace for NAMESPACE like fail_key_busy"

    fail_key_busy counts failures since the node started, so look for a value that grows between iterations, and compare nodes. Each node also prints the special-errors ticker line every ticker-interval when any of its counters is nonzero:

    Oct 26 2026 20:50:06 GMT: INFO (info): (ticker.c:1209) {test} special-errors: key-busy (1234,0) record-too-big 0 lost-conflict (0,0)

    The first number after key-busy is fail_key_busy. The second number is fail_xdr_key_busy, which counts a different error that cross-datacenter replication (XDR) raises.

    If fail_key_busy grows, continue with step 2. If it stays flat while the indicators persist, the hot transactions aren’t failing with KEY_BUSY. They can be reads, or writes that stay under the limit. Skip to step 3, which logs every transaction.

  2. Log the digests of failing transactions with the key-busy context.

    List the node’s log sinks and their IDs, record the current key-busy level of the sink, then enable detail level for key-busy:

    Terminal window
    asinfo -h NODE_IP -v "logs"
    asinfo -h NODE_IP -v "log/SINK_ID/key-busy"
    asinfo -h NODE_IP -v "log-set:id=SINK_ID;key-busy=detail"

    The server logs the first failure of each combination of namespace, digest, transaction type, and source right away. At each ticker interval (10 seconds by default), it logs one (repeated:N) line per combination, where N counts that combination’s failures since the previous interval:

    Oct 26 2026 20:50:06 GMT: DETAIL (key-busy): (rw_request_hash.c:224) {test} 8df238affec6f8e3a2c22d6c54c91c5bc4f3ff81 write from 10.0.3.182:51160
    Oct 26 2026 20:50:16 GMT: DETAIL (key-busy): (rw_request_hash.c:224) (repeated:57) {test} 8df238affec6f8e3a2c22d6c54c91c5bc4f3ff81 write from 10.0.3.182:51160

    Each line shows the namespace in braces, the 40-character hexadecimal digest, the transaction type, and the source. The source is the client address and port, or proxy, bg-udf, or bg-ops for proxied and background transactions. For the full list of values, see key-busy detail.

    After at least one ticker interval, count failures per record by adding up the (repeated:N) lines. If the node logs to the journal instead of a file, pipe journalctl -u aerospike -a -o cat into the grep command. Omit the log filename from grep.

    Terminal window
    grep -E 'DETAIL \(key-busy\).*\(repeated:[0-9]+\)' /var/log/aerospike/aerospike.log \
      | awk '{ for (i = 1; i <= NF; i++) {
                 if ($i ~ /^\(repeated:[0-9]+\)$/) { n = substr($i, 11, length($i) - 11) }
                 if ($i ~ /^\{.+\}$/) { count[$i " " $(i + 1) " " $(i + 2)] += n; break }
               } }
             END { for (k in count) print count[k], k }' \
      | sort -rn | head -10

    Example output:

    57 {test} 8df238affec6f8e3a2c22d6c54c91c5bc4f3ff81 write
    1 {test} 1f0e3c9b4a7d2e6f8a1b3c5d7e9f0a2b4c6d8e0f batch-sub-write

    The record with the highest count is the hot key. When you have the digest, restore the level that you recorded. Replace RECORDED_LEVEL with that level in the following command:

    Terminal window
    asinfo -h NODE_IP -v "log-set:id=SINK_ID;key-busy=RECORDED_LEVEL"

  3. Sample every transaction with the rw-client or batch-sub context. These contexts log both successful and failed transactions, so they also find hot keys that never fail with KEY_BUSY. The two contexts log different transactions:

    • rw-client logs single-record client transactions.
    • batch-sub logs batch subtransactions.

    Use batch-sub when the application reaches the hot key through batch requests.

    The following command enables rw-client detail logging for 10 seconds and then sets the level back to info:

    Terminal window
    asinfo -h NODE_IP -v "log-set:id=SINK_ID;rw-client=detail"; sleep 10; asinfo -h NODE_IP -v "log-set:id=SINK_ID;rw-client=info"

    If asinfo -h NODE_IP -v "log/SINK_ID/rw-client" returns a level other than info before you start, use that level in the last command. If you interrupt the command, run the last asinfo command yourself to restore the level. For batch subtransactions, replace rw-client with batch-sub in both log-set commands.

    To keep these lines out of the main log, define a separate log sink for the context, as described in Create a log sink for debugging. A new log sink takes effect only after the node restarts, so set it up before you need it.

    The lines look like the following:

    Oct 26 2026 20:51:00 GMT: DETAIL (rw-client): (thr_tsvc.c:103) {test} digest 8df238affec6f8e3a2c22d6c54c91c5bc4f3ff81 client 10.0.3.182:51160 read
    Oct 26 2026 20:51:00 GMT: DETAIL (batch-sub): (thr_tsvc.c:108) {test} digest 690f4b1eb7ef704dc79268a3b62fa9609280a2ea client 10.0.3.190:48548 read

    Count transactions per record and transaction type. If the node logs to the journal instead of a file, pipe journalctl -u aerospike -a -o cat into the grep command. Omit the log filename from grep.

    Terminal window
    grep -E 'DETAIL \((rw-client|batch-sub)\)' /var/log/aerospike/aerospike.log \
      | awk '{ for (i = 1; i <= NF; i++) if ($i == "digest") { print $(i - 1), $(i + 1), $NF; break } }' \
      | sort | uniq -c | sort -rn | head -10

    Example output:

     9214 {test} 8df238affec6f8e3a2c22d6c54c91c5bc4f3ff81 read
       12 {test} 2222b1c4e9f0a7d3c6b5e8f1a2d4c7b0e3f6a9d2 read
       11 {test} 690f4b1eb7ef704dc79268a3b62fa9609280a2ea read

    A hot key has a count orders of magnitude higher than the other records. If the counts are uniform while the indicators remain on one node, the cause is more likely a slow node than a hot key.

    Keep the following in mind when you read the lines:

    • The client address includes the source port, which differs per connection. Group by IP address to find the application host.
    • The lines don’t include the set name or the user key.
    • The filenames, line numbers, and message format can change between releases. Check a few lines from your version before you rely on a script.

  4. If you are unable to change log levels on the node, capture request packets. Use this method only as a last resort.

    The capture has no readable digests when the client connects with TLS, or for requests that the client compresses. It also misses batch requests, which carry their digests in a different layout.

    Capture inbound requests on the service port for at most 30 seconds or 1,000,000 packets, keeping the first 512 bytes of each packet:

    Terminal window
    timeout 30 tcpdump -i any -n -s 512 -c 1000000 -w /tmp/hotkey.pcap 'tcp dst port 3000'

    Replace 3000 if the service listens on a different port. The first 512 bytes of a request hold the message header and the digest field, and leave out most of the data in larger writes.

    Count digests in the capture:

    Terminal window
    tcpdump -r /tmp/hotkey.pcap -n -x \
      | grep -E '^[[:space:]]+0x[0-9a-f]+:' \
      | cut -d ' ' -f 3-10 \
      | tr -d ' \n' \
      | grep -oE '00001504[0-9a-f]{40}' \
      | cut -c 9- \
      | sort | uniq -c | sort -rn | head -10

    The pipeline joins the hexadecimal dump of every packet into one string, so the counts are approximate. Confirm a candidate digest with rw-client logging when you can.

    Delete the capture file:

    Terminal window
    rm /tmp/hotkey.pcap

  5. Optional: find the record that the digest belongs to. The digest is a one-way hash of the set name and user key, so the user key cannot be computed from it. To see the record, read it by digest, as described in Get a record using DIGEST. For example, use Aerospike Quick Look (AQL):

    Terminal window
    aql -h NODE_IP -c "SELECT * FROM NAMESPACE WHERE DIGEST='DIGEST_HEX'"

    Replace DIGEST_HEX with the 40-character digest. The record includes the user key only if the application wrote it with the send key policy. Use the bin values and the key, if present, to find the code path in the application that reaches the record.

Reduce the load on a hot key

A hot key comes from the application’s access pattern, so the lasting fix is in the data model. The client and server settings in this section reduce the impact while you change the application.

  • Shard the record. Split a write hot key into several sub-records, or keep several copies of a read hot key, so the load spreads across partitions and nodes. See Hot-key sharding.

  • Spread reads across replicas. With the client replica policy MASTER_PROLES, the client sends AP reads to the master and replica nodes in turn. The C, Python, and Node.js clients name this policy ANY. This pattern divides read load by up to the replication factor. A read from a replica can return an earlier version of the record. In SC namespaces, replicas serve reads only with the ALLOW_REPLICA or ALLOW_UNAVAILABLE SC read mode.

  • Shorten the time each write spends in the rw-hash. In AP namespaces, set write-commit-level-override to master. The server responds without waiting for replicas, so the next write to the record completes sooner. However, a write that returns success to the client can be lost if the master node fails before the replicas apply it. The override doesn’t apply to batch writes or background query writes, and SC namespaces reject it.

  • Serve reads from memory with read-page-cache. read-page-cache lets record reads use the operating system’s page cache. The page cache can serve repeated reads of a hot key from memory. Before you enable it, consider the following:

    • Test the change on a single node before you roll it out.
    • Don’t enable it for namespaces with data-in-memory set to true.
    • In a container with cgroup-mem-tracking enabled, page cache can count toward stop-writes-sys-memory-pct.
    • A uniform read workload with no hot keys might not benefit, and spends extra CPU.
    • cache_read_pct doesn’t count page cache hits, so measure the effect with read latency.

  • Size post-write-cache for read-after-write patterns. post-write-cache keeps recently written data in memory. Reads of recently written records can be served from memory instead of the device. The cache helps a hot key that is also updated often, because each update writes the record to a new write block. The size is per device, a multiple of 8 MiB, and 256 MiB by default. Write blocks in the post-write cache aren’t eligible for defragmentation, so keep the cache small compared to the device size.

  • Change transaction-pending-limit only as a temporary measure. Raising the limit lets more transactions remain pending instead of failing with KEY_BUSY. It doesn’t reduce the load on the record, and the pending transactions add latency and can time out. Lowering the limit to one or two fails transactions sooner. Fewer transactions stay pending on the server, but clients receive more errors. A value of 0 removes the limit.

    Terminal window
    asadm -e "enable; manage config namespace NAMESPACE param transaction-pending-limit to 25"

Verify the fix

After you change the application or the configuration, confirm the following on every node:

  • Latency and operations per second are similar across nodes.

    Terminal window
    asadm -e "show latencies for NAMESPACE"

  • Client connections are similar across nodes.

    Terminal window
    asadm -e "show statistics service like client_connections"

  • fail_key_busy stops growing.

    Terminal window
    asadm -e "watch 10 6 show statistics namespace for NAMESPACE like fail_key_busy"

  • With read benchmark histograms enabled, the read-restart and read-local slices drop on the node that served the hot key. The read-restart slice is the time spent in the rw-hash, and the read-local slice is the time spent reading the record. For writes, check the write-restart slice in write benchmark histograms.

    Terminal window
    asadm -e "enable; manage config namespace NAMESPACE param enable-benchmarks-read to true"
    asadm -e "show latencies -v for NAMESPACE like read"

    Set enable-benchmarks-read back to false when you finish.

  • A new 10-second rw-client sample no longer has one digest with a count far above the rest.