--- title: "Diagnose hot keys" description: "Find the record behind uneven node load or KEY_BUSY errors in Aerospike Database, for write hot keys, strong consistency reads, and AP reads." --- # Diagnose hot keys > For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt) > > All documentation pages available in markdown. This page explains how to find a hot key and reduce the load it puts on one node. A hot key is a single record that receives a disproportionate share of requests. **Applies to:** Aerospike Database Community Edition and Enterprise Edition, in Available and Partition-tolerant (AP) and strong consistency (SC) namespaces. SC namespaces require Enterprise Edition. **Audience:** Operators and site reliability engineers who troubleshoot uneven node load or `KEY_BUSY` errors in a production cluster. **Outcome:** You know the namespace, digest, and serving node of the hot key, or you have ruled out a hot key as the cause of the imbalance. ## Prerequisites - [`asadm`](https://aerospike.com/docs/database/tools/asadm) and [`asinfo`](https://aerospike.com/docs/database/tools/asinfo) access to every node in the cluster. - If security is enabled, a user with the `sys-admin` role, which is required for [`log-set`](https://aerospike.com/docs/database/reference/info#log-set) and configuration changes. - Read access to the server log on each node. In Database 8.1.0 and later, the packaged configuration file logs to the console, and systemd stores the log in the journal. Read it with `journalctl -u aerospike -a -o cat`. If the node runs in a container and logs to the console, read the log through your container runtime. ::: note Prior to Database 8.1.0, the packaged configuration file logs to `/var/log/aerospike/aerospike.log`. ::: - For packet capture only: `tcpdump` on the node and packet-capture privileges. ## How hot keys cause node imbalance - Each record belongs to one of 4,096 partitions, and each partition has one master node. - All writes to a record go to the master node for its partition. A write hot key also adds replica-write load to the nodes that hold replicas of the record’s partition. - With the default [replica policy](https://aerospike.com/docs/database/learn/policies#replica), `SEQUENCE`, each read also goes to the master first. - A hot key therefore concentrates load on one node, while the load on nodes that don’t hold the record stays normal. For example, suppose an application counts views for a whole site by incrementing one `site-views` record on every page request. Every increment goes to the master node for that record’s partition. That node’s write latency and throughput rise, while nodes with no copy of the partition stay level. At peak traffic, more than 20 increments are pending for the record at once. Clients therefore start to receive `KEY_BUSY` errors, and [`fail_key_busy`](https://aerospike.com/docs/database/reference/metrics#namespace__fail_key_busy) grows on that node. If you enable [`key-busy`](https://aerospike.com/docs/database/reference/logs#key-busy__1634172377) detail logging on the node, the log includes the digest of the `site-views` record. Responses from that node take longer, so more requests are in flight and clients open more connections to the node, which makes the imbalance worse. ### Which transactions can fail with KEY\_BUSY The server tracks in-progress transactions in the _rw-hash_. When a transaction for a record is already in the rw-hash, the next transaction for the same record is pending until the first one completes. When more than [`transaction-pending-limit`](https://aerospike.com/docs/database/reference/config#namespace__transaction-pending-limit) transactions are already pending, the server does the following: - Fails the new transaction with `KEY_BUSY` (error code 14). - Increments [`fail_key_busy`](https://aerospike.com/docs/database/reference/metrics#namespace__fail_key_busy). Whether a transaction can fail this way depends on its type and the namespace mode: | Transaction | Can fail with `KEY_BUSY` | | --- | --- | | Writes, deletes, and User-Defined Function (UDF) calls, including batch writes and background query operations, in any namespace | Yes | | Reads in an AP namespace | Only during migrations, when the [read consistency level](https://aerospike.com/docs/database/learn/policies#ap-read-mode) is `all` | | Linearized reads and multi-record transaction (MRT) read verification in an SC namespace, with replication factor 2 or more | Yes | | Other reads in an SC namespace | Only during migrations, or while a write to the same record is still replicating | A read hot key in a stable AP namespace therefore never fails with `KEY_BUSY` and leaves `fail_key_busy` flat. Raising `transaction-pending-limit` doesn’t change that. For the complete rules, see [`fail_key_busy`](https://aerospike.com/docs/database/reference/metrics#namespace__fail_key_busy). ### Indicators of a hot key Compare each node with its peers. The indicators of a hot key usually appear on one node. - Higher latency on one node. - Higher throughput, in operations per second, on one node. - More client connections on one node, in the [`client_connections`](https://aerospike.com/docs/database/reference/metrics#node_stats__client_connections) statistic. Slower responses mean more requests in flight on that node, so clients open more connections to it. - Higher device or network utilization on one node, for example in `iostat` or `sar` output. - The imbalance moves to another node after you [quiesce](https://aerospike.com/docs/database/manage/cluster/quiesce-node) or restart the node, because the master role for the hot key’s partition moves with it. Use the following commands to compare latency, throughput, and connections across nodes. Replace _`NAMESPACE`_ with the namespace name. Terminal window ```bash asadm -e "show latencies for NAMESPACE" asadm -e "show statistics service like client_connections" ``` A slow node, such as one with a failing device or a network problem, has similar indicators. The difference is that a slow node’s indicators remain on that node after its partitions move to other nodes. For slow-node causes, see [Node problems](https://aerospike.com/docs/database/advanced/troubleshoot/node). ## Identify the hot key Run the following steps in order, and stop at the first step that gives you a digest. Steps 3 and 4 add load to the node. Run them only on the affected node and only for a few seconds. In the commands, replace _`NODE_IP`_ with the address of the affected node. Replace _`SINK_ID`_ with the ID of the log sink to change. Add your usual `asadm` and `asinfo` connection and authentication options. This page uses the `log-set` info command. `set-log` is a synonym that some Support articles use. 1. Check whether `fail_key_busy` is growing. Terminal window ```bash asadm -e "watch 10 6 show statistics namespace for NAMESPACE like fail_key_busy" ``` `fail_key_busy` counts failures since the node started, so look for a value that grows between iterations, and compare nodes. Each node also prints the [`special-errors`](https://aerospike.com/docs/database/reference/logs#info__994802748) ticker line every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval) when any of its counters is nonzero: ```text Oct 26 2026 20:50:06 GMT: INFO (info): (ticker.c:1209) {test} special-errors: key-busy (1234,0) record-too-big 0 lost-conflict (0,0) ``` The first number after `key-busy` is `fail_key_busy`. The second number is [`fail_xdr_key_busy`](https://aerospike.com/docs/database/reference/metrics#namespace__fail_xdr_key_busy), which counts a different error that cross-datacenter replication (XDR) raises. If `fail_key_busy` grows, continue with step 2. If it stays flat while the indicators persist, the hot transactions aren’t failing with `KEY_BUSY`. They can be reads, or writes that stay under the limit. Skip to step 3, which logs every transaction. 2. Log the digests of failing transactions with the `key-busy` context. List the node’s log sinks and their IDs, record the current `key-busy` level of the sink, then enable `detail` level for `key-busy`: Terminal window ```bash asinfo -h NODE_IP -v "logs" asinfo -h NODE_IP -v "log/SINK_ID/key-busy" asinfo -h NODE_IP -v "log-set:id=SINK_ID;key-busy=detail" ``` The server logs the first failure of each combination of namespace, digest, transaction type, and source right away. At each ticker interval (10 seconds by default), it logs one `(repeated:N)` line per combination, where N counts that combination’s failures since the previous interval: ```text Oct 26 2026 20:50:06 GMT: DETAIL (key-busy): (rw_request_hash.c:224) {test} 8df238affec6f8e3a2c22d6c54c91c5bc4f3ff81 write from 10.0.3.182:51160 Oct 26 2026 20:50:16 GMT: DETAIL (key-busy): (rw_request_hash.c:224) (repeated:57) {test} 8df238affec6f8e3a2c22d6c54c91c5bc4f3ff81 write from 10.0.3.182:51160 ``` Each line shows the namespace in braces, the 40-character hexadecimal digest, the transaction type, and the source. The source is the client address and port, or `proxy`, `bg-udf`, or `bg-ops` for proxied and background transactions. For the full list of values, see [key-busy detail](https://aerospike.com/docs/database/reference/logs#key-busy__1634172377). After at least one ticker interval, count failures per record by adding up the `(repeated:N)` lines. If the node logs to the journal instead of a file, pipe `journalctl -u aerospike -a -o cat` into the `grep` command. Omit the log filename from `grep`. Terminal window ```bash grep -E 'DETAIL \(key-busy\).*\(repeated:[0-9]+\)' /var/log/aerospike/aerospike.log \ | awk '{ for (i = 1; i <= NF; i++) { if ($i ~ /^\(repeated:[0-9]+\)$/) { n = substr($i, 11, length($i) - 11) } if ($i ~ /^\{.+\}$/) { count[$i " " $(i + 1) " " $(i + 2)] += n; break } } } END { for (k in count) print count[k], k }' \ | sort -rn | head -10 ``` Example output: ```text 57 {test} 8df238affec6f8e3a2c22d6c54c91c5bc4f3ff81 write 1 {test} 1f0e3c9b4a7d2e6f8a1b3c5d7e9f0a2b4c6d8e0f batch-sub-write ``` The record with the highest count is the hot key. When you have the digest, restore the level that you recorded. Replace _`RECORDED_LEVEL`_ with that level in the following command: Terminal window ```bash asinfo -h NODE_IP -v "log-set:id=SINK_ID;key-busy=RECORDED_LEVEL" ``` 3. Sample every transaction with the `rw-client` or `batch-sub` context. These contexts log both successful and failed transactions, so they also find hot keys that never fail with `KEY_BUSY`. The two contexts log different transactions: - `rw-client` logs single-record client transactions. - `batch-sub` logs batch subtransactions. Use `batch-sub` when the application reaches the hot key through batch requests. ::: caution `rw-client` and `batch-sub` detail logging writes one line for every transaction on the node. At 100,000 transactions per second, that’s more than 10 MB of log per second, which adds CPU and disk load and can fill the log volume. Enable it on one node at a time, for no more than about 10 seconds. ::: The following command enables `rw-client` detail logging for 10 seconds and then sets the level back to `info`: Terminal window ```bash asinfo -h NODE_IP -v "log-set:id=SINK_ID;rw-client=detail"; sleep 10; asinfo -h NODE_IP -v "log-set:id=SINK_ID;rw-client=info" ``` If `asinfo -h NODE_IP -v "log/SINK_ID/rw-client"` returns a level other than `info` before you start, use that level in the last command. If you interrupt the command, run the last `asinfo` command yourself to restore the level. For batch subtransactions, replace `rw-client` with `batch-sub` in both `log-set` commands. To keep these lines out of the main log, define a separate log sink for the context, as described in [Create a log sink for debugging](https://aerospike.com/docs/database/manage/logging/logs#create-a-log-sink-for-debugging). A new log sink takes effect only after the node restarts, so set it up before you need it. The lines look like the following: ```text Oct 26 2026 20:51:00 GMT: DETAIL (rw-client): (thr_tsvc.c:103) {test} digest 8df238affec6f8e3a2c22d6c54c91c5bc4f3ff81 client 10.0.3.182:51160 read Oct 26 2026 20:51:00 GMT: DETAIL (batch-sub): (thr_tsvc.c:108) {test} digest 690f4b1eb7ef704dc79268a3b62fa9609280a2ea client 10.0.3.190:48548 read ``` Count transactions per record and transaction type. If the node logs to the journal instead of a file, pipe `journalctl -u aerospike -a -o cat` into the `grep` command. Omit the log filename from `grep`. Terminal window ```bash grep -E 'DETAIL \((rw-client|batch-sub)\)' /var/log/aerospike/aerospike.log \ | awk '{ for (i = 1; i <= NF; i++) if ($i == "digest") { print $(i - 1), $(i + 1), $NF; break } }' \ | sort | uniq -c | sort -rn | head -10 ``` Example output: ```text 9214 {test} 8df238affec6f8e3a2c22d6c54c91c5bc4f3ff81 read 12 {test} 2222b1c4e9f0a7d3c6b5e8f1a2d4c7b0e3f6a9d2 read 11 {test} 690f4b1eb7ef704dc79268a3b62fa9609280a2ea read ``` A hot key has a count orders of magnitude higher than the other records. If the counts are uniform while the indicators remain on one node, the cause is more likely a slow node than a hot key. Keep the following in mind when you read the lines: - The client address includes the source port, which differs per connection. Group by IP address to find the application host. - The lines don’t include the set name or the user key. - The filenames, line numbers, and message format can change between releases. Check a few lines from your version before you rely on a script. 4. If you are unable to change log levels on the node, capture request packets. Use this method only as a last resort. ::: caution Packet capture requires root or the `CAP_NET_RAW` capability, and the capture file can contain user keys and record data. Follow your organization’s policy for capturing production traffic, and delete the capture file when you finish. ::: The capture has no readable digests when the client connects with TLS, or for requests that the client compresses. It also misses batch requests, which carry their digests in a different layout. Capture inbound requests on the service port for at most 30 seconds or 1,000,000 packets, keeping the first 512 bytes of each packet: Terminal window ```bash timeout 30 tcpdump -i any -n -s 512 -c 1000000 -w /tmp/hotkey.pcap 'tcp dst port 3000' ``` Replace `3000` if the service listens on a different port. The first 512 bytes of a request hold the message header and the digest field, and leave out most of the data in larger writes. Count digests in the capture: Terminal window ```bash tcpdump -r /tmp/hotkey.pcap -n -x \ | grep -E '^[[:space:]]+0x[0-9a-f]+:' \ | cut -d ' ' -f 3-10 \ | tr -d ' \n' \ | grep -oE '00001504[0-9a-f]{40}' \ | cut -c 9- \ | sort | uniq -c | sort -rn | head -10 ``` The pipeline joins the hexadecimal dump of every packet into one string, so the counts are approximate. Confirm a candidate digest with `rw-client` logging when you can. Delete the capture file: Terminal window ```bash rm /tmp/hotkey.pcap ``` 5. Optional: find the record that the digest belongs to. The digest is a one-way hash of the set name and user key, so the user key cannot be computed from it. To see the record, read it by digest, as described in [Get a record using DIGEST](https://aerospike.com/docs/database/tools/aql/records#get-a-record-using-digest). For example, use Aerospike Quick Look (AQL): Terminal window ```bash aql -h NODE_IP -c "SELECT * FROM NAMESPACE WHERE DIGEST='DIGEST_HEX'" ``` Replace _`DIGEST_HEX`_ with the 40-character digest. The record includes the user key only if the application wrote it with the [send key](https://aerospike.com/docs/database/learn/policies#send-key) policy. Use the bin values and the key, if present, to find the code path in the application that reaches the record. ## Reduce the load on a hot key A hot key comes from the application’s access pattern, so the lasting fix is in the data model. The client and server settings in this section reduce the impact while you change the application. - **Shard the record.** Split a write hot key into several sub-records, or keep several copies of a read hot key, so the load spreads across partitions and nodes. See [Hot-key sharding](https://aerospike.com/docs/develop/data-modeling/key-design#hot-key-sharding). - **Spread reads across replicas.** With the client [replica policy](https://aerospike.com/docs/database/learn/policies#replica) `MASTER_PROLES`, the client sends AP reads to the master and replica nodes in turn. The C, Python, and Node.js clients name this policy `ANY`. This pattern divides read load by up to the replication factor. A read from a replica can return an earlier version of the record. In SC namespaces, replicas serve reads only with the `ALLOW_REPLICA` or `ALLOW_UNAVAILABLE` [SC read mode](https://aerospike.com/docs/database/learn/policies#sc-read-mode). - **Shorten the time each write spends in the rw-hash.** In AP namespaces, set [`write-commit-level-override`](https://aerospike.com/docs/database/reference/config#namespace__write-commit-level-override) to `master`. The server responds without waiting for replicas, so the next write to the record completes sooner. However, a write that returns success to the client can be lost if the master node fails before the replicas apply it. The override doesn’t apply to batch writes or background query writes, and SC namespaces reject it. - **Serve reads from memory with `read-page-cache`.** [`read-page-cache`](https://aerospike.com/docs/database/reference/config#namespace__read-page-cache) lets record reads use the operating system’s page cache. The page cache can serve repeated reads of a hot key from memory. Before you enable it, consider the following: - Test the change on a single node before you roll it out. - Don’t enable it for namespaces with [`data-in-memory`](https://aerospike.com/docs/database/reference/config#namespace__data-in-memory) set to `true`. - In a container with [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking) enabled, page cache can count toward [`stop-writes-sys-memory-pct`](https://aerospike.com/docs/database/reference/config#namespace__stop-writes-sys-memory-pct). - A uniform read workload with no hot keys might not benefit, and spends extra CPU. - [`cache_read_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__cache_read_pct) doesn’t count page cache hits, so measure the effect with read latency. - **Size `post-write-cache` for read-after-write patterns.** [`post-write-cache`](https://aerospike.com/docs/database/reference/config#namespace__post-write-cache) keeps recently written data in memory. Reads of recently written records can be served from memory instead of the device. The cache helps a hot key that is also updated often, because each update writes the record to a new write block. The size is per device, a multiple of 8 MiB, and 256 MiB by default. Write blocks in the post-write cache aren’t eligible for defragmentation, so keep the cache small compared to the device size. - **Change `transaction-pending-limit` only as a temporary measure.** Raising the limit lets more transactions remain pending instead of failing with `KEY_BUSY`. It doesn’t reduce the load on the record, and the pending transactions add latency and can time out. Lowering the limit to one or two fails transactions sooner. Fewer transactions stay pending on the server, but clients receive more errors. A value of 0 removes the limit. ::: note [Tools package 6.0.3](https://aerospike.com/docs/database/tools/release-notes#tools-603) and later is required to use asadm’s [manage config](https://aerospike.com/docs/database/tools/asadm/live-mode/#manage) commands. Otherwise, use the equivalent [asinfo - set-config](https://aerospike.com/docs/database/reference/info#set-config) command. ::: Terminal window ```bash asadm -e "enable; manage config namespace NAMESPACE param transaction-pending-limit to 25" ``` ## Verify the fix After you change the application or the configuration, confirm the following on every node: - Latency and operations per second are similar across nodes. Terminal window ```bash asadm -e "show latencies for NAMESPACE" ``` - Client connections are similar across nodes. Terminal window ```bash asadm -e "show statistics service like client_connections" ``` - `fail_key_busy` stops growing. Terminal window ```bash asadm -e "watch 10 6 show statistics namespace for NAMESPACE like fail_key_busy" ``` - With [read benchmark histograms](https://aerospike.com/docs/database/observe/latency#read-transaction-analysis) enabled, the `read-restart` and `read-local` slices drop on the node that served the hot key. The `read-restart` slice is the time spent in the rw-hash, and the `read-local` slice is the time spent reading the record. For writes, check the `write-restart` slice in [write benchmark histograms](https://aerospike.com/docs/database/observe/latency#write-transaction-analysis). Terminal window ```bash asadm -e "enable; manage config namespace NAMESPACE param enable-benchmarks-read to true" asadm -e "show latencies -v for NAMESPACE like read" ``` Set `enable-benchmarks-read` back to `false` when you finish. - A new 10-second `rw-client` sample no longer has one digest with a count far above the rest. ## Related pages - [Key design](https://aerospike.com/docs/develop/data-modeling/key-design) - [Client errors](https://aerospike.com/docs/database/advanced/troubleshoot/client) - [Error codes](https://aerospike.com/docs/database/reference/error-codes) - [Monitor latency with histograms](https://aerospike.com/docs/database/observe/latency) - [Log management](https://aerospike.com/docs/database/manage/logging/logs) - Aerospike Support: [Why does my client return Error code 14 Hot key?](https://support.aerospike.com/hc/en-us/articles/49938493644315-Why-does-my-client-return-Error-code-14-Hot-key) - Aerospike Support: [How To identify read hotkeys](https://support.aerospike.com/hc/en-us/articles/49939303451547-How-To-identify-read-hotkeys) - Aerospike Support: [High client connections on certain nodes due to underperformance](https://support.aerospike.com/hc/en-us/articles/49939401809563-High-client-connections-on-certain-nodes-due-to-underperformance) - Aerospike Support: [Buffering and Caching in Aerospike](https://support.aerospike.com/hc/en-us/articles/49938335148315-Buffering-and-Caching-in-Aerospike)