--- title: "Configure wire compression" description: "Reduce intra-cluster replica-write and migration network traffic with wire compression in Aerospike Database Enterprise Edition." --- # Configure wire compression > For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt) > > All documentation pages available in markdown. This page describes how to enable wire compression for replica writes and partition migrations in Aerospike Database Enterprise Edition 8.2.0 and later. **Applies to:** Aerospike Database Enterprise Edition 8.2.0 and later. **Audience:** Cluster operators and SREs who manage replication bandwidth, migration traffic, or cross-Availability Zone (AZ) network cost. **Outcome:** You enable and tune wire compression for replica writes and partition migrations, then confirm bandwidth savings with namespace statistics. ## Prerequisites - Aerospike Database Enterprise Edition 8.2.0 or later on every node that participates in replication for the namespace. Wire compression requires Enterprise Edition. It is not gated by a feature key entry. - Write access to namespace configuration (`aerospike.conf`, [`asadm`](https://aerospike.com/docs/database/tools/asadm), or [`asinfo`](https://aerospike.com/docs/database/tools/asinfo)). - Familiarity with [namespace replication](https://aerospike.com/docs/database/reference/config#namespace__replication-factor) and [rack awareness](https://aerospike.com/docs/database/manage/namespace/rack-aware) for multi-AZ deployments. ::: caution Wire compression directives in `aerospike.conf` are exclusive to the Enterprise Edition. Including any of the following parameters in a Community Edition node causes a crash at startup: - `replication-compression-mode` - `replication-compression-level` - `migrate-compression-mode` - `migrate-compression-level` - `smd-compression-mode` - `smd-compression-level` Older Database builds that do not recognize these settings also crash on launch with an unknown config parameter name error. `enable-benchmarks-wire-compression` is the exception: Community Edition accepts it, where it has no effect. ::: ## Overview Wire compression applies Zstd to replica writes and partition migrations on the fabric link between cluster nodes. Replica-write and migration settings are per namespace. Wire compression is a cost trade, not a performance feature. It spends CPU on the sending and receiving nodes, and can reduce peak write throughput, in exchange for fewer bytes on the fabric. That trade only pays where those bytes are billed, such as a [rack awareness](https://aerospike.com/docs/database/manage/namespace/rack-aware) deployment whose replicas cross AZ boundaries where cloud providers charge for transfer in both directions. In a single-AZ cluster the fabric traffic is free, so compressing it buys nothing for the CPU it costs. The same namespace `replication-compression-mode` settings also apply to read-touch replica traffic (TTL-extension writes that replicate to replicas). ### Storage compression vs wire compression [Storage compression](https://aerospike.com/docs/database/manage/namespace/storage/compression) compresses records on disk or in memory/PMem storage. Wire compression compresses payloads on the fabric link between nodes. The two features are independent: | Traffic | Scope | Mode setting | Level setting | | --- | --- | --- | --- | | Storage compression | On-disk (or in-memory storage) record format | [`compression`](https://aerospike.com/docs/database/reference/config#namespace__compression) | [`compression-level`](https://aerospike.com/docs/database/reference/config#namespace__compression-level) | | Replica writes | Per namespace | `replication-compression-mode` | `replication-compression-level` | | Partition migrations | Per namespace | `migrate-compression-mode` | `migrate-compression-level` | A record already compressed at the storage layer crosses the fabric in its stored, compressed form. Wire compression does not run on it a second time, so its bytes never contribute to the wire compression `bytes_saved` metrics. Those bytes are already being saved by storage compression. The more of a namespace’s records are stored compressed, the less wire compression has left to save. This decision is made per record, not per namespace. A record is stored compressed only when compressing it actually made it smaller, so a namespace with storage compression enabled always holds some records stored plain, and those are still compressed on the wire. A namespace that enabled storage compression recently holds many of them: records written before then stay stored plain until they are next written. ::: note Wire compression is intra-cluster only, operating on the fabric link between nodes of the same cluster. Existing client configuration and drivers continue to work unchanged. It does not cover records shipped to another cluster by [XDR](https://aerospike.com/docs/database/learn/architecture/xdr). XDR compresses what it ships using its own per-destination-namespace settings, [`enable-compression`](https://aerospike.com/docs/database/reference/config#xdr__enable-compression), [`compression-level`](https://aerospike.com/docs/database/reference/config#xdr__compression-level), and [`compression-threshold`](https://aerospike.com/docs/database/reference/config#xdr__compression-threshold). If cross-region XDR egress is the cost you are targeting, configure those instead. ::: Service-wide SMD full-sync wire compression uses separate `smd-compression-mode` parameters in the `service` context. See [Configure SMD wire compression](https://aerospike.com/docs/database/manage/network/smd-compression). ### When to use wire compression Wire compression helps when: - Replicas cross AZ boundaries and network egress is a significant cost (common on cloud infrastructure deployments). - Write volume or record size drives high steady-state replication bandwidth. - Partition [migrations](https://aerospike.com/docs/database/manage/cluster/migrations) after topology changes generate large, bursty network traffic. - Records are large but updates change only a small portion of each record (`delta-zstd` mode). Wire compression is applied on a per-record basis. Each replica write pays for compressing the record on the partition master and decompressing it on the replica. The master-side compression is on the client’s write path at any commit level, because it happens as part of the write itself. At write commit level `all`, the replica’s decompression is on that path too, because the client is not answered until the replica acknowledges. The exposure is larger there, and write latency can increase. Because every replicated record pays this cost, the cumulative effect can be lower overall write throughput. Measure it on your own workload rather than assuming a fixed cost. The impact varies with the compute environment, record size, and load. On a namespace where storage compression is already working, most records are skipped on the wire. The ones that are not skipped are the ones storage compression could not shrink, so you pay the compressor’s full cost on them for little return. Expect [`repl_wire_compression_not_beneficial`](https://aerospike.com/docs/database/reference/metrics#namespace__repl_wire_compression_not_beneficial) to climb rather than `bytes_saved`. On a namespace that uses multi-record transactions, each record in a committed transaction is replicated twice, once when the transaction writes it, and again when the transaction commits. Only the first is compressed on the wire. Expect roughly half the saving the same records would give outside a transaction, and judge a pilot on such a namespace accordingly. How much bandwidth wire compression saves is entirely dependent on your data, so do not assume a particular reduction in replication traffic or network cost before measuring it. With `zstd`, savings depend on record size and on how compressible the record contents are. With `delta-zstd`, savings depend on how many bytes each update changes relative to the size of the whole record. If your updates change a known, consistent amount of data, you can estimate the saving up front. If update sizes vary widely, the only reliable guide is observing `repl_wire_compression_bytes_saved` against `fabric_rw_bytes_sent` over time. In either case, make the decision empirically, per namespace, using the [Evaluate wire compression](#evaluate-wire-compression) procedure. See [Evaluate wire compression](#evaluate-wire-compression) before enabling the feature on additional namespaces. ## Choose a compression mode Use the following tables to pick `replication-compression-mode` and `migrate-compression-mode` values for each namespace. ### Replica writes | Mode | What it sends | Enable when | Skip when | | --- | --- | --- | --- | | `none` | The whole record, uncompressed. The default. | — | — | | `zstd` | Each replica write compressed on its own. | Replica payloads are large enough to compress and are compressible (JSON, text, repeated structure). Works under any effective write commit level, on creates as well as updates, and needs nothing from the replica. | Records are already compressed at the storage layer, records are small, data is high-entropy or pre-compressed, or replication does not cross a billed boundary. | | `delta-zstd` | Only the bytes that changed since the replica’s copy of the record. | Records are large and only partly updated. A few fields changing in a multi-KB record can ship as tens of bytes, well below compressing the whole record. The replica reliably holds the prior version. | Single-record client writes commit at write commit level `master` and the namespace has little batch, background-job, read-touch, or XDR-received write traffic. Delta patches are disabled for those writes, though plain Zstd still applies. The workload is create-heavy or write-once. Replica read I/O is already the bottleneck. | ### Partition migrations | Setting | Enable when | Skip when | | --- | --- | --- | | `migrate-compression-mode = zstd` | Recovery windows after node events are long or costly, or migration bandwidth is the binding constraint. Worth enabling even when replication compression is off. | Migrations are rare and small, or CPU headroom during migration is already tight. | ## Compression modes for replica writes ### Plain `zstd` The partition master compresses each replicated record independently when the record is at least 256 bytes. Smaller records are sent uncompressed, because compressing them does not reliably save anything. [`replication-compression-level`](#compression-levels) sets the compression strength. This 256-byte floor applies to plain `zstd` on both replica writes and partition migrations. Delta patches are sent regardless of record size. ### `delta-zstd` The partition master sends only the difference between the new record and the version it held before the write. The wire payload is a patch, often much smaller than a full Zstd-compressed record. The replica must already hold that same base version, identified by matching generation and last update time. If the base version does not match, the replica signals a version mismatch, the partition master drops the delta, and retransmits the full record as an uncompressed plain replica write. This fallback is transparent to clients. To rebuild the record, the replica loads its copy of the prior version. Delta patches are disabled for direct single-record client writes whose effective write commit level is `master`. Batch writes, background UDF and background operations jobs, read-touch TTL extensions, writes a client sent to a non-master node that were forwarded to the master, and writes received from another cluster by XDR still build deltas regardless of commit level. The namespace [`write-commit-level-override`](https://aerospike.com/docs/database/reference/config#namespace__write-commit-level-override) fixes commit level at the namespace. When the override is `off`, the client [write policy](https://aerospike.com/docs/database/learn/policies#write-commit-level) supplies the commit level for the single-record writes it governs. Namespaces with strong consistency use `write-commit-level all`, so `delta-zstd` applies normally. ::: caution In Available and Partition-tolerant (AP) namespaces where the effective write commit level is `master`, `delta-zstd` disables only delta patches. Plain Zstd compression still runs, so `repl_wire_compression_bytes_saved` keeps climbing while `repl_wire_compression_delta_attempts` stops increasing. Nothing is logged. `repl_wire_compression_delta_hit_pct` does not reveal this. It is a cumulative lifetime ratio of `delta_hits` to `delta_attempts`, so it reads `0.000` on a namespace that never produced a delta but holds its last value, often `100.000`, on a namespace that produced deltas and then stopped. The attempt rate can also stay misleadingly high for the wrong reason such as read-touch TTL extensions building deltas regardless of commit level. Because a touch changes very little of the record, those patches are small and nearly always succeed. As a result, both counters can look good while no client write has produced a delta. Read-touch can be enabled for a single set with [`default-read-touch-ttl-pct`](https://aerospike.com/docs/database/reference/config#namespace__default-read-touch-ttl-pct). Compare the attempt rate against your single-record client write rate specifically, not against total writes. Watch the rate of `repl_wire_compression_delta_attempts` against that rate instead, and check [`write-commit-level-override`](https://aerospike.com/docs/database/reference/config#namespace__write-commit-level-override) and client write policy before concluding the workload is unsuitable for `delta-zstd`. ::: See [Tuning signals](#tuning-signals) for when to switch from `delta-zstd` to `zstd`. ## Configure wire compression Replica writes and partition migrations have separate compression controls. Both require Aerospike Database Enterprise Edition and support dynamic configuration. | Traffic | Mode setting | Level setting | Mode values | Default mode | Default level | | --- | --- | --- | --- | --- | --- | | Replica writes | `replication-compression-mode` | `replication-compression-level` | `none`, `zstd`, `delta-zstd` | `none` | `1` | | Partition migrations | `migrate-compression-mode` | `migrate-compression-level` | `none`, `zstd` | `none` | `1` | Level parameters accept `-10` through `22` for both traffic types. See [Compression levels](#compression-levels). A higher `migrate-compression-level` is more defensible than a higher `replication-compression-level`: migration is bulk transfer, while replication sits in the synchronous write path, so every client write waits on the replication level you choose. Configure the same namespace-level mode and level on every node for a given namespace. A node fails to start if `aerospike.conf` contains an invalid mode or level. A dynamic `set-config` with an invalid value fails and leaves the current setting in place. ### Compression levels `replication-compression-level` and `migrate-compression-level` set the Zstd compression level. Level behavior is the same for both parameters. | Level | Behavior | | --- | --- | | Negative (`-10` through `-1`) | Zstd _fast compression_: faster encoding, lower compression ratio. More negative values increase speed at the cost of ratio. | | `0` | Selects Zstd’s own default level, which is `3`, not the Aerospike default of `1`. | | Positive (`1` through `22`) | Standard Zstd compression levels range from higher speed at lower values to higher compression ratios at higher values. | | Default `1` | Low positive level that favors speed over maximum compression. | Setting a level of `1` favors speed over compression ratio in the synchronous write path. In contrast, setting the level to `0` enables Zstd’s internal default level of `3` rather than disabling compression or using the Aerospike default, causing the node to compress harder and more slowly. Always set the level explicitly to `1` rather than leaving it at `0`. `replication-compression-level` applies when `replication-compression-mode` is `zstd` or `delta-zstd`. `migrate-compression-level` applies when `migrate-compression-mode` is `zstd`. These level parameters are separate from storage [`compression-level`](https://aerospike.com/docs/database/reference/config#namespace__compression-level) and from the [Aerospike Backup Service (ABS)](https://aerospike.com/docs/database/tools/backup-and-restore/backup-service/performance-tuning#compression-zstd) `compression.level` setting. ::: note ABS backup compression uses different presets. ABS uses a Go Zstd implementation that maps levels `-1` through `22` to four presets. Values in the same preset bucket produce identical output. ABS level `-1` selects the Fastest preset. Wire compression level `-1` uses native Zstd fast compression instead. Do not reuse ABS preset tables when tuning wire compression levels. ::: Storage [`compression-level`](https://aerospike.com/docs/database/reference/config#namespace__compression-level) and XDR [`compression-level`](https://aerospike.com/docs/database/reference/config#xdr__compression-level) accept only positive values `1` through `9`. Wire compression level parameters also accept negative values, which trade compression ratio for speed. For background, see the [Zstd manual](https://facebook.github.io/zstd/zstd_manual.html). Dynamic changes take effect immediately and last until the node restarts. Add matching directives to `aerospike.conf` to keep the settings across restarts. ::: caution [`asadm`](https://aerospike.com/docs/database/tools/asadm) `manage config` applies the change to every node in the cluster. [`asinfo`](https://aerospike.com/docs/database/tools/asinfo) targets one node, so repeat each `asinfo` command against every node (`asinfo -h NODE -v '...'`, where _`NODE`_ is the IP address or hostname of that node) or the namespace ends up with a different mode on different nodes. A partially configured namespace does not error: masters on configured nodes compress, masters on the rest do not, and the namespace savings counters reflect only the configured nodes. ::: ### Configure replica writes Use the [Choose a compression mode](#choose-a-compression-mode) and [Compression modes for replica writes](#compression-modes-for-replica-writes) sections to pick `none`, `zstd`, or `delta-zstd`. #### Static configuration Add the directives to the `namespace` context in `aerospike.conf`: aerospike.conf ```ruby namespace test { replication-factor 2 # ... other namespace settings ... replication-compression-mode zstd replication-compression-level 1 } ``` For partial-update-heavy workloads after piloting plain `zstd`, set `replication-compression-mode` to `delta-zstd`: aerospike.conf ```ruby namespace test { replication-factor 2 # ... other namespace settings ... replication-compression-mode delta-zstd replication-compression-level 1 } ``` #### Dynamic configuration Terminal window ```bash asadm -e 'enable; manage config namespace test param replication-compression-mode to zstd' asadm -e 'enable; manage config namespace test param replication-compression-level to 1' ``` Equivalent `asinfo` form: Terminal window ```bash asinfo -v 'set-config:context=namespace;namespace=test;replication-compression-mode=zstd' asinfo -v 'set-config:context=namespace;namespace=test;replication-compression-level=1' ``` ### Configure partition migrations `migrate-compression-mode` controls wire compression for [partition migrations](https://aerospike.com/docs/database/manage/cluster/migrations), and takes its own mode and level independently of replica writes. Migration supports plain `zstd`, because the sending and receiving nodes share no base record version to build a delta against. Migration compression runs when `migrate-compression-mode` is `zstd` and both nodes on the migration link run a build that supports wire compression. Records too small to benefit are sent uncompressed. If the compressor fails on a record, the server sends that record’s bytes uncompressed and increments `migrate_wire_compression_fallbacks` on the sending node only. That counter means the sender’s codec failed (for example under memory pressure), not that a peer refused the message. If the compressed payload is no smaller than the original, the server sends the original and increments `migrate_wire_compression_not_beneficial`. If the receiving node cannot decompress a migration record, the server logs `handle insert: failed to decompress record` on that node and drops the message so migration can retry. No migration wire-compression counter increments for receive-side decompress failures. Stalled migrations with flat `migrate_wire_compression_fallbacks` may still show this warning on receivers. #### Static configuration aerospike.conf ```ruby namespace test { replication-factor 2 # ... other namespace settings ... migrate-compression-mode zstd migrate-compression-level 1 } ``` #### Dynamic configuration Terminal window ```bash asadm -e 'enable; manage config namespace test param migrate-compression-mode to zstd' asadm -e 'enable; manage config namespace test param migrate-compression-level to 1' ``` Equivalent `asinfo` form: Terminal window ```bash asinfo -v 'set-config:context=namespace;namespace=test;migrate-compression-mode=zstd' asinfo -v 'set-config:context=namespace;namespace=test;migrate-compression-level=1' ``` ### Disable wire compression Set the relevant mode parameter back to `none`. Wire compression affects only the payload format on the fabric link, so disabling it needs no data conversion: records already on storage keep whatever [storage compression](https://aerospike.com/docs/database/manage/namespace/storage/compression) format they were written in, and no migration or restart is required. To move off `delta-zstd` while keeping bandwidth savings, set `replication-compression-mode` to `zstd` instead of `none`. 1. Set the mode to `none` on every node in the cluster. A node stops compressing the traffic it sends as soon as the change applies, and continues to accept and decompress inbound compressed messages from peers that remain enabled. Terminal window ```bash asadm -e 'enable; manage config namespace test param replication-compression-mode to none' asadm -e 'enable; manage config namespace test param migrate-compression-mode to none' ``` 2. Remove the matching directives from `aerospike.conf` on every node, or set them to `none` there. A dynamic change does not persist, so a node that restarts with the old directives in place re-enables compression. 3. Confirm the modes, then confirm the savings counters have stopped moving. The modes are configuration and the savings are statistics, so they take two commands. Terminal window ```bash asadm -e 'show config namespace for test like replication-compression migrate-compression' asadm -e 'show statistics namespace for test like wire_compression' ``` Pass: every node reports `replication-compression-mode=none` and `migrate-compression-mode=none`, and `repl_wire_compression_bytes_saved` is unchanged between two readings taken minutes apart under write traffic. The savings counters are lifetime totals and do not reset to zero when you disable the feature, so compare two readings rather than looking for zeros. ## Rolling upgrades ::: caution Replica-write wire compression activates only after every node in the cluster runs Aerospike Database Enterprise Edition 8.2.0 or later. Enabling compression on upgraded nodes during a mixed-version rollout does not reduce replication bandwidth until the last node upgrades, and costs compression CPU for no saving in the meantime. In `delta-zstd`, `repl_wire_compression_delta_attempts` and `repl_wire_compression_delta_hit_pct` move normally during a mixed-version rollout even though nothing compressed is being sent, because the master builds the patch before the cluster-wide check is consulted. `repl_wire_compression_bytes_saved` staying flat is the signal that matters. Do not read the delta counters as confirmation until the rollout is complete. ::: Mixed-version clusters operate correctly during a rolling upgrade. How compression activates depends on the traffic path: - **Replica writes** use a cluster-wide compatibility check. Until every node runs a build that includes wire compression, replication traffic stays uncompressed even between two upgraded nodes. - **Partition migrations** negotiate compression per peer. Compression starts on a link as soon as both of its nodes run a build that supports wire compression and `migrate-compression-mode` is `zstd`. This is the only transport that compresses anything during a mixed-version rollout. - **[SMD full sync](https://aerospike.com/docs/database/manage/network/smd-compression)** uses the same cluster-wide check as replica writes, so it also stays uncompressed until the last node upgrades. Leave `replication-compression-mode` at `none` until the rollout completes. A node with the mode set during a mixed-version rollout still compresses every replica write and then sends the record uncompressed, because the cluster-wide check has not opened yet: you pay the CPU and get no bandwidth back, during the window when the cluster is also carrying migration load. `delta-zstd` costs the most in this state, because the master also builds a patch that is discarded. Migration compression is the exception. It negotiates per link, so you can set `migrate-compression-mode` as the rollout proceeds and it starts working on links between upgraded peers. Whichever you enable, use dynamic configuration (`asadm` or `asinfo`) first. Add static directives to `aerospike.conf` only after every node runs Aerospike Database Enterprise Edition 8.2.0 or later. An earlier build does not recognize the wire compression parameters, and a node that reads them from its configuration file fails to start with `unknown config parameter name`. ## Monitor and tune Use namespace statistics to confirm savings and choose between `zstd` and `delta-zstd`. ### Evaluate wire compression Pilot wire compression on one namespace that has representative write traffic. Wire compression applies per namespace, so you can evaluate it on one namespace before you enable it on others. 1. Choose a pilot namespace with production-like record sizes, update patterns, and cross-node replication traffic. 2. Record a baseline with `replication-compression-mode` set to `none`. - Record client write latency. Save this baseline to compare against step 5. - Mark the uncompressed boundary. Record the current `fabric_rw_bytes_sent` value to identify where uncompressed traffic ends. Do not use this baseline `fabric_rw_bytes_sent` reading to calculate the compression savings ratio. That ratio requires two separate readings taken after compression is enabled. Terminal window ```bash asadm -e 'show statistics namespace for test like repl_wire_compression' asadm -e 'show statistics service like fabric_rw_bytes_sent' ``` Wire compression counters are namespace statistics. `fabric_rw_bytes_sent` is a node statistic covering every namespace on the node, so it comes from `show statistics service` (or `asinfo -v 'statistics'`) rather than the namespace command. 3. Enable plain `zstd` on the pilot namespace using [replica-write dynamic configuration](#configure-replica-writes). Start with `replication-compression-level` at `1`. 4. Run representative write traffic on the namespace for long enough to populate wire compression metrics. 5. Compare results against your baseline: - `repl_wire_compression_bytes_saved` increases if records compress on the wire. - For a savings ratio, see [Estimate the savings ratio](#estimate-the-savings-ratio). - `repl_wire_compression_delta_attempts` stays at `0` throughout this step. Plain `zstd` never builds delta patches, so a flat attempts counter here tells you nothing about your workload. Evaluate it in step 6, after the namespace is in `delta-zstd`. - Client write latency and node CPU must stay within your service limits. See [Tuning signals](#tuning-signals) if latency rises. 6. Evaluate delta compression suitability. Recommended when records are large and updates touch only small portions of data. Note: Skip this entire step if the effective write commit level is `master` and the namespace workload consists almost exclusively of single-record client writes. Switch the pilot namespace to `delta-zstd` and assess whether delta compression provides a net benefit over standard Zstd. - Wait until `migrate_tx_partitions_remaining` and `migrate_rx_partitions_remaining` both reach `0` for the namespace before evaluating counters. - Compare `repl_wire_compression_delta_hit_pct`, `repl_wire_compression_fallbacks`, and the `repl_wire_compression_delta_reject_*` metrics. A low hit rate combined with high fallbacks or version rejections indicates that standard Zstd is a better fit. If `repl_wire_compression_delta_attempts` does not scale with write volume after switching, check the following: - [write-commit-level-override](https://aerospike.com/docs/database/reference/config#namespace__write-commit-level-override) and client write policies. - Whether the effective commit level is `master` (single-record client writes disable deltas under master, though standard Zstd still shows `repl_wire_compression_bytes_saved` growth). Batch, background-job, read-touch, and XDR-received writes still generate deltas under master. 7. Adjust `replication-compression-level` or change mode based on [Tuning signals](#tuning-signals). - Apply the validated replication settings to remaining target namespaces. - Enable [migration](#configure-partition-migrations) compression separately if that traffic path is relevant to your rollout. - Save the validated settings in `aerospike.conf` across cluster nodes only after the rollout is complete. Then follow the [Rolling upgrades](#rolling-upgrades) procedure to apply configuration changes across the cluster safely without downtime. ### Namespace metrics Query namespace statistics with `asadm` or `asinfo`. The `wire_compression` pattern matches replication and migration metrics: Terminal window ```bash asadm -e 'show statistics namespace for test like wire_compression' ``` Wire compression metrics were introduced in Database 8.2.0. For metric descriptions, see the [Metrics reference](https://aerospike.com/docs/database/reference/metrics). The [Tuning signals](#tuning-signals) section explains how to interpret them for your workload. #### Estimate the savings ratio `repl_wire_compression_bytes_saved` and `migrate_wire_compression_bytes_saved` are cumulative byte counts, not ratios, and so are the fabric counters you pair them with. To turn one into a percentage, pair it with the fabric channel it rides on: | Traffic | Saved bytes | Channel counter | | --- | --- | --- | | Replica writes | `repl_wire_compression_bytes_saved` | `fabric_rw_bytes_sent` | | Partition migrations | `migrate_wire_compression_bytes_saved` | `fabric_bulk_bytes_sent` | Take two readings a fixed interval apart, both after enablement and both under normal load, and divide the differences: ```text saving_pct = Δbytes_saved / (Δchannel_bytes_sent + Δbytes_saved) * 100 ``` Three things to get right before you trust the number: - Use differences, not totals. The fabric counters run from node start, so they include every byte the node sent before you enabled wire compression, including the whole baseline period from step 2 of [Evaluate wire compression](#evaluate-wire-compression), when the mode was still `none`. Dividing one lifetime total by another therefore reports far less saving than the namespace is actually getting, and reports less the longer the node has been up. - The saved-bytes counters are per namespace and the fabric counters are per node. Sum the namespace counter across every namespace on the node before dividing, or the result understates the saving. - The result is a floor, not an exact figure. The numerator counts record payload only, while the fabric counter includes per-message headers and every other message on the same channel. The real percentage is somewhat higher. - On a namespace that uses multi-record transactions, the commit traffic rides the same fabric channel but is never compressed, so it enlarges the divisor without enlarging the numerator. The ratio understates the saving on those namespaces by more than the amount described above. - On a [strong-consistency](https://aerospike.com/docs/database/manage/namespace/consistency) namespace, the catch-up traffic that follows a node returning to the cluster is not compressed on the wire. While it runs the divisor grows and the numerator does not, so the ratio dips. That is recovery traffic, not a compression regression, so wait for migrations to finish before rereading the ratio. Namespaces with a `replication-factor` above 2 produce more of it. ### Profiling wire compression CPU For short-term diagnostics, enable service-wide codec profiling with `enable-benchmarks-wire-compression`. The parameter is `false` by default. When enabled, the server records CPU time for each of the four codec operations: patch generation on the sending node and patch application on the replica, both in `delta-zstd` mode, plus compression and decompression. #### Static configuration aerospike.conf ```ruby service { # ... other service settings ... enable-benchmarks-wire-compression true } ``` #### Dynamic configuration Terminal window ```bash asadm -e 'enable; manage config service param enable-benchmarks-wire-compression to true' ``` Equivalent `asinfo` form: Terminal window ```bash asinfo -v 'set-config:context=service;enable-benchmarks-wire-compression=true' ``` Query profiling output with `asinfo`: Terminal window ```bash asinfo -v 'zstd-wire-stats' ``` The `statistics` info command always includes [`wire_comp_cpu_pct`](https://aerospike.com/docs/database/reference/metrics#node_stats__wire_comp_cpu_pct) and its four per-operation breakouts, `wire_comp_make_patch_cpu_pct`, `wire_comp_apply_patch_cpu_pct`, `wire_comp_compress_cpu_pct`, and `wire_comp_decompress_cpu_pct`. Each is a gauge: a rolling ten-second average rather than a cumulative total, so graph it directly rather than as a rate. They read `0.00` while profiling is disabled. These counters are scaled out of (number of CPUs × 100)%, not out of 100%. On a 16-core node the range is `0.00` to `1600.00`, and `100.00` means the codec is using one CPU’s worth of time. Compare the value against [`process_cpu_pct`](https://aerospike.com/docs/database/reference/metrics#node_stats__process_cpu_pct), which uses the same scale, to see the codec’s share of the process. If that share grows faster than `repl_wire_compression_bytes_saved`, the compression level is too high for this workload. The four histograms behind these counters are written to the server log only. See [Wire compression CPU analysis](https://aerospike.com/docs/database/observe/latency#wire-compression-cpu-analysis) for what each one measures and how to read it. Disable profiling after you finish tuning. The extra per-operation timing adds overhead on the replication path. ### Tuning signals Wire compression skips certain payloads entirely: - Plain `zstd` on replica writes and migrations skips payloads too small to benefit. Delta patches are sent regardless of record size. - Durable-delete tombstones and records already compressed at the [storage](https://aerospike.com/docs/database/manage/namespace/storage/compression) layer are not wire-compressed, and neither increments any counter. On a delete-heavy namespace, check the namespace `tombstones` statistic before concluding compression is misconfigured. - When compression produces a payload no smaller than the plain record, the server sends uncompressed bytes and increments [`repl_wire_compression_not_beneficial`](https://aerospike.com/docs/database/reference/metrics#namespace__repl_wire_compression_not_beneficial), or [`repl_wire_compression_delta_not_beneficial`](https://aerospike.com/docs/database/reference/metrics#namespace__repl_wire_compression_delta_not_beneficial) when the patch is the payload that did not shrink. The sender counts a fallback. The replica counts the reason. `repl_wire_compression_fallbacks` increments on the node that sent the replica write, and all four `repl_wire_compression_delta_reject_*` counters increment on the node that refused it. Read them across the whole cluster rather than against a single node. `asadm -e 'show statistics namespace for test like wire_compression'` reports every node in one table. Interpret `repl_wire_compression_delta_reject_*` counters as follows: | Counter | Meaning | | --- | --- | | `repl_wire_compression_delta_reject_no_record` | Replica does not hold the base record the patch was built against | | `repl_wire_compression_delta_reject_version` | Replica holds the record but not the matching generation or last-update-time version | | `repl_wire_compression_delta_reject_apply` | Replica held the matching base version but could not rebuild the record from the patch | | `repl_wire_compression_delta_reject_other` | Replica could not use its stored copy of the base record: it could not be read, it is stored compressed, or it did not pass validation | Additional signals: - Low `repl_wire_compression_delta_hit_pct` with high `repl_wire_compression_fallbacks`: switch from `delta-zstd` to `zstd`. - `repl_wire_compression_delta_attempts` not increasing while writes continue on a `delta-zstd` namespace: deltas have stopped being attempted, usually because the effective write commit level for single-record client writes became `master` through a client write policy change. Batch, background-job, read-touch and XDR-received writes are unaffected by commit level, so a namespace carrying those still shows attempts. `repl_wire_compression_delta_hit_pct` is a lifetime ratio and holds its last value in this state, so it does not fall. Alert on the attempt rate, not on the percentage. - High `repl_wire_compression_delta_reject_version` or `repl_wire_compression_delta_reject_no_record`: replicas do not hold the version the master built the patch against. Check that the namespace has finished migrating before acting on this. Both `migrate_tx_partitions_remaining` and `migrate_rx_partitions_remaining` read `0` on every node: Terminal window ```bash asadm -e 'show statistics namespace for test like partitions_remaining' ``` While partitions are still migrating, after a rolling upgrade, a node restart, or any other topology change, replicas routinely hold an older copy of a record or none at all, so both counters rise for that reason alone, along with `repl_wire_compression_fallbacks`. They settle once migrations finish. Only when they stay high on a cluster with no migrations outstanding is the workload itself a poor fit, and `zstd` the better choice. - Rising `repl_wire_compression_delta_apply_device_reads`: delta apply loads the prior record version on the replica. The counter increments once per delta apply on namespaces using `storage-engine device` or `storage-engine pmem`, before the load runs and without checking whether the load hits cache or storage. It stays at zero on `storage-engine memory`. With [`cache-replica-writes`](https://aerospike.com/docs/database/reference/config#namespace__cache-replica-writes) enabled, some loads are served from the post-write cache, so the counter can exceed physical device reads. What to do about it depends on the storage engine. On a namespace using `storage-engine device`, read the counter alongside `cache_read_pct` for a sense of how much of the namespace’s read traffic is reaching the drive at all. That figure is a short-interval percentage covering every read the namespace serves, not a delta-apply hit rate, so treat it as a trend rather than a measurement. If it is low and write latency is rising, either raise [`post-write-cache`](https://aerospike.com/docs/database/reference/config#namespace__post-write-cache) so more recently written records are served from memory, or switch to `zstd`, which needs no base version on the replica. For rack awareness deployments that span multiple Availability Zones, [`cache-replica-writes`](https://aerospike.com/docs/database/reference/config#namespace__cache-replica-writes) on `storage-engine device` allows replica writes into the post-write cache, which can improve latency and reduce read I/O. On a namespace using `storage-engine pmem`, the counter still increments on every delta apply when the prior record is loaded from PMem. `cache_read_pct` is not reported for that engine. Neither [`post-write-cache`](https://aerospike.com/docs/database/reference/config#namespace__post-write-cache) nor [`cache-replica-writes`](https://aerospike.com/docs/database/reference/config#namespace__cache-replica-writes) is valid in a `storage-engine pmem` stanza: the node fails to start if either directive is present. Treat the counter as a count of delta-apply loads, and use `zstd` if the replica-side cost is the problem. - Rising write latency after enablement: try a more negative `replication-compression-level` (fast compression), a lower positive level such as `1`, or plain `zstd` instead of `delta-zstd`. To back the feature out entirely, see [Disable wire compression](#disable-wire-compression). - High `migrate_wire_compression_not_beneficial` with low `migrate_wire_compression_bytes_saved`: migration records may already be incompressible. Wire compression may not help that namespace during migrations. ## Verify 1. Confirm every node reports the expected mode. Use `asadm`, which reports all nodes in one table. A single `asinfo` call answers for one node only, and a namespace set on some nodes and not others does not error. Terminal window ```bash asadm -e 'show config namespace for test like replication-compression migrate-compression' ``` For a namespace using plain Zstd on replica writes, each node reports: ```text migrate-compression-level |1 migrate-compression-mode |none replication-compression-level|1 replication-compression-mode |zstd ``` Pass: every node shows the same values, and they are the ones you configured. A node showing a different mode is still sending uncompressed replica writes for the partitions it masters. To read a single node, for example to isolate one that disagrees, run the following command, where _`NODE`_ is the IP address or hostname of that node: Terminal window ```bash asinfo -h NODE -v 'get-config:context=namespace;namespace=test' | tr ';' '\n' | grep -E 'replication-compression|migrate-compression' ``` 2. Run representative write traffic, then confirm the savings counters move. Terminal window ```bash asadm -e 'show statistics namespace for test like wire_compression' ``` `repl_wire_compression_bytes_saved` increases. That is the criterion that confirms compressed bytes actually reached the replica. In `delta-zstd` mode, `repl_wire_compression_delta_attempts` also increases with the write rate and `repl_wire_compression_delta_hit_pct` is acceptable for your workload, but those two move on the sending node whether or not anything compressed went on the wire, so treat them as workload signals rather than proof. Attempts flat while writes continue means deltas have stopped. See [Tuning signals](#tuning-signals). 3. If you enabled `migrate-compression-mode`, verify it separately. Write traffic does not exercise it, because migration compression runs only while partitions are actually moving, on the node sending them. Confirm it during the next planned node restart or rolling maintenance rather than forcing a topology change on a production cluster. Terminal window ```bash asadm -e 'show statistics namespace for test like migrate_wire_compression' ``` Pass: `migrate_wire_compression_bytes_saved` increases while migrations are in progress. All three migration counters at zero with no migrations running means nothing was migrated, not that the setting failed, so check the mode with `show config` instead. 4. If the cluster is mid-upgrade, expect replication `bytes_saved` to stay flat until the rollout completes, and migration `bytes_saved` to move earlier on links between compatible peers. See [Rolling upgrades](#rolling-upgrades). ## Troubleshoot | Symptom | Likely cause | Action | | --- | --- | --- | | No `bytes_saved` despite enabled mode | Storage-compressed records, records too small for plain Zstd to compress, tombstones, mixed-version cluster, or mode still `none` | Check [`repl_wire_compression_not_beneficial`](https://aerospike.com/docs/database/reference/metrics#namespace__repl_wire_compression_not_beneficial), and [`repl_wire_compression_delta_not_beneficial`](https://aerospike.com/docs/database/reference/metrics#namespace__repl_wire_compression_delta_not_beneficial) in `delta-zstd`. If either climbs, records are reaching the compressor and not shrinking. If both stay at zero, confirm `replication-compression-mode`, `migrate-compression-mode`, that the [rollout](#rolling-upgrades) is complete, and storage [`compression`](https://aerospike.com/docs/database/reference/config#namespace__compression), then check record sizes: payloads too small to compress are skipped without incrementing any counter | | `delta_attempts` flat while writes continue, `delta_hit_pct` unchanged (`delta-zstd`) | Effective write commit level for single-record client writes became `master`: a client write policy change, a new service writing with commit level `master`, or the namespace override set to `master`. Batch, background-job, read-touch and XDR-received writes are unaffected, so attempts stopping entirely means the namespace carries little of that traffic. `delta_hit_pct` is cumulative and cannot fall when attempts stop | Check client write policies and [`write-commit-level-override`](https://aerospike.com/docs/database/reference/config#namespace__write-commit-level-override). Plain Zstd still applies, so `bytes_saved` keeps climbing | | High `repl_wire_compression_fallbacks` | A replica refused a compressed or delta payload and the sender retransmitted the full record | In `delta-zstd`, check all four `repl_wire_compression_delta_reject_*` counters on the receiving nodes. If version or no-record rejections dominate, confirm `migrate_tx_partitions_remaining` and `migrate_rx_partitions_remaining` are `0` for the namespace first, because migrating partitions raise both rejection counters for a reason that clears itself. Switch the namespace to `zstd` only if they stay high with no migrations outstanding. In plain `zstd` the fallback is a decode failure on the replica and increments no rejection counter: check the replica’s server log and its health instead | | High `migrate_wire_compression_not_beneficial` | Migration records do not compress smaller than their original size | Expect limited migration savings for this namespace. The counter covers only records the compressor attempted, so record size is not the cause. Set `migrate-compression-mode` back to `none` to stop paying the compression cost for no gain | | Rising `migrate_wire_compression_fallbacks` | Compressor failed on the sending node while migrating a record (often memory pressure) | Check the sender’s server log around migration time. Lower `migrate-compression-level`, relieve memory pressure, or set `migrate-compression-mode` to `none`. This counter does not rise when a peer refuses a message | | Migrations stall or lag with migration compression enabled, flat `migrate_wire_compression_fallbacks` | Receiver failed to decompress a compressed migration record | On receiving nodes, search the server log for `handle insert: failed to decompress record`. No namespace counter increments. Fix codec or memory issues on the receiver, or set `migrate-compression-mode` to `none` while investigating | | Increased write latency | Compression level too aggressive, or delta apply loading base records on the replica | Lower `replication-compression-level` or use a negative level. Check `repl_wire_compression_delta_apply_device_reads` together with `cache_read_pct` on `storage-engine device`: the counter increments before each prior-record load and may include cache-served loads when [`cache-replica-writes`](https://aerospike.com/docs/database/reference/config#namespace__cache-replica-writes) is enabled. Switching to `zstd` removes the delta-apply load entirely, because it needs no base version on the replica | | Dynamic configuration appears ignored | Invalid value, or the parameter sent to the wrong context or namespace | `set-config` rejects these rather than ignoring them. Re-run the command and read its response, which reports an invalid `set-config` parameter on failure. The server log records an enterprise-only rejection, but not an invalid value | | Node fails to start after adding directives | Community Edition, or a Database version earlier than 8.2.0, which does not recognize the parameter names | Remove wire-compression directives from `aerospike.conf`, or complete the upgrade described in [Prerequisites](#prerequisites) before adding static directives | ## Next steps - [Configure storage compression](https://aerospike.com/docs/database/manage/namespace/storage/compression) if you also want on-disk space savings. - [Configure rack awareness](https://aerospike.com/docs/database/manage/namespace/rack-aware) to place replicas across failure domains and combine with client [`PREFER_RACK`](https://aerospike.com/docs/database/learn/policies#replica) reads. - [Manage migrations](https://aerospike.com/docs/database/manage/cluster/migrations) when tuning cluster topology and migration traffic.