Configure SMD wire compression

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

This page describes how to enable Zstd compression for System Metadata (SMD) full-sync messages.

Applies to: Aerospike Database Enterprise Edition 8.2.0 and later.

Audience: Cluster operators who manage intra-cluster fabric bandwidth, especially on cross-AZ deployments with large namespace or sindex catalogs.

Outcome: You enable and tune SMD wire compression, then verify bandwidth savings using smd-info metrics.

Overview

Aerospike System Metadata (SMD) stores cluster-wide catalog data in eight modules: secondary index definitions, registered UDFs, security users and roles, Cross-Datacenter Replication (XDR) configuration, data-masking rules, truncate cutoffs, eviction depth, and the strong consistency roster. Each is a separate SMD module, persisted as its own file in the smd directory under the work directory. On a cluster with many XDR destinations, the XDR module is often the largest of the eight. Namespace configuration is not SMD data. It comes from aerospike.conf.

SMD wire compression applies Zstd to full-sync messages before they are sent. It operates independently of namespace wire compression (replica writes and migrations) and of storage compression.

What a full-sync is

A full-sync is the exchange that reconciles one SMD module’s catalog across the cluster. Instead of propagating a single change, each node sends its entire catalog for that module to the cluster principal. The principal merges every node’s copy into one agreed version and sends that merged catalog back to every node.

Full-syncs happen when a node brings catalog state the cluster has to reconcile: a node joins, or a node restarts and rejoins. A node leaving cleanly does not trigger one, because the remaining nodes already agree on every module. Routine metadata changes are not full-syncs either. Creating a secondary index or registering a UDF propagates as a single item through the principal, and single-item updates are never compressed.

This is what determines when the compression counters move. They advance when a node joins or restarts, and stay flat on a cluster with stable membership, however much index or UDF work the cluster does. To generate a full-sync deliberately, restart a node and let it rejoin. Draining one does not do it.

When to use SMD wire compression

SMD wire compression helps when:

Any single SMD module’s catalog is large. Each module is compressed on its own, and a module’s packed catalog must reach 1 KiB before compression is attempted. The files under the smd directory in the work directory are a usable proxy, one file per module. They are stored as JSON, so they are always larger than the packed form. A sindex.smd of a few hundred bytes is certainly below the threshold.
SMD full-sync events (membership changes, node restarts) generate detectable fabric traffic.
The cluster spans AZ boundaries and all fabric traffic contributes to egress cost.

On a cluster whose modules are all small, compression_fallbacks climbs while compression_bytes_saved stays at zero. That is the expected result, not a fault.

SMD full-sync volume is lower than replica-write or migration traffic in most workloads. If you are primarily targeting cross-AZ replication cost, see Configure wire compression first.

Prerequisites

Aerospike Database Enterprise Edition 8.2.0 or later on every node in the cluster. SMD wire compression requires all nodes to be at a compatible build before the first compressed full-sync is sent. During a rolling upgrade, the cluster holds compression until every node has upgraded. See Rolling upgrades. It is not gated by a feature key.
Write access to the service context in aerospike.conf, or to asadm/asinfo.

Configure SMD compression

smd-compression-mode and smd-compression-level belong in the service context, not in a namespace context.

Setting	Values	Default
`smd-compression-mode`	`none`, `zstd`	`none`
`smd-compression-level`	`-10` through `22`	`1`

SMD compression supports only zstd. There is no delta-zstd mode, because SMD full-sync sends the complete catalog state for a module rather than a per-record delta.

With smd-compression-mode zstd, payloads whose packed item list is smaller than 1 KiB are sent uncompressed at any compression level. With mode none, nothing is compressed and no attempt is counted. This threshold matches the size at which Zstd compression produces meaningful savings on short, structured payloads.

Compression levels

smd-compression-level uses the same Zstd level range and the same level semantics as replication-compression-level and migrate-compression-level. See Compression levels for the level table and for how these levels differ from Aerospike Backup Service presets. smd-compression-level is independent of storage compression-level.

SMD full-sync is not on the client write path, so a higher level costs less here than it would on replica writes.

Static configuration

Add both directives to the service context in aerospike.conf on every node in the cluster. Each node decides independently whether to compress the full-sync messages it sends, so a node left at none sends uncompressed. The principal, which sends the merged catalog to every other node, is the one that matters most.

service {
    # ... other service settings ...
    smd-compression-mode zstd
    smd-compression-level 1
}

Dynamic configuration

Dynamic changes take effect immediately and persist until the node restarts. To keep them across restarts, also add the directives to aerospike.conf.

asadm -e 'enable; manage config service param smd-compression-mode to zstd'
asadm -e 'enable; manage config service param smd-compression-level to 1'

Equivalent asinfo form:

asinfo -v 'set-config:context=service;smd-compression-mode=zstd'
asinfo -v 'set-config:context=service;smd-compression-level=1'

The asadm form applies to every node in the cluster. The asinfo form applies to one node only, so repeat it against each node (asinfo -h NODE -v '...', where NODE is the IP address or hostname of that node) or use the asadm form.

Disable SMD compression

Set smd-compression-mode back to none on every node in the cluster. No migration or restart is required, and a node stops compressing on its next full-sync event. A node set to none still accepts and decompresses inbound compressed full-syncs from peers that remain enabled, so a partial rollback is safe.

asadm -e 'enable; manage config service param smd-compression-mode to none'

Equivalent asinfo form:

asinfo -v 'set-config:context=service;smd-compression-mode=none'

Rolling upgrades

SMD wire compression activates only when every node in the cluster is running Database 8.2.0 or later. During a rolling upgrade, the cluster is in a mixed state and no compressed SMD full-sync messages are sent, even if smd-compression-mode zstd is already in aerospike.conf on upgraded nodes.

Once the last node upgrades, the cluster exits the mixed state and compression becomes active on the next full-sync. SMD full sync is gated cluster-wide, like replica-write compression. Partition migration is the exception among the three transports: it negotiates per peer, so it compresses between two upgraded nodes before the rollout finishes.

SMD wire compression and namespace replica-write wire compression share the same cluster-wide build gate, so both activate together when the rolling upgrade completes.

Verify the mixed state

asinfo -v 'smd-info'

The mixed_cluster field in the output shows whether the cluster is still in a mixed state. It appears in the global field list before the per-module records:

smd:n_pending_sets=0,n_events=0,n_nodes=3,principal=bb9...,initial_sync_done=true,mixed_cluster=true,cluster_key=...,compression_hit_pct=0.000,compression_bytes_saved=0,compression_fallbacks=0;<module>:...

mixed_cluster=true means at least one node is still on an older build, and no SMD full-sync is being compressed anywhere in the cluster. All three compression counters stay at 0 while this is true, whatever smd-compression-mode is set to. Finish the upgrade. There is nothing to fix. mixed_cluster=false means every node is at a compatible build and compression can proceed.

Verify SMD compression

After enabling smd-compression-mode zstd, use smd-info to confirm the feature is active and check savings.

These counters are per node, and they count only the full-sync messages that node sent. A node that only receives compressed full-syncs reports zeros. The cluster principal builds the merged catalog and sends it to every other node, so it carries most of the compressed volume, while every other node compresses only its own catalog on the way to the principal. Read smd-info from every node, and from the principal in particular. In the sample output, the principal= field names the principal node.

Check metrics

asinfo -v 'smd-info'

The compression fields appear at the end of the global field list, before the per-module records:

smd:n_pending_sets=0,n_events=0,n_nodes=3,principal=BB906C88F074A66,initial_sync_done=true,mixed_cluster=false,cluster_key=36F690320D37,compression_hit_pct=95.000,compression_bytes_saved=204800,compression_fallbacks=1;evict:...;roster:...

Each SMD module is compressed on its own, so one full-sync counts once per module rather than once per event. A cluster typically has one or two catalogs large enough to compress and several that are small or empty, so a correctly working cluster normally reports a low compression_hit_pct. The preceding sample shows one module of eight clearing the threshold. The percentage measures how many of your catalogs are worth compressing, not how well the feature is working. Judge the feature by whether compression_bytes_saved grows.

Metric	Meaning
`compression_hit_pct`	Percentage of compression attempts that produced a smaller message (hits ÷ attempts × 100). `0.000` has four possible causes: `smd-compression-mode` is still `none`, the cluster is still mixed, no full-sync has happened since this node started, or every catalog that was tried was too small to benefit. Check `compression_fallbacks` to tell them apart.
`compression_bytes_saved`	Cumulative bytes kept off the wire, as (original size − compressed size). Credited once per compressed message built, regardless of how many peers receive it, so it is a lower bound on the total saving. Resets on node restart.
`compression_fallbacks`	Full-sync messages that were compressed and then sent uncompressed anyway. Covers three cases: the module catalog was too small to benefit, the compressed result was no smaller than the original, or the codec failed. The counter does not distinguish them. It counts only messages compression was actually tried on, so it stays at `0` while `smd-compression-mode` is `none` or the cluster is still mixed.

Whether compression_hit_pct ever becomes non-zero depends on how large this cluster’s metadata catalogs are. Catalogs below the size threshold are always sent uncompressed, so a cluster with few secondary indexes or UDFs can sit at 0.000 indefinitely and still be configured correctly. To confirm the setting took effect, check that it reports zstd and that the cluster is not still mixed:

asinfo -v 'get-config:context=service' | tr ';' '\n' | grep smd-compression
asinfo -v 'smd-info'

Pass: every node reports smd-compression-mode=zstd and mixed_cluster=false. That is the criterion that holds on every cluster. Savings are a separate question. A growing compression_bytes_saved, read on the principal, means the feature is also paying off here, and a flat zero on a cluster with small catalogs is expected rather than a failure.

Because compression_fallbacks moves only after compression has been tried, any value above 0 confirms that smd-compression-mode is zstd and that the cluster has left the mixed state. On a cluster with small SMD catalogs, a rising compression_fallbacks alongside a flat compression_bytes_saved is the normal steady state and the clearest sign the feature is correctly enabled. It also means SMD compression is not paying for itself here, either because the catalogs are small or because their contents do not compress.

Check the CPU cost

SMD compression and decompression are timed by the same codec counters as replica-write and migration compression. To see what a level change costs, enable enable-benchmarks-wire-compression, read wire_comp_compress_cpu_pct and wire_comp_decompress_cpu_pct, then turn profiling off again. On a node that also runs namespace wire compression, those counters report the combined cost of every transport. See Profiling wire compression CPU.

Next steps

Configure wire compression to reduce replica-write and migration traffic on the fabric.
Network configuration for fabric and service port settings.