--- title: "Aerospike Backup Service performance tuning" description: "Tune the Aerospike Backup Service process for throughput and efficiency." --- # Aerospike Backup Service performance tuning > For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt) > > All documentation pages available in markdown. This page describes how to tune Aerospike Backup Service (ABS) by adjusting read parallelism, write parallelism, compression, file sizing, and throttling. The goal is to finish backups within your target window without overloading the Aerospike cluster, the ABS host, or the storage backend. Use this guidance when you already have ABS running and want to diagnose a throughput, CPU, memory, or storage bottleneck during backup jobs. ## Tuning goals Tuning decisions balance two stages of the backup process: - Readers read records from Aerospike using primary index (PI) queries. - Writers serialize, compress, encrypt, and move the data to storage. The best tuning path depends on your goal: - Faster backups usually require more read or write parallelism. - Lower cluster impact usually requires read throttling or lower read parallelism. - Smaller backup files usually require more compression CPU. - Lower memory usage usually requires fewer writers or smaller upload buffers. ### Tune one bottleneck at a time Pick a goal, check the `pipeline` metric to find whether readers or writers are limiting throughput, adjust the parameters for that stage, and validate the result with representative data. Stop increasing concurrency when throughput gains become marginal or when CPU, memory, cluster load, or storage throttling become limiting factors. ## Before you tune This guidance applies to ABS 3.6.1 and later. It is for operators and site reliability engineers who tune backup throughput, compression, and resource usage in production or production-like environments. Before tuning ABS performance, you need: - A running ABS instance with at least one configured backup routine. - Access to the ABS status API and logs. - Access to host or container CPU and memory metrics. - Representative backup data and a storage path that matches your production environment. After tuning, you have a configuration that balances reader and writer throughput and is validated with the `metrics.pipeline` value, backup duration, CPU usage, memory usage, and ABS logs. The guidance is directional and does not include benchmark results. Always validate final values with your own data and storage or network path. ## Reader and writer stages Each backup runs a pool of reader workers and a pool of writer workers, connected by buffered Go channels. ABS publishes a `metrics.pipeline` value that reports how full these channel buffers between readers and writers are. If readers fill the channels faster than writers drain them, the channels fill up and `pipeline` approaches capacity. If writers drain faster than readers fill, the channels stay empty and `pipeline` stays near `0`. ### Reader stage 1. Open a PI query against an Aerospike node for a partition range. 2. Receive records using Aerospike binary protocol over TCP. 3. Push each record into a channel buffer with capacity 256 per reader. The constraint for readers is usually the query capacity of the Aerospike Database nodes. Common slowdown causes include node I/O bottlenecks, CPU contention from production read/write traffic, and network saturation between ABS and the cluster. A cluster over capacity results in `FAIL_FORBIDDEN` errors in the ABS logs. ### Writer stage 1. Pull a record from the channel buffer. 2. Encode it into the `.asb` binary format. 3. Compress with ZSTD if enabled. 4. Encrypt if enabled. 5. Write the encoded bytes into an in-memory buffer until it reaches the size limit set by `min-part-size`. 6. Upload the buffer as one part of a multipart upload to object storage. 7. When cumulative bytes reach `file-limit`, complete the current multipart upload and start a new file. A writer stalls when any step in this chain is a bottleneck. The most common bottlenecks are: - CPU-bound ZSTD compression at high levels - Network-bound uploads to Amazon S3, Google Cloud Storage, and Azure Blob Storage - Memory pressure from large upload buffers, which can trigger garbage collection pauses The constraint for writers is usually CPU or storage. More writers running ZSTD on the same machine means more goroutines competing for the same cores. Each writer also allocates memory for its channel buffer, upload buffer, and encoder state. Total memory grows linearly with the number of writers and can overflow the container’s memory limit. Storage backends can throttle when they receive too many concurrent upload requests. Always check the [`pipeline` metric](#use-metricspipeline-to-find-the-bottleneck) to identify the bottleneck stage before adding threads to either side. ## Use `metrics.pipeline` to find the bottleneck Before changing any configuration, check the `pipeline` metric during a running backup: Terminal window ```bash curl http://ABS_HOST:8080/v1/backups/currentBackup/ROUTINE_NAME | \ jq '{full: .full.metrics.pipeline, incremental: .incremental.metrics.pipeline}' ``` Example output: ```json { "full": 2048, "incremental": 0 } ``` The response has separate `full` and `incremental` blocks, one per running job. The `metrics.pipeline` field inside each block reports the total number of records sitting in the channel buffers between readers and writers for that job. Pipeline capacity equals `256 × parallel + 256 × parallel-write`. When `parallel-write` is not set, it defaults to `parallel`, so the capacity simplifies to `512 × parallel`. Use this table to estimate pipeline capacity from read and write parallelism. | Configuration | Capacity | | --- | --- | | `parallel=8` (default, both read and write) | 256×8 + 256×8 = 4096 | | `parallel=4`, `parallel-write=8` | 256×4 + 256×8 = 3072 | | `parallel=4`, `parallel-write=4` | 256×4 + 256×4 = 2048 | A balanced pipeline can briefly reach `0` or capacity during normal fluctuations. In a busy system, watch for sustained readings near either extreme. - Pipeline stays near `0`: Writers are idle, waiting for data. Readers are the bottleneck. - Pipeline fluctuates across the range: Neither stage is a clear bottleneck. The configuration is near optimal. - Pipeline stays near capacity: Readers are blocked because the buffers are full. Writers are the bottleneck. ## Tuning workflow 1. Start with concurrency defaults. Set `parallel: 8`. Leave `parallel-write` unset so it inherits the `parallel` value. 2. Set compression and file-size defaults. Enable ZSTD at `compression.level: 3` to use the Default preset. Keep `file-limit: 250`. 3. Monitor the `pipeline` metric. Check `GET /v1/backups/currentBackup/ROUTINE_NAME` during a backup run. 4. Match the `pipeline` reading to a stage in [Reader and writer stages](#reader-and-writer-stages): - Stays near `0`: Readers are constrained by query latency, network limits, or a low `parallel` setting. Increase `parallel` and check cluster responsiveness and resource metrics. - Fluctuates across the range: Readers and writers are balanced. Configuration is close to optimal. - Stays near capacity: Writers are constrained by compression CPU, upload latency, or a low `parallel-write` setting. Increase `parallel-write`, reduce the compression level, or increase storage bandwidth. 5. Check system resources. - Run each ABS instance with at least 4 vCPUs and 2 GiB of RAM for production backups and restores. - CPU > 90%: Decrease `parallel-write` or lower the compression level. - RAM high: Decrease `parallel-write` or `min-part-size`. 6. Run short tests that use representative data. ABS reaches steady-state throughput within seconds of backup start, so test runs of 2–5 minutes are often enough to observe it. 7. Stop tuning the single instance when gains flatten or resource limits appear. Use the final verification results to decide whether to scale horizontally. ## Tune reader bottlenecks Use the following parameters when `pipeline` stays near `0`, which means writers are waiting for records. ### `parallel` (read parallelism) Controls the maximum number of parallel read operations to run against the cluster. Each operation processes a subset of partition data and issues PI queries against the Aerospike cluster. In ABS 3.6.1 and later, this limit applies across all namespaces in a routine rather than separately to each namespace. - **When to increase:** If `pipeline` stays near `0`, increase `parallel` incrementally until readers stop being the bottleneck. Throughput gains taper off once readers saturate the cluster’s query capacity. Stop increasing when gains are marginal. - **When to decrease:** You see `FAIL_FORBIDDEN` errors in ABS logs, which means the cluster’s query thread capacity is exhausted. - **Default:** `8` ### `max-concurrent-nodes` Controls the maximum number of concurrent requests ABS issues to Aerospike cluster nodes during backup PI queries. - **When to decrease:** You need to reduce PI query load on the cluster and can accept slower backups. - **Default:** Unset or `0` (issue requests to all nodes in parallel). Do not use `parallel: 1` with `max-concurrent-nodes: 1` in production. With one concurrent PI query and one cluster node at a time, ABS reads each node sequentially before moving to the next, and a single reader writes backup files sequentially. Backups run much slower than with higher parallelism, and PI query load stays on one cluster node at a time instead of spread across the cluster. For production, use at least `parallel: 2` so backup reads are distributed more evenly. Leave `max-concurrent-nodes` unset, or set it higher than `1`, unless you deliberately throttle cluster load and accept slower backups. Tune restore `parallel` separately in the restore policy to prioritize recovery speed over backup throughput. ::: note `aerospike-clusters.CLUSTER_NAME.max-parallel-scans` caps the number of concurrent PI queries that ABS runs against a cluster. PI queries were historically called scans. The limit is shared across every ABS routine that references the same cluster entry, but does not affect other Aerospike clients connected to the cluster. If multiple routines run at once, keep the sum of their active `parallel` values within `max-parallel-scans`. ::: ### `records-per-second` (RPS throttle) Limits the number of records read from Aerospike per second across all readers. Use this to limit ABS’s impact on a production cluster without changing `parallel`. - **When to set:** Production database latency or resource usage rises during backups, but you still want to keep enough read parallelism to scan partitions efficiently. - **When to increase:** Backups are taking too long, the Aerospike cluster has headroom, and `pipeline` stays near `0` because writers are waiting for records. - **When to decrease:** Backups are competing with production traffic, or cluster CPU, disk, or network usage rises beyond the level you want to allow for backup work. - **Default:** No limit. Omitting the parameter or setting it to `0` disables the RPS throttle. `records-per-second` limits returned records, not query concurrency. If the cluster is rejecting PI queries or running out of query threads, reduce `parallel` or lower `max-parallel-scans` instead. ### `use-scan-compression` Enables built-in compression during PI queries between the Aerospike cluster and ABS. When enabled, cluster nodes compress record data before sending it over the network, then ABS decompresses it on receipt. This is separate from the [`compression`](#compression-zstd) policy, which compresses encoded backup data for storage after ABS receives records. This setting requires Aerospike Database Enterprise Edition. - **When to enable:** Network bandwidth limits throughput (for example, cross-region backups) and cluster nodes are not bottlenecked by CPU. - **When to avoid:** Cluster nodes or the ABS host are already CPU-bound, or ABS and the cluster share a fast local network where decompression overhead outweighs bandwidth savings. - **Default:** `false`. Omit the parameter or set it to `false` to disable PI query compression. ## Tune writer bottlenecks Use the following parameters when `pipeline` stays near capacity, which means readers are waiting for writers. ### `parallel-write` (write parallelism) Controls the number of threads for serialization, compression, encryption, and uploading. If not set, it defaults to the value of `parallel`. - **When to increase:** Pipeline is near capacity and you have CPU and memory headroom. - **When to decrease:** CPU is saturated above 90% or memory usage is high. - **Default:** The value of `parallel`. - **Memory impact:** Each writer allocates: - A 256-record channel buffer - A storage upload buffer sized to `min-part-size`, which defaults to 50 MiB - ZSTD encoder state if compression is enabled, sized between about 1 MiB and 8 MiB depending on preset Reader and writer counts do not need to match. Set `parallel-write` based on writer bottlenecks and available CPU, memory, and storage bandwidth. ABS handles distribution between readers and writers. ### `compact` Controls how ABS encodes binary data types in `.asb` backup files: Bytes, HLL, RawMap, and RawList. When disabled (default), ABS Base64-encodes these values. Base64 is the default encoding across Aerospike backup tools and makes `.asb` files easier to inspect. Base64 expands binary data by about 33%. When enabled, ABS stores binary data directly. This produces smaller backup files for workloads with heavy binary bin usage and avoids the CPU cost of Base64 encoding and decoding. The same mode is available in [`absctl backup`](https://aerospike.com/docs/database/tools/backup-and-restore/absctl/backup/use#performance-and-throughput-options) as `-C` / `--compact`. - **When to enable:** Your full backup and restore toolchain supports compact `.asb` files and workloads include heavy binary bin usage. Set `compact: true` in `backup-policies`. - **When to avoid:** External tools or scripts parse `.asb` files and expect the default Base64-encoded binary bins. - **Default:** `false`. Omit the parameter or set it to `false` to use Base64 encoding (the default across Aerospike backup tools). ### `compression` (ZSTD) ABS applies ZSTD compression per writer thread. The `compression.level` parameter accepts values from -1 through 22. ABS uses a Go implementation of ZSTD that provides four rather than 22 discrete levels of compression. The library maps any numeric level to one of four presets. Values within the same bucket produce identical compression output, speed, and CPU usage. For meaningful configuration changes, use values at preset boundaries: `1`, `3`, `6`, and `10`. Use this table to map `compression.level` values to the ZSTD preset that ABS uses. | Level | Preset | Approximate ZSTD equivalent | Behavior | | --- | --- | --- | --- | | \-1, 0–2 | Fastest | ZSTD 1 | Least CPU, largest files | | 3–5 | Default | ZSTD 3 | Balanced speed and compression | | 6–9 | Better | ZSTD 7–8 | More CPU, smaller files | | 10–22 | Best | ZSTD 11 | Most CPU, smallest files | If `compression.level` is omitted, the API default is `0`. For balanced compression, set `compression.level` to `3`. Practical tips: - Start with `compression.level=3` to use the Default preset. You can also experiment with compression level `0` to get a baseline for speed and size when testing. - Compression helps most on high-latency or low-bandwidth storage paths when records are highly compressible. - If writer CPU saturates or `pipeline` trends toward capacity after enabling a more aggressive preset, reduce `compression.level`. - Move from level `3` to `6` only after validating with your own baseline runs. ### `file-limit` (max file size) Controls the maximum size of individual `.asb` backup files, in MiB. - **Default:** 250 MiB. - **When a file reaches this limit:** ABS starts writing a new file. - **No limit:** Set to `0`, which produces one file per writer. ::: note Very low values can create excess file rotation and object-operation overhead. Very high values, including `0`, keep each multipart upload open for the full backup. If a backup is canceled or fails, more data is stranded in incomplete multipart uploads until lifecycle rules or manual cleanup remove it. ::: ### `min-part-size` (multipart upload chunk size) Controls the minimum size of each part in a multipart upload to object storage. ABS buffers this amount of data in memory per writer before uploading a part. - **Default:** 50 MiB (52,428,800 bytes). - **Minimum values:** Minimum values vary by storage backend. Use this table to choose a value that meets the storage backend minimum. | Backend | Minimum `min-part-size` | | --- | --- | | S3 | 5 MiB | | GCS | 256 KiB | | Azure Blob Storage | 1 MiB | - **Memory impact:** This parameter is the primary driver of upload buffer memory. Peak memory from chunk buffers alone is approximately `parallel-write × min-part-size`. Larger min-part-size values reduce the number of chunk upload requests, which can improve throughput on high-latency connections. However, each writer holds this buffer in memory, so large values with high `parallel-write` risk out-of-memory (OOM) conditions. ### `bandwidth` (write speed cap) Throttles the total encoded backup write speed in MiB/s across all writers. Use this to prevent ABS from saturating a shared network connection. - **When to increase:** `pipeline` stays near capacity because the configured write cap is lower than the storage path can support, and the ABS host, network, and storage backend all have headroom. - **When to decrease:** Backup uploads are competing with other traffic, or the storage backend starts throttling concurrent writes. - **Default:** No limit. Omitting the parameter or setting it to `0` disables the bandwidth cap. The limit is shared by the writer pipeline. Increasing `parallel-write` can improve serialization, compression, encryption, and upload concurrency, but it cannot raise throughput above the configured `bandwidth` value. With bandwidth limiting, the writer stage runs at a fixed rate. If readers can produce faster than that cap, the pipeline trends toward capacity; if not, readers remain the bottleneck and `pipeline` can stay low. ## Verify tuning results After each tuning test, compare the results with your baseline run. Use short runs with representative data before applying the configuration to longer backups. Check the pipeline balance during the backup: Terminal window ```bash curl http://ABS_HOST:8080/v1/backups/currentBackup/ROUTINE_NAME | \ jq '{full: .full.metrics.pipeline, incremental: .incremental.metrics.pipeline}' ``` A tuned configuration has no sustained `pipeline` reading near `0` or capacity. Brief spikes to either extreme are expected during normal fluctuations. Also verify these signals: - Backup duration decreases or stays within your target window. - ABS host or container CPU stays below the limit you allow for backup work. - Memory usage stays below the host or container limit. - ABS logs do not show repeated `FAIL_FORBIDDEN` errors, out-of-memory errors, or storage throttling errors. - The storage backend receives the expected number and size of backup files. ## Limits and next steps Stop tuning a single ABS instance when throughput gains become marginal or CPU, memory, cluster load, network capacity, or storage throttling becomes the limiting factor. If one ABS instance is fully saturated, use [partition-list slicing](https://aerospike.com/docs/database/tools/backup-and-restore/backup-service/parallel-backup) to split workload across multiple instances. For longer-term visibility into backup duration and process metrics, see [ABS monitoring](https://aerospike.com/docs/database/tools/backup-and-restore/backup-service/monitoring).