Parallel backups

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

You can run two or more identical Aerospike Backup Service (ABS) instances, each pointed at a different slice of the cluster, to back up and restore the full data set in parallel. You declare the cluster slice with node-list on each backup routine.

ABS treats every backup routine as an independent job definition.

If you configure two backup routines on separate ABS instances that share the same source cluster and storage destination, and differ only in the nodes each routine reads, you can start both routines at the same time, double the aggregate scan bandwidth, and still end up with a complete and non-overlapping backup set.

:::caution Topology changes during backup When the job starts, ABS scans the listed nodes. If partition ownership changes during a long backup, scans can read from the current master for a selected partition, which may be on a different node than when the job started. If a partition’s master migrates onto a listed node after the job starts, that partition was not in the initial slice and is not included in that routine’s backup. See the topology note in ABS configuration file examples. :::

Prerequisites

A running Aerospike Database deployment
Two or more ABS instances
asbench (for generating test data)
asadm (for verifying test data restoration)

Configuration file sections

The following example shows the minimum content necessary for the first part of the configuration file, aerospike-backup-service.yml.

The aerospike-clusters and storage sections are identical between ABS instances in a parallel deployment. Only the cluster slice changes between the two backup routines in the backup-routines section.

aerospike-clusters:
  absDefaultCluster:
    seed-nodes:
      - host-name: 10.0.48.7
        port: 4333
        tls-name: asd.aerospike.com
    credentials:
      user: tester
      password: psw
    tls:
      name: asd.aerospike.com
      ca-file: /etc/ssl/certs/ca.aerospike.com.pem

storage:
  gcp:
    gcp-storage:
      bucket-name: abs-testing-bucket
      path: parallel-backup-test

See ABS configuration file examples and ABS configuration file reference for more example configuration files and information about additional parameters.

Backup routines

The following examples show two backup routines that differ only in their node-list.

Assign each routine a slice that does not overlap the others: together they must cover every node you intend to back up, with no node scanned by more than one routine.

backup-routines:
  fullBackupA:
    source-cluster: absDefaultCluster
    storage: gcp
    interval-cron: '@daily'
    namespaces: [source-ns1]
    node-list:
      - 10.0.48.7:asd.aerospike.com:4333
      - 10.0.48.8:asd.aerospike.com:4333

backup-routines:
  fullBackupB:
    source-cluster: absDefaultCluster
    storage: gcp
    interval-cron: '@daily'
    namespaces: [source-ns1]
    node-list:
      - 10.0.48.9:asd.aerospike.com:4333
      - 10.0.48.10:asd.aerospike.com:4333
      - 10.0.48.11:asd.aerospike.com:4333

node-list limits each routine to the listed cluster nodes. Use node-list when you run multiple ABS instances in parallel and want each instance to back up a fixed set of cluster nodes, such as nodes in one availability zone or datacenter. Each routine is potentially partial: together, non-overlapping node-list slices must cover every node you intend to back up. If partition ownership changes during a long backup, see the topology note in ABS configuration file examples.

Example parallel backup and restore workflow

This section outlines a series of steps to write test records to a namespace, set up parallel backups with ABS, run the backups, and restore the database.

Follow this end-to-end workflow to populate a namespace with test data, configure parallel backups using ABS, and verify the process through a full database restore.

Execute a parallel backup test

Generate test data using asbench.

The benchmark tool writes one million 1 KiB records into the namespace:

asbench -U tester -P psw -h 10.0.48.7:asd.aerospike.com:4333 -n source-ns1 -w I -k 1000000 -o B960 --tls-enable --tls-cafile=/etc/ssl/certs/ca.aerospike.com.pem

See the asbench documentation for more information.

Start each backup routine separately through the REST API:
Terminal window
```
curl -X POST http://service1:8080/v1/backups/schedule/fullBackupA
curl -X POST http://service2:8080/v1/backups/schedule/fullBackupB
```
A POST request to the /v1/backups/schedule/{routine} endpoint triggers an immediate full backup, regardless of how the backup routine is scheduled in its interval-cron parameter. Each routine writes to parallel-backup-test/fullBackupA/<timestamp> or parallel-backup-test/fullBackupB/<timestamp>.
Inspect the metadata.yaml file in each namespace’s subdirectory inside the backup output directory. Their record-count values should sum to the expected total of 1000000. ABS writes one metadata file per namespace, per routine.

Restore test data

Truncate the target cluster so the restore starts without any data present. Run the three following commands in sequence.
Terminal window
```
asadm -h 10.0.48.7:asd.aerospike.com:4333 -U tester -P psw --tls-enable --tls-cafile=/etc/ssl/certs/ca.aerospike.com.pem

Admin> enable

Admin+> manage truncate ns source-ns1
```
The manage truncate command removes all records immediately.

Create separate request bodies for your restore requests.

{
    "routine": "fullBackupA",
    "time": '"$(($(date +%s)*1000))"',
    "destination": {
      "seed-nodes": [
        {
          "host-name": "10.0.48.7",
          "port": 4333,
          "tls-name": "asd.aerospike.com"
        }
      ],
      "credentials": {
        "user": "tester",
        "password": "psw"
      },
      "tls": {
        "name": "asd.aerospike.com",
        "ca-file": "/etc/ssl/certs/ca.aerospike.com.pem"
      }
    },
    "policy": {
      "parallel": 8,
      "batch-size": 100
    }
  }

{
    "routine": "fullBackupB",
    "time": '"$(($(date +%s)*1000))"',
    "destination": {
      "seed-nodes": [
        {
          "host-name": "10.0.48.7",
          "port": 4333,
          "tls-name": "asd.aerospike.com"
        }
      ],
      "credentials": {
        "user": "tester",
        "password": "psw"
      },
      "tls": {
        "name": "asd.aerospike.com",
        "ca-file": "/etc/ssl/certs/ca.aerospike.com.pem"
      }
    },
    "policy": {
      "parallel": 8,
      "batch-size": 100
    }
  }

Restore each routine’s backup with the JSON requests you defined in the previous step.
Terminal window
```
curl -X POST http://service1:8080/v1/restore/timestamp -H "Content-Type: application/json" -d @restoreA.json

curl -X POST http://service2:8080/v1/restore/timestamp -H "Content-Type: application/json" -d @restoreB.json
```
The JSON body follows the schema in the API examples and includes parallel and batch-size under policy to control writer concurrency.

Note
You can also pass the raw JSON content after the -d flag rather than saving it as a file.

Verify the object counts:

asadm -h 10.0.48.7:4333 -U tester -P psw --tls-enable --tls-cafile=/etc/ssl/certs/ca.aerospike.com.pem
Admin> info

Each of the five nodes should report about 200,000 objects and zero migrations.

Scaling guidelines

ABS exposes a cluster-wide throttle with the max-parallel-scans parameter. Keep this value below the total vCPU count of the Aerospike nodes. Adjust per-reader threads with parallel in a backup policy. Adjust writer batch size with batch-size in the restore request.

You can monitor throughput using the built-in /metrics endpoint, and basic health with /health and /ready.