Upgrade Aerospike Backup Service

Use this task to replace Aerospike Backup Service (ABS) 3.x with a later ABS 3.x release. Before you replace ABS, update the configuration for every intervening release. The replacement process rescans backup history from storage when routine names, storage roots, credentials, and persistent mounts remain unchanged.

Applies to

• Product: Aerospike Backup Service (ABS) 3.x.
• Audience: operators who run ABS on Linux, Docker, or Kubernetes with Helm.
• Outcome: /version reports the target version, routines are loaded, and the next scheduled backup runs on time.

For an ABS 2.x deployment managed by Aerospike Kubernetes Operator (AKO), follow Upgrade Aerospike Backup Service on Kubernetes. For other ABS 2.x deployments, contact Aerospike Support to confirm a supported migration path, then read Breaking changes in ABS 3.0.

Prerequisites

• You know the current ABS version from /version.
• The target release documentation supports upgrading from your current ABS release.
• The target package or image is published and, for Helm, you know its matching published chart version.
• You know the configured listener base URL, including the scheme, port, and context-path.
• Local backup storage, configuration, certificates, and credentials use persistent mounts that you can attach to the replacement process.
• A Helm deployment uses one replica with autoscaling disabled during the upgrade.
• You reviewed every breaking change between the current version and the target version in Breaking changes by version.

Caution

ABS validates the configuration strictly at startup and refuses to start on unknown or removed fields. Update the YAML before you replace ABS. An upgrade interrupts the service, and running jobs do not survive a restart.

Upgrade ABS

1. Record the current version.

   Set BASE_URL to the configured listener URL and context-path. The following value is the default:

   Terminal

   export BASE_URL=http://localhost:8080

   Terminal

   curl --silent --show-error --fail-with-body "$BASE_URL/version"

   Note the reported version so you know which breaking changes apply.

2. Save protected rollback information.

   Linux

   Save the configuration with permissions that restrict access to your account.

   Terminal

   install -m 600 /etc/aerospike-backup-service/aerospike-backup-service.yml "$HOME/abs-config-backup.yml"

   Keep the earlier package until the rollback window closes.

   Docker

   Keep the Compose file or original docker run command, including environment variables, networks, restart policy, resource limits, security options, secrets, volume mounts, and port mappings. Save the current container definition in a protected file:

   Terminal

   umask 077
   docker inspect backup-service > "$HOME/backup-service-inspect.json"

   Back it up separately (replace /path/to/aerospike-backup-service.yml with the host path):

   Terminal

   install -m 600 /path/to/aerospike-backup-service.yml "$HOME/abs-config-backup.yml"

   Helm

   Export all installed values and the rendered manifest, not only backupServiceConfig.

   Terminal

   umask 077
   helm get values abs --namespace aerospike --all > "$HOME/abs-values-backup.yaml"
   helm get manifest abs --namespace aerospike > "$HOME/abs-manifest-backup.yaml"
   helm history abs --namespace aerospike

   Record the revision that you can restore with helm rollback.

3. Pause routines and drain running jobs.

   Stop clients from submitting on-demand backup and restore requests. Disable each routine to prevent a scheduled job from starting while you replace ABS.

   Linux and Docker

   Terminal

   curl --silent --show-error --fail-with-body \
     -X PUT "$BASE_URL/v1/config/routines/ROUTINE_NAME/disable"

   Helm

   Set disabled: true on every routine in backupServiceConfig. Also set autoscaling.enabled: false and replicaCount: 1 in abs-values.yaml. Apply the paused configuration with the chart version that is already installed:

   Terminal

   helm upgrade abs aerospike/aerospike-backup-service \
     --namespace aerospike \
     --values abs-values.yaml \
     --version CURRENT_CHART_VERSION \
     --wait \
     --timeout 10m
   helm history abs --namespace aerospike

   CURRENT_CHART_VERSION is the chart version shown by helm history. Record this paused revision for rollback. Keep routines disabled in abs-values.yaml during the target upgrade.

   On Linux and Docker, the disable endpoint also cancels any backup that the routine is running. To let a running backup finish, wait for it to reach a terminal state before you disable the routine. Wait until every backup and restore reaches a terminal state. See Back up data with ABS and Restore data with ABS.

4. Update the configuration for breaking changes.

   Apply every entry between the current version (exclusive) and the target version (inclusive) in Breaking changes by version. Do not enable new optional fields yet. Upgrade first, then opt in. Validate the edited file against the target release’s config.schema.json when it is available. Startup still performs the definitive storage, credential, Secret Agent, and cluster checks.

5. Replace ABS with the target version.

   Linux

   Install the package for your distribution over the existing installation. The package manager preserves the configuration file.

   For Debian:

   Terminal

   dpkg -i aerospike-backup-service_3.7.0_amd64.deb

   For RPM:

   Terminal

   rpm -U aerospike-backup-service-3.7.0-1.x86_64.rpm
   systemctl enable --now aerospike-backup-service

   The explicit systemctl command protects against earlier package removal scripts that can leave the service stopped and disabled during an RPM upgrade.

   Docker

   Pull the target image.

   Terminal

   docker pull aerospike/aerospike-backup-service:3.7.0

   Stop and retain the old container for rollback:

   Terminal

   docker stop backup-service
   docker rename backup-service backup-service-rollback

   Run the recorded Compose or docker run definition with the target image. Preserve every runtime option and mount. Local backup paths and other persistent paths must use bind mounts or named volumes. See Run ABS in Docker for the base command.

   Helm

   Verify that local storage and log volumes are writable by UID 65532. Confirm that the installed revision has disabled routines, one replica, and no Horizontal Pod Autoscaler. The chart’s default rolling strategy can otherwise run old and new schedulers at the same time. Scale the Deployment to zero and wait for the old Pod to exit:

   Terminal

   kubectl scale deployment/abs-aerospike-backup-service \
     --namespace aerospike \
     --replicas=0
   kubectl wait \
     --namespace aerospike \
     --for=delete pod \
     --selector=app=abs-aerospike-backup-service \
     --timeout=5m

   Refresh the chart repository and upgrade with the chart version that corresponds to the target ABS release. TARGET_CHART_VERSION is the chart’s version, which is distinct from the ABS image tag.

   Terminal

   helm repo update
   helm upgrade abs aerospike/aerospike-backup-service \
   --namespace aerospike \
   --values abs-values.yaml \
   --version TARGET_CHART_VERSION \
   --set-string image.tag=3.7.0 \
   --set replicaCount=1 \
   --atomic \
   --timeout 10m

Verify

1. Confirm that the replacement process is running.

   Linux

   Terminal

   systemctl is-active aerospike-backup-service
   systemctl is-enabled aerospike-backup-service

   Docker

   Terminal

   docker ps --filter name=backup-service

   Helm

   Terminal

   kubectl rollout status deployment/abs-aerospike-backup-service \
     --namespace aerospike \
     --timeout=10m
   kubectl get pods \
     --namespace aerospike \
     --selector=app=abs-aerospike-backup-service

   Confirm that exactly one replacement Pod is running.

2. Confirm the version and the /health endpoint using the configured listener and context-path.

   Terminal

   curl --silent --show-error --fail-with-body "$BASE_URL/version"
   curl --silent --show-error --fail-with-body "$BASE_URL/health"

   The version response reports the target ABS version, and /health returns Ok.

3. Confirm that routines loaded and history was rescanned.

   Linux

   Terminal

   journalctl -u aerospike-backup-service --since "10 minutes ago" --follow

   Docker

   Terminal

   docker logs --since 10m --follow backup-service

   Helm

   Terminal

   kubectl logs \
     --namespace aerospike \
     deployment/abs-aerospike-backup-service \
     --since=10m \
     --follow

   Follow the logs until they show History synchronization completed and a Last existing backup entry for every expected routine, then press CTRL+C. Investigate any storage, credential, or configuration errors before you resume schedules.

4. Enable each routine:

   Linux and Docker

   Terminal

   curl --silent --show-error --fail-with-body \
     -X PUT "$BASE_URL/v1/config/routines/ROUTINE_NAME/enable"

   Helm

   Set disabled: false on each routine in abs-values.yaml, then apply the values with the target chart:

   Terminal

   helm upgrade abs aerospike/aerospike-backup-service \
   --namespace aerospike \
   --values abs-values.yaml \
   --version TARGET_CHART_VERSION \
   --set-string image.tag=3.7.0 \
   --wait \
   --timeout 10m

5. Observe the next scheduled backup.

   Confirm that it starts on time, reaches a successful terminal state, and appears in backup history. An on-demand full backup can invoke retention, so use an isolated routine and storage prefix if you need an immediate canary. See Verify a backup.

Breaking changes by version

ABS has no automatic configuration migration. Read every section between your current version (exclusive) and your target version (inclusive), and apply all of them before you restart. The listed ABS 3.x patch releases add no further breaking changes.

Breaking changes in ABS 3.7.0

See the Aerospike Backup Service 3.7.0 release notes for details.

• Configuration API redacts secrets. GET responses replace literal secrets with "[secret]". Aerospike Secret Agent references (secrets:...) are returned unchanged. If you retrieve configuration with GET, edit it, and update it with PUT, leave "[secret]" in unchanged secret fields. Sending "[secret]" for a new entity is invalid. Prefer editing the YAML file and applying it with POST /v1/config/apply. See API secrets.
• Configuration endpoints return 404 and 409. Endpoints that create, update, or delete routines, storage, clusters, and policies, and the routine enable and disable endpoints, return 404 Not Found for an entity that does not exist. They return 409 Conflict for a name that is already taken, or when you delete an entity that a routine still references. A malformed body or failed validation still returns 400 Bad Request. Update clients that branch on 400 or match response text.
• Configuration paths must be canonical. ABS validates local, S3, GCP, and Azure storage paths, restore backup-data-path, password paths, TLS and HTTPS files, encryption-key files, GCP key files, Secret Agent TLS files, and log file paths. Paths must not contain .., start with ~, contain redundant separators, end with /, or use a non-canonical prefix such as ./. Relative storage paths must stay under their storage root.
• Names must be usable as a path segment. Routine, storage, cluster, policy, and Secret Agent names cannot be . or .., contain /, \, or a NUL byte, start with ~, or start or end with whitespace. Namespace names in routines and restore requests must be at most 31 bytes of Latin letters, digits, _, -, and $, and cannot be null.
• Compression and encryption blocks require mode. Set mode in every compression and encryption block in backup policies and restore requests. ZSTD compression requires level from -1 to 22, and NONE must not set level. To turn off compression or encryption, set mode: NONE or omit the block.
• An empty rate-limiter white-list limits every client. To disable rate limiting for all clients, use 0.0.0.0/0, which represents all IPv4 addresses in Classless Inter-Domain Routing (CIDR) notation. A value such as 0.0.0.0 without /0 is not a full exemption. See HTTP server.
• Linux packages run ABS as an unprivileged account. The .deb and .rpm packages create the aerospike-backup-service account and run ABS under a systemd sandbox. Most of the following problems appear only when ABS first uses the path or file, so check them before you upgrade:
  • Only /etc/aerospike-backup-service, /var/lib/aerospike-backup-service, and /var/log/aerospike-backup-service are writable. For a local-storage path anywhere else, add it to ReadWritePaths with systemctl edit aerospike-backup-service, and make aerospike-backup-service the owner of the directory. The package installs an example drop-in at /usr/share/doc/aerospike-backup-service/local-storage-path.conf.example.
  • /root and /home are hidden. Move cloud credentials such as ~/.aws/credentials under /var/lib/aerospike-backup-service, or set them in /etc/default/aerospike-backup-service (Debian) or /etc/sysconfig/aerospike-backup-service (RPM).
  • Cluster and HTTPS TLS files and password-path files must be mode 0600 with owner aerospike-backup-service.
  • A listener on a port below 1024 needs CAP_NET_BIND_SERVICE, granted with AmbientCapabilities and CapabilityBoundingSet in a drop-in.
  • The log file moves to /var/log/aerospike-backup-service/aerospike-backup-service.log. If you customized the configuration file, update service.logger.file-writer.filename, because ABS refuses to start when it cannot write its log file.
  • The unit file moves to /usr/lib/systemd/system. The package renames an edited unit file in /etc/systemd/system to aerospike-backup-service.service.pre-hardening.bak. Reapply its changes as a drop-in.
  • The RPM upgrade to 3.7.0 leaves ABS stopped and disabled. The systemctl enable --now command in Upgrade ABS restarts it.
• Local backup files are owner-only. In every deployment, local-storage backups are created with file mode 0600 and directory mode 0700, owned by the account ABS runs as. A tool that reads local backups as another user, such as asrestore run from a separate account, must run as the ABS account, or you must restore through the REST API.
• The Helm chart is hardened. The container runs as non-root UID 65532 with a read-only root filesystem, drops all capabilities, and does not automount the service account token. Local disk storage or a log file on disk needs an extra writable volume. See Install ABS on Kubernetes with Helm.
• List fields reject empty and duplicate values. The namespaces, set-list, bin-list, node-list, and rate-limiter white-list fields fail validation on empty strings or duplicates. The rate-limiter tps and size values must be positive.
• Secret Agent references validate at load. A malformed secrets: path, or a reference without a configured agent, fails configuration load or apply. New jobs that need a referenced secret fail when Secret Agent is unreachable. Running jobs keep the values they already resolved.
• New listener timeouts apply by default. The read-timeout (30 seconds), write-timeout (60 seconds), and idle-timeout (120 seconds) values take effect when omitted.
• Set indexes are backed up. On Database 8.1.2 and later, a namespace backup now includes set indexes unless the policy sets no-indexes. When a namespace has a set index, ABS writes the backup metadata in ASB protocol version 3.3, which ABS 3.6.2 and earlier, absctl 1.1.1 and earlier, and asrestore cannot read. See Roll back a failed upgrade.

Breaking changes in ABS 3.6

ABS 3.6.0 introduced these changes. They are documented in the Aerospike Backup Service 3.6.1 release notes. ABS 3.6.1 and ABS 3.6.2 add no further breaking changes.

• Restore job status is lowercase. Values are running, success, failure, and canceled. The previous values were Running, Done, Failed, and Canceled. Update integrations that compare these values.
• The default log level is INFO. Set service.logger.level: DEBUG explicitly to keep verbose logging.
• The default cloud min-part-size is 50 MiB. This applies to S3, Azure, and GCP storage. At the default parallel-write value of 8, chunk buffers can use about 400 MiB per active namespace. See min-part-size tuning.
• Chunk-count validation can reject cloud storage. Configurations that need 10,000 or more chunks per backup file fail validation. Increase min-part-size or decrease file-limit. A file-limit of 0 disables the check.
• parallel is enforced per routine. The total concurrent primary index (PI) queries across all namespaces in a routine cannot exceed parallel. Confirm backup duration still fits the schedule window.
• parallel cannot exceed max-parallel-scans. When the source cluster sets that limit, an explicit parallel value above it fails validation at startup and on configuration change.
• restore_in_progress counts active jobs only. Update queries or alerts that relied on completed jobs in this gauge.
• Startup checks storage permissions in more detail. Missing read or list permission on S3, GCS, or Azure prevents startup. Write or delete failures warn at startup and can fail backups or cleanup later.

Breaking changes in ABS 3.5

• Startup checks cloud storage connectivity. ABS checks configured S3, GCS, and Azure resources when it starts. An inaccessible bucket or container, invalid credentials, or a connectivity failure can prevent startup.

ABS 3.5.1 adds no further breaking changes.

Breaking changes in ABS 3.4.0

See the Aerospike Backup Service 3.4.0 release notes for details.

• prefer-racks moved to the cluster. Configure it under aerospike-clusters, not under backup-routines. The old location fails validation. See Rack awareness: prefer-racks vs rack-list.
• Backup retry delays increase by default. The default retry-policy.multiplier changed from 1 to 1.5. Set it to 1 explicitly to retain the constant retry delay from ABS 3.3.1.

Breaking changes in ABS 3.3.0

See the Aerospike Backup Service 3.3.0 release notes for details.

• TLS validation is strict. Incomplete or inconsistent TLS settings fail startup. Review certificate, key, and CA parameters.
• Restore-by-timestamp reads storage from the routine. Do not include storage configuration in the request. ABS 3.6 added optional source and source-name overrides for cross-region restores.
• Default restore concurrency is lower. The default restore-policy parallel value changed from 20 to 8. Set parallel: 20 explicitly to retain the previous concurrency after checking that the target cluster and ABS host have sufficient capacity.

Breaking changes in ABS 3.2.0

See the Aerospike Backup Service 3.2.0 release notes for details.

• namespaces is mandatory. Omitting it no longer backs up all namespaces. Use namespaces: [] for a whole-cluster backup.
• Restore bandwidth is MiB/s. Convert bytes-per-second values. The minimum is 8 MiB/s.
• The partition-list default is null. The effect is unchanged: all partitions are backed up.
• The aerospike_backup_service_restore_progress_pct metric is removed. Poll /v1/restore/status/{jobId} for restore progress.

Breaking changes in ABS 3.1.0

See the Aerospike Backup Service 3.1.0 release notes for details.

• Remaining parameters use kebab-case. Rename max_async_connections to max-async-connections and min_part_size to min-part-size. The s3-log-level field already used kebab-case.
• Validation is strict at startup. Invalid configurations prevent the service from starting.
• Default backup concurrency is higher. The default backup-policy parallel value changed from 1 to 8. Set parallel: 1 explicitly to retain the previous cluster, network, memory, and storage load.

Breaking changes in ABS 3.0

This section applies when you upgrade from ABS 2.x through a migration path confirmed by Aerospike Support or AKO documentation. See the Aerospike Backup Service 3.0.0 release notes for details.

• Parameters use kebab-case. Rename snake_case keys, except max_async_connections and min_part_size, which changed in 3.1.0.
• secret-agent is now secret-agents.
• Storage uses per-type blocks. Replace the unified type field with one of local-storage, s3-storage, azure-storage, or gcp-storage. Only one type is allowed per storage entry.
• S3 bucket and path are separate fields. Split each s3://BUCKET/PATH value into separate bucket and path fields.
• Path-based restore requires backup-data-path. Requests to /v1/restore/full and /v1/restore/incremental use this field. The Storage.path field is the storage root only.
• Configuration API changes apply immediately. The separate apply step is gone. Each update saves to the file and applies to the runtime. Invalid updates are rejected. The apply endpoint remains for reloading a file edited outside the API.
• remove-files is now retention. Use retention.full and retention.incremental counts instead of KeepAll, RemoveAll, and RemoveIncremental.

After you upgrade

New-version features are opt-in. Existing configurations that omit the new fields keep UTC schedules, differential incrementals, and HTTP-only listen behavior. Enable only the features you need.

Optional change	Why
service.https	Terminate TLS on ABS. Set service.http.disabled: true for TLS-only. Mount certs at stable paths so rotation does not require a restart. For mutual TLS, set client-auth and client-ca-file. Add crl-file to revoke client certificates (require-and-verify only).
incr-mode: cumulative	Shorter restore-by-timestamp chains. Each incremental copies everything changed since the last full backup, so storage and query cost grow until the next full.
schedule-timezone	Evaluate cron in Local or an IANA zone. Omit the field to keep UTC. Set the host TZ only if you use Local. Backup folder names stay UTC.
filter-exp	Partial backups of records that match a Base64 filter expression. Mutually exclusive with a multi-set set-list.

Roll back a failed upgrade

Roll back if the replacement process does not start or cannot load its routines and history after you correct configuration, storage, and credential errors.

Caution

Before you roll back, confirm that the earlier ABS release can read every backup written after the upgrade. If compatibility is not documented, stop scheduled jobs and preserve the later-version backup data until Aerospike Support confirms a recovery path.

ABS 3.6.2 and earlier cannot restore a backup that ABS 3.7.0 or later took of a namespace that has a set index. ABS 3.7.0 writes the metadata of that backup in ASB protocol version 3.3, which asrestore and absctl 1.1.1 and earlier cannot read either. Keep those backups after you roll back. ABS 3.7.0 and later and absctl 1.2.0 and later can read version 3.3 metadata.

1. Disable routines and stop clients from submitting on-demand jobs.

2. Stop the replacement process.

3. Restore the earlier release.

   Linux

   Stop ABS and restore the protected configuration before the earlier package’s post-install script starts the service:

   Terminal

   systemctl stop aerospike-backup-service
   cp "$HOME/abs-config-backup.yml" /etc/aerospike-backup-service/aerospike-backup-service.yml

   For Debian, install the earlier package:

   Terminal

   dpkg -i OLD_DEB_PACKAGE

   To roll back from ABS 3.7.0 and later to an earlier Debian package, purge the package first, because the earlier package’s post-install script fails otherwise. Purging removes the configuration file and the aerospike-backup-service account, but keeps backup data. Restore the configuration after you install the earlier package:

   Terminal

   apt-get purge -y aerospike-backup-service
   dpkg -i OLD_DEB_PACKAGE
   systemctl stop aerospike-backup-service
   cp "$HOME/abs-config-backup.yml" /etc/aerospike-backup-service/aerospike-backup-service.yml
   systemctl start aerospike-backup-service

   For RPM, permit installation of the earlier package:

   Terminal

   rpm -U --oldpackage OLD_RPM_PACKAGE
   systemctl enable --now aerospike-backup-service

   Docker

   Stop the replacement container, restore the earlier configuration to HOST_CONFIG_PATH, restore the retained container name, and start it:

   Terminal

   docker stop backup-service
   docker rm backup-service
   cp "$HOME/abs-config-backup.yml" HOST_CONFIG_PATH
   docker rename backup-service-rollback backup-service
   docker start backup-service

   Helm

   Prevent Pod overlap, wait for the replacement Pod to exit, then restore the paused revision recorded before the upgrade:

   Terminal

   kubectl scale deployment/abs-aerospike-backup-service \
     --namespace aerospike \
     --replicas=0
   kubectl wait \
     --namespace aerospike \
     --for=delete pod \
     --selector=app=abs-aerospike-backup-service \
     --timeout=5m
   helm rollback abs PAUSED_REVISION \
     --namespace aerospike \
     --wait \
     --timeout 10m

4. Repeat the version, response, log, history, and scheduled-backup checks in Verify.

Troubleshoot

• The process exits at startup with a field error. The message names the unknown or invalid field, such as prefer-racks under a routine or a type field under storage. Fix the YAML per Breaking changes by version and restart.
• Startup reports storage permission errors. ABS 3.5 and later check S3, GCS, and Azure connectivity during startup. ABS 3.6 adds specific read, list, write, and delete permission checks. Grant the required permissions and restart.
• A Helm Pod cannot write local backups or logs. Verify that the volume is mounted read-write and grants UID 65532 or a compatible fsGroup write access.
• Backups run longer after the upgrade. ABS 3.6 enforces parallel per routine instead of per namespace. Retune reader parallelism. See Aerospike Backup Service performance tuning.
• Monitoring alerts changed behavior. Restore status values, the restore_in_progress gauge, and the removed restore_progress_pct metric affect dashboards carried from earlier versions. Update queries as described in Breaking changes in ABS 3.6 and Breaking changes in ABS 3.2.0.

Next steps

• Install ABS
• Configure ABS
• Secure ABS
• Back up data
• Restore data
• Aerospike Backup Service release notes