ABS Configuration

Overview

Aerospike Backup Service (ABS) reads the configuration file aerospike-backup-service.yml to create connections to Aerospike Database namespaces and storage destinations, as well as backup policies, routines, and schedules.

This page explains two sample configuration files included in the ABS GitHub repository and details about the most common parameters that you may want to adjust. These parameters cover a majority of use cases for testing ABS in an application workflow.

For a full list of parameters, see the Schemas section in the REST API specification.

Example configuration files

Docker Compose configuration

The following sample backup service configuration supplied with the Docker Compose stack for ABS consists of four sections:

• aerospike-clusters defines the location and access credentials for ABS to communicate to the Aerospike database, calling that cluster absCluster1. Since this is a Docker Compose stack, it uses the Aerospike Database Docker container name "aerospike-cluster" as the hostname instead of an IP address.
• storage defines the location of the storage for database backups. Here, it creates a storage type called minioStorage that uses the s3-endpoint-override parameter to send backed up data to MinIO instead of Amazon S3. You can define multiple storage types that can later be used in multiple backup policies.
• In backup-policies, a new policy called keepFilesPolicy is defined with simple instructions to run in a single thread and keep all previous backups. A policy is a set of instructions defining how to do a specific type of backup. You can define multiple policies that can be used in various backup routines.
  note

  Backup policies are defined in the configuration file. In contrast, restore policies are not defined beforehand; they are sent in the body of each restore request.

• backup-routines specifies a routine called minioKeepFilesRoutine that runs the keepFilesPolicy policy daily for full backups and hourly for incremental backups. Routines specify the source cluster to back up data from, a storage type as defined under the storage section, and a namespace from the source cluster to back up. You can define multiple routines that can be run according to different schedules or on demand.

Default Docker Compose ABS Configuration:

aerospike-clusters:
  absCluster1:
    seed-nodes:
      - host-name: "aerospike-cluster"
        port: 3000
    credentials: 
      user: admin
      password: admin
      
storage:
  minioStorage:
    # Use "aws-s3" for S3 or compatible and "local" for local storage.
    type: "aws-s3"
    # as-backup-bucket is expected to exist in MinIO.
    path: s3://as-backup-bucket/minioStorage
    s3-region: eu-central-1
    s3-profile: minio
    s3-endpoint-override: http://minio:9000
    
backup-policies:
  keepFilesPolicy:
    # Run backup operations in a single thread.
    parallel: 1
    # Previous full backups are not deleted when a new one is created.
    remove-files: KeepAll
    
backup-routines:
  minioKeepFilesRoutine:
    # 24 hours interval for full backups.
    interval-cron: "@daily"
    # 1 hour interval for incremental backups.
    incr-interval-cron: "@hourly"
    source-cluster: absCluster1
    storage: minioStorage
    namespace: test
    backup-policy: keepFilesPolicy

Linux configuration

The default configuration file supplied with Linux distributions is smaller and simpler than the configuration in the Docker Compose setup. By default, it sets up a connection to a namespace called "test" in an Aerospike database accessible at 127.0.0.1:3000. It stores backup files locally at /var/lib/aerospike-backup-service.

Default Linux ABS Configuration:

aerospike-clusters:
  cluster1:
    use-services-alternate: false
    seed-nodes:
      - host-name: "127.0.0.1"
        port: 3000
    credentials:
      user: "admin"
      password: "admin"

storage:
  local1:
    type: "local"
    path: "/var/lib/aerospike-backup-service"

backup-policies:
  policy1:
    type: 1
    parallel: 1
    remove-files: KeepAll

backup-routines:
  routine1:
    interval-cron: "1/30 * * * * *" # every 30 seconds
    incr-interval-cron: "1/5 * * * * *" # every 5 seconds
    backup-policy: "policy1"
    source-cluster: "cluster1"
    storage: "local1"
    namespace: "test"

aerospike-clusters

Parameter	Type	Description	Example
conn-timeout	integer	Connection timeout in milliseconds.	5000
credentials	object	Authentication details to the Aerospike cluster.
label	string	Cluster name.	testCluster
seed-nodes	array	Seed nodes details.
tls	object	Cluster TLS configuration.
use-services-alternate	boolean	Whether to use "services-alternate" instead of "services" in info request during cluster tending.	false

credentials

aerospike-clusters -> credentials

Parameter	Type	Description	Example
auth-mode	string	Authentication mode string (INTERNAL, EXTERNAL, EXTERNAL_INSECURE, PKI).
password	string	Password for cluster authentication.	testPswd
password-path	string	File path with the password string, will take precedence over the password field.	/path/to/pass.txt
user	string	Username for cluster authentication.	testUser

seed-nodes

aerospike-clusters -> seed-nodes

Parameter	Type	Description	Example
host-name	string	Host name of the node.	localhost
port	integer	Port of the node.	3000
tls-name	string	Optional TLS certificate name used for secure connections.	certName

tls

aerospike-clusters -> tls

Parameter	Type	Description	Example
ca-file	string	Path to a trusted CA certificate file.	/path/to/cafile.pem
ca-path	string	Path to a directory of trusted CA certificates.	/path/to/ca
cert-file	string	Path to the chain file for mutual authentication if Aerospike Cluster supports it.	/path/to/certfile.pem
cipher-suite	string	TLS cipher selection criteria. The format is the same as OpenSSL's Cipher List Format.	ECDHE-ECDSA-AES256-GCM-SHA384
key-file	string	Path to the key for mutual authentication (if Aerospike cluster supports it).	/path/to/keyfile.pem
key-file-password	string	Password to load protected TLS-keyfile (env:VAR, file:PATH, PASSWORD).	file:/path/to/password
name	string	Default TLS name used to authenticate each TLS socket connection.	tls-name
protocols	string	TLS protocol selection criteria. This format is the same as Apache's SSL Protocol.	TLSv1.2

storage

Parameter	Type	Description	Example
path	string	Root path for the backup repository.	backups
s3-endpoint-override	string	Alternative endpoint for the S3 SDK to communicate (AWS S3 optional).	http://host.docker.internal:9000
s3-log-level	string	Log level of the AWS S3 SDK (AWS S3 optional).	FATAL
s3-profile	string	The S3 profile name (AWS S3 optional).	default
s3-region	string	S3 region string (AWS S3 optional).	eu-central-1
type	string	Type of the storage provider, "local" or "aws-s3".	local

backup-policies

Parameter	Type	Description	Example
bandwidth	integer	Throttles backup write operations to the backup file(s) to not exceed the given bandwidth in MiB/s.	10000
compression	object	Compression details.
encryption	object	Encryption details.
file-limit	integer	File size limit (in MB) for the backup directory. If an .asb backup file crosses this size threshold, a new backup file is created.	1024
max-records	integer	Approximate limit for the number of records to process. Available in Database 4.9 and later.	10000
max-retries	integer	Maximum number of retries before aborting the current transaction.	3
no-bins	boolean	Only backup record metadata; digest, TTL, generation count, key.
no-indexes	boolean	Whether to back up secondary index definitions.
no-records	boolean	Whether to back up record data, metadata or bin data.
no-udfs	boolean	Whether to back up UDF modules.
parallel	integer	Maximum number of scan calls to run in parallel.	1
records-per-second	integer	Limit total returned records per second (RPS). If RPS is zero (the default), the records-per-second limit is not applied.	1000
remove-artifacts	boolean	Clear directory or remove output file.
remove-files	object	Whether to clear the output directory when a backup is written to it.	Options are: KeepAll (do not remove anything), RemoveAll (clear the directory before writing a backup), and RemoveIncremental (remove any incremental backups when a full backup is performed, leaving any existing full backups untouched).
retry-delay	integer	Delay in milliseconds to wait before retrying a failed operation.	500
sealed	boolean	Whether backup should include keys updated during the backup process. When true, the backup contains only records that last modified before backup started. When false (default), records updated during backup might be included in the backup, but it's not guaranteed.
socket-timeout	integer	Socket timeout in milliseconds. If this value is 0, it is set to total-timeout. If both socket-timeout and total-timeout are 0, there is no socket idle time limit.	1000
total-timeout	integer	Total socket timeout in milliseconds. Default is 0, that is, no timeout.	2000

compression

backup-policies -> compression

Parameter	Type	Description	Example
level	integer	Compression level to use, or -1 if unspecified.
mode	string	Compression mode to use. Options are NONE (default) or ZSTD.	NONE

encryption

backup-policies -> encryption

Parameter	Type	Description	Example
key-env	string	Name of the environment variable containing the encryption key.
key-file	string	Path to the file containing the encryption key.
key-secret	string	Secret keyword in Aerospike Secret Agent containing the encryption key.
mode	string	Encryption mode to use. Options are NONE, AES128, or AES256.	NONE

backup-routines

Parameter	Type	Description	Example
backup-policy	string	Name of the corresponding backup policy.	"daily"
bin-list	array	List of bin names to back up. An empty list backs up all bins.	["dataBin"]
incr-interval-cron	string	Interval for incremental backup as a cron expression string (optional).	*/10 * * * * *
interval-cron	string	Interval for full backup as a cron expression string.	0 0 * * * *
namespaces	array	List of namespaces to back up. Optional: Empty list implies backup of whole cluster.	["source-ns1"]
partition-list	string	Back up list of partition filters. Partition filters can be ranges, individual partitions, or records after a specific digest within a single partition. Default number of partitions to back up: 0 to 4095: all partitions.	0-1000
prefer-racks	array	List of Aerospike Database rack IDs to prefer when reading records for a backup.	[0]
secret-agent	string	The Secret Agent configuration for the routine (optional).	sa
set-list	array	List of set names to back up. An empty list backs up all sets.	["set1"]
storage	string	Name of the corresponding storage provider configuration.	aws