Estimate resources for Aerospike backup

Estimate memory, disk space, and file descriptor usage for asbackup.

Overview

• asbackup can use significant resources. Run the tool on a host or container separate from Aerospike Database.

• asbackup resource consumption depends on the options that you set when running the tool.

How asbackup uses memory, storage, and file descriptors is described in the following explanations and formulas. The command line flags used in the following formulas are defined in Run Aerospike backup.

Note

The information outlined here applies to asbackup 3.11.0 and later.

These formulas describe how resource usage scales. They do not provide exact numbers for resource usage. You should allocate slightly more resources to asbackup than the formulas suggest.

Estimate memory usage for a backup

asbackup’s memory usage is largely affected by the --parallel flag. By default, asbackup distributes work across threads by assigning each thread a unique file to write to. These files each have a constant size buffer associated with them. The buffer in asbackup is 4KiB.

Formula to approximate memory usage for backup

To calculate the approximate amount of memory usage that will be used for a backup:

• Multiply the value that you set for the --parallel flag by 4KiB.

The result is the approximate amount of memory needed to back up your data.

Backing up to Amazon S3 uses more memory to maintain the internal S3 client and buffers.

Estimate disk space for a backup

To estimate how much disk space is needed to back up your data, use the --estimate flag when running asbackup. Using the --directory and --output-file flags is not necessary.

You cannot use the --estimate flag and --parallel flag at the same time. The two flags are mutually exclusive.

asbackup --namespace NAME --estimate

The --estimate flag reads 10,000 records by default from the specified namespace and prints the average size of the sampled records. The number of records can be changed with --estimate-samples.

Formula to estimate disk space for a backup

To estimate the amount of disk space needed for a backup:

1. Multiply the estimated record size, returned by the --estimate flag, by the number of records in the namespace.

2. Add 10% of the estimated record size.

The result is the approximate disk space needed for a backup.

Note

Per-record filters (filter-exp, modified-after, modified-before, no-ttl-only, after-digest, and partition-list) and node-list are not accounted for in the estimate, and using these options will have no effect on the estimate.

Calculate number of file descriptors

asbackup may need to open many backup files and network sockets. If asbackup is unable to open the required number of file descriptors, it can fail with “too many open files” errors. Use the following guidance to estimate the required file descriptors.

By default, asbackup opens a new backup file for each of its --parallel threads. Each thread may have to open a network socket to each node in the cluster.

Formula to calculate approximate file descriptors for backup

To calculate the approximate number of file descriptors needed for a backup:

1. Multiply the value that you set for the --parallel flag by the value returned from nodes_in_cluster.

2. Add the result of the previous step to the value returned from the backup_file_count flag.

In this calculation, backup_file_count is equal to --parallel when using --directory.

The final sum is the approximate number of file descriptors needed for a backup.