Backup and Recovery in AWS

This page describes backup and recovery options for Aerospike Database on AWS, including Aerospike backup tools, Amazon S3, EBS snapshots, and EBS volume recovery. Use Aerospike backup tools for logical backups and restores. Use EBS snapshots or EBS volume recovery when your namespace storage depends on EBS volumes.

Aerospike tools

For new backup automation, start with Aerospike Backup Service. For command-line backups and restores, use absctl backup and absctl restore.

Backup

1. Back up data with absctl backup to local storage or Amazon S3.
2. Store local backup files in a durable location.

Restore

Upload the backup file to a system with absctl restore, or restore directly from Amazon S3.

cat BACKUP.GZ | gunzip | absctl restore [...] --input-file -

Benefits

• Portable across instance types and storage layouts
• Can include system metadata (SMD)
• Supports direct Amazon S3 backup and restore workflows

Drawbacks

• Long duration
• Additional server load while running
• All or nothing across the cluster

EBS snapshots

If you are using persistence, either through EBS directly, or using shadow device(s), you can take advantage of EBS snapshots. EBS snapshots are point-in-time backups of data. This is done at the data block level, transparent to the application (Aerospike).

Backup

To create a backup, initiate snapshots on your EBS data volumes.

Note

• EBS snapshots are consistent within a volume, but may not be consistent between volumes. As such, writing to a particular system needs to be stopped entirely, if said system utilizes multiple EBS volumes per namespace. This is due to the (possible) delay between the snapshot events.

• Snapshots do not contain system metadata (SMD). SMD lives under /opt/aerospike/smd and needs to be separately accounted for in a full cluster restore. Individual node restores are unaffected due to SMD being repopulated using other nodes.

• Snapshots are taken quickly, but that does not mean it is immediately available for use. It will take some time before a snapshot is ready to be used.

• Since snapshots are incremental, the first snapshot of any volume will also be the slowest to be ready.

Restore

If your backups were taken using EBS snapshots, you must deploy those snapshots into EBS volumes. After provisioning your replacement EBS volumes, configure the replacement instance exactly as the failed instance had been configured.

This means that EBS volumes are mounted at the same locations and the same number of ephemeral drives are provisioned.

Note

Shadow devices are always treated as authoritative upon cold-start. That means no matter what data may be on the local device, data will always be synchronized from the shadow device back into the local device.

Caution

Be careful when restoring anything short of a full cluster. Due to how Aerospike manages data, there is a risk of records coming back.

Benefits

• Quick
• Minimal impact
• Individual node based

Drawbacks

• More costly
• Additional scripting/coordination required
• Does not account for SMD

Instance failure

If you are using EBS for persistence and your instance fails, you can detach the EBS volume from the failed instance and then attach it to a replacement instance.

1. Remove/Terminate the failed instance, but keep the EBS volume.
2. Deploy a new Instance, with the same private IP as the old instance in the same zone.
3. Attach EBS volume from Step 1 to the replacement instance.
4. Restore the config to the replacement node and restart the node.

Once the node rejoins the cluster, migrations will occur and data will be rebalanced.