Back up data to Amazon S3

This page describes how to use asbackup to back up data directly to Amazon S3.

Overview

To back up files to Amazon S3, run a normal asbackup command and prefix the file and directory names with s3://BUCKET/KEY.

• BUCKET is the name of the target S3 bucket.
• KEY is the key of the object to download/prefix of files in the S3 “directory”.
• If using the default S3 endpoint, you must set --s3-region REGION to the region where the bucket is located.
• If using a non-default S3 endpoint, specify that endpoint with --s3-endpoint-override URL.

Files are uploaded in parts asynchronously. --s3-max-async-uploads controls the maximum number of simultaneous asynchronous upload parts across all threads. Each connection to S3 is throttled, so you may need to adjust this number to maximize throughput.

You can upload a maximum of 10,000 parts to S3. Each part must be between 5MB and 5GB, except for the last part which has no lower bound. When backing up to a directory, the value of --file-limit calculates what part size should be used, down to the minimum size of 5MB. The part size is either --file-limit divided by 10,000, or 5MB, whichever is larger.

When backing up to a file, run the --estimate flag before starting the backup. The result is used in the prior calculation to find the proper upload part size.

Back up to S3 example

Back up records to an Amazon S3 bucket with the following parameters:

• Namespace: test
• Target S3 bucket: test-bucket
• Region: us-west-1
• Directory: test-dir

Run the following command:

asbackup -n test -d s3://test-bucket/test-dir --s3-region us-west-1

Back up to S3 with an HTTP/HTTPS proxy

You can configure asbackup to route traffic to Amazon S3 through an HTTP/HTTPS proxy using one of two methods:

• Configuration file: Set the option globally or per-stanza in the Aerospike tools configuration file.
• Command-line flag: Pass the --s3-allow-system-proxy flag directly on the host command line.

Backup with the tools configuration file

To permanently configure the proxy settings, add the s3-allow-system-proxy parameter to the [asbackup] section of your tools configuration file:

[asbackup]
s3-allow-system-proxy = true

Caution

If HTTPS_PROXY or HTTP_PROXY includes credentials, avoid --s3-log-level Debug or Trace, which can write the proxy URL to stderr.

Backup with the command line flag

This flag is required because system proxy environment variables are ignored by default.

1. Set proxy environment variables on the host. For example:

   export HTTPS_PROXY=http://corp-proxy.internal:3128
   export NO_PROXY=.internal.corp,127.0.0.1,169.254.169.254

2. Confirm the variables are set in your current shell:

   echo "$HTTPS_PROXY"

3. Run asbackup with the proxy flag enabled:

   asbackup -n test -d s3://bucket/path --s3-region us-west-1 --s3-allow-system-proxy

4. Verify the backup completes without any S3 connection errors in the terminal output.

Required S3 permissions

To configure the required permissions for use with Amazon S3, include the elements listed in the following policy example and replace backup-bucket with the name of the S3 bucket you are using for the backup.

{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::backup-bucket"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::backup-bucket/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

S3 backup options

The following options are available for backup to Amazon S3:

Option	Default	Description
--s3-region REGION	-	Sets the S3 region of the bucket being uploaded to or downloaded from. Must be set if using the default S3 endpoint.
--s3-endpoint-override URL	-	Sets the S3 endpoint to use. Must point to an S3-compatible storage system.
--s3-profile PROFILE_NAME	default	Sets the S3 profile to use for credentials.
--s3-min-part-size SIZE IN MEGABYTES	-	An override for the minimum S3 part size to use for file uploads. By default, this size is calculated based on the expected backup file size found either with the value of --file-limit for backup-to-directory or from the backup estimate run before backup-to-file.
--s3-max-async-downloads N	32	Maximum number of simultaneous download requests from S3.
--s3-max-async-uploads N	16	Maximum number of simultaneous upload requests from S3.
--s3-connect-timeout MILLISECONDS	1000	AWS S3 client connection timeout in milliseconds. Equivalent to cli-connect-timeout in the AWS CLI, or connectTimeoutMS in the aws-sdk-cpp client configuration.
--s3-allow-system-proxy	Off	Allow the S3 client to honor system proxy environment variables when contacting S3. Precedence and URL syntax follow libcurl. With this flag off, proxy environment variables are ignored.
--s3-log-level LEVEL	Fatal	Log level of the AWS S3 C++ SDK. The possible levels are, from least to most granular: Off Fatal Error Warn Info Debug Trace