Restore data from Amazon S3

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

This page describes how to use asrestore to restore backups from Amazon S3.

Overview

To restore from Amazon S3, use the format s3://BUCKET/KEY for files or directories.

Endpoints: Specify --s3-region REGION for default S3 endpoints, or use --s3-endpoint-override ENDPOINT_URL for custom endpoints.
Performance: Files download in 5MB chunks. Limit concurrent async requests with --s3-max-async-downloads.

Example

Restore from test-bucket in region us-west-1 from the test-dir directory:

asrestore -d s3://test-bucket/test-dir --s3-region us-west-1

Restore using an HTTP/HTTPS Proxy

By default, asrestore ignores system proxy environment variables. To route S3 traffic through a proxy, you must explicitly enable support using either the tools configuration file or a command-line flag.

Tools configuration file

To enable proxy support globally or per-stanza, add the parameter to your tools configuration file:

[asrestore]
s3-allow-system-proxy = true

Command line flag

Use the --s3-allow-system-proxy flag when the host accesses S3 through an HTTP/HTTPS proxy configured with environment variables.

This flag is required because system proxy environment variables are ignored by default.

Set your environment variables:

export HTTPS_PROXY=http://corp-proxy.internal:3128
export NO_PROXY=.internal.corp,127.0.0.1,169.254.169.254

Confirm the variables are set in your current shell:
Terminal window
```
echo "$HTTPS_PROXY"
```

Run asrestore with the proxy flag:

asrestore -d s3://bucket/path --s3-region us-west-1 --s3-allow-system-proxy

Verify the restore completes without any S3 connection errors in the terminal output.

You can also set the option in the Aerospike tools configuration file:

Required S3 permissions for `asrestore`

asrestore requires certain permissions to restore from Amazon S3. In the following IAM JSON policy, include the elements listed and replace backup-bucket with the name of the S3 bucket you are using for the restore.

{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::backup-bucket"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::backup-bucket/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

S3 restore options

The following options are available for restore from Amazon S3:

Option	Default	Description
`--s3-region REGION`	-	Sets the S3 region of the bucket being uploaded to/downloaded from. Must be set if using the default S3 endpoint.
`--s3-endpoint-override URL`	-	Sets the S3 endpoint to use. Must point to an S3-compatible storage system.
`--s3-profile PROFILE_NAME`	default	Sets the S3 profile to use for credentials.
`--s3-max-async-downloads N`	32	The maximum number of simultaneous download requests from S3.
`--s3-connect-timeout MILLISECONDS`	1000	The AWS S3 client’s connection timeout in milliseconds. Equivalent to `cli-connect-timeout` in the AWS CLI, or `connectTimeoutMS` in the `aws-sdk-cpp` client configuration.
`--s3-allow-system-proxy`	Off	Allow the S3 client to honor system proxy environment variables when contacting S3. Precedence and URL syntax follow libcurl. With this flag off, proxy environment variables are ignored.
`--s3-log-level LEVEL`	Fatal	The log level of the AWS S3 C++ SDK. The possible levels are, from least to most granular. Off Fatal Error Warn Info Debug Trace