--- title: "Back up data to Amazon S3" description: "Learn how to use the asbackup tool to back up Aerospike database data directly to Amazon S3 buckets." --- # Back up data to Amazon S3 > For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt) > > All documentation pages available in markdown. This page describes how to use `asbackup` to back up data directly to Amazon S3. ## Overview To back up files to Amazon S3, run a normal `asbackup` command and prefix the file and directory names with `s3://BUCKET/KEY`. - `BUCKET` is the name of the target S3 bucket. - `KEY` is the key of the object to download/prefix of files in the S3 “directory”. - If using the default S3 endpoint, you must set `--s3-region REGION` to the region where the bucket is located. - If using a non-default S3 endpoint, specify that endpoint with `--s3-endpoint-override URL`. Files are uploaded in parts asynchronously. `--s3-max-async-uploads` controls the maximum number of simultaneous asynchronous upload parts across all threads. Each connection to S3 is throttled, so you may need to adjust this number to maximize throughput. You can upload a maximum of 10,000 parts to S3. Each part must be between 5MB and 5GB, except for the last part which has no lower bound. When backing up to a directory, the value of `--file-limit` calculates what part size should be used, down to the minimum size of 5MB. The part size is either `--file-limit` divided by 10,000, or 5MB, whichever is larger. When backing up to a file, [run the `--estimate` flag before starting the backup](https://aerospike.com/docs/database/tools/backup-and-restore/asbackup/resources#estimate-disk-space-for-a-backup). The result is used in the prior calculation to find the proper upload part size. ## Back up to S3 example Back up records to an Amazon S3 bucket with the following parameters: - Namespace: `test` - Target S3 bucket: `test-bucket` - Region: `us-west-1` - Directory: `test-dir` Run the following command: Terminal window ```shell asbackup -n test -d s3://test-bucket/test-dir --s3-region us-west-1 ``` ## Back up to S3 with an HTTP/HTTPS proxy You can configure `asbackup` to route traffic to Amazon S3 through an HTTP/HTTPS proxy using one of two methods: - Configuration file: Set the option globally or per-section in the [Aerospike tools configuration file](https://aerospike.com/docs/database/tools/conffile). - Command-line flag: Pass the `--s3-allow-system-proxy` flag directly on the host command line. ### Backup with the tools configuration file To permanently configure the proxy settings, add the s`3-allow-system-proxy` parameter to the \[asbackup\] section of your tools configuration file: ```toml [asbackup] s3-allow-system-proxy = true ``` ::: caution If `HTTPS_PROXY` or `HTTP_PROXY` includes credentials, avoid `--s3-log-level` `Debug` or `Trace`, which can write the proxy URL to stderr. ::: ### Backup with the command line flag This flag is required because system proxy environment variables are ignored by default. 1. Set proxy environment variables on the host. For example: Terminal window ```shell export HTTPS_PROXY=http://corp-proxy.internal:3128 export NO_PROXY=.internal.corp,127.0.0.1,169.254.169.254 ``` 2. Confirm the variables are set in your current shell: Terminal window ```shell echo "$HTTPS_PROXY" ``` 3. Run `asbackup` with the proxy flag enabled: Terminal window ```shell asbackup -n test -d s3://bucket/path --s3-region us-west-1 --s3-allow-system-proxy ``` 4. Verify the backup completes without any S3 connection errors in the terminal output. ## Required S3 permissions To configure the required permissions for use with Amazon S3, include the elements listed in the following policy example and replace `backup-bucket` with the name of the S3 bucket you are using for the backup. ```json { "Statement": [ { "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads", "s3:ListBucketVersions" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::backup-bucket" ] }, { "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::backup-bucket/*" ] } ], "Version": "2012-10-17" } ``` ## S3 backup options The following options are available for backup to Amazon S3: | Option | Default | Description | | --- | --- | --- | | `--s3-region REGION` | \- | Sets the S3 region of the bucket being uploaded to or downloaded from. Must be set if using the default S3 endpoint. | | `--s3-endpoint-override URL` | \- | Sets the S3 endpoint to use. Must point to an S3-compatible storage system. | | `--s3-profile PROFILE_NAME` | default | Sets the S3 profile to use for credentials. | | `--s3-min-part-size SIZE IN MEGABYTES` | \- | An override for the minimum S3 part size to use for file uploads. By default, this size is calculated based on the expected backup file size found either with the value of `--file-limit` for backup-to-directory or from the backup estimate run before backup-to-file. | | `--s3-max-async-downloads N` | 32 | Maximum number of simultaneous download requests from S3. | | `--s3-max-async-uploads N` | 16 | Maximum number of simultaneous upload requests from S3. | | `--s3-connect-timeout MILLISECONDS` | 1000 | AWS S3 client connection timeout in milliseconds. Equivalent to cli-connect-timeout in the AWS CLI, or connectTimeoutMS in the aws-sdk-cpp client configuration. | | `--s3-allow-system-proxy` | Off | Allow the S3 client to honor system proxy environment variables when contacting S3. Precedence and URL syntax follow libcurl. With this flag off, proxy environment variables are ignored. | | `--s3-log-level LEVEL` | Fatal | Log level of the AWS S3 C++ SDK. The possible levels are, from least to most granular: - Off - Fatal - Error - Warn - Info - Debug - Trace |