Configure Aerolab for AWS

This page guides you through configuring Aerolab to work with AWS and cloning the repository containing the bulk loading scripts.

Configure Aerolab for AWS

Aerolab needs access to your AWS account to create and manage resources. The configuration depends on how you set up your AWS credentials in the prerequisites.

1. Configure the Aerolab backend for AWS.

   Choose the appropriate command based on your AWS credential setup:

   AWS SSO

   Configure Aerolab to use your SSO profile (replace YOUR_PROFILE_NAME and YOUR_REGION):

   aerolab config backend --type aws --aws-profile YOUR_PROFILE_NAME --region YOUR_REGION

   For example, to use a profile named my-sso-profile in the us-west-2 region:

   aerolab config backend --type aws --aws-profile my-sso-profile --region us-west-2

   OK: Config.Backend.Type = aws
   Config.Backend.SshKeyPath = /Users/username/aerolab-keys/
   Config.Backend.AWSProfile = my-sso-profile
   Config.Backend.Region = us-west-2
   Config.Backend.AWSNoPublicIps = false
   Config.Backend.TmpDir = /Users/username/aerolab

   Example response

   Make sure your SSO session is fresh before running Aerolab commands. If you see “credentials expired” errors, refresh your session:

   aws sso login --profile YOUR_PROFILE_NAME

   Alternatively, you can set the AWS_PROFILE environment variable and omit the --aws-profile flag:

   export AWS_PROFILE=YOUR_PROFILE_NAME
   aerolab config backend --type aws --region YOUR_REGION

   Access keys

   If you configured AWS using aws configure with access keys (the default profile), simply set the region:

   aerolab config backend --type aws --region YOUR_REGION

   For example, to use the us-east-1 region:

   aerolab config backend --type aws --region us-east-1

   OK: Config.Backend.Type = aws
   Config.Backend.SshKeyPath = /Users/username/aerolab-keys/
   Config.Backend.Region = us-east-1
   Config.Backend.AWSNoPublicIps = false
   Config.Backend.TmpDir = /Users/username/aerolab

   Example response

   For detailed configuration instructions, see the Aerolab AWS setup guide.

Ensure your default zone in Aerolab matches the zone where you intend to create your cluster. Using different zones may cause unexpected behavior and connectivity issues between cluster nodes.

Clone the repository

1. Clone the Aerospike Graph repository.

   The repository contains the distributed bulk loading sample under misc/distributed-bulkload-example/AWS:

   git clone https://github.com/aerospike/aerospike-graph.git
   cd aerospike-graph/misc/distributed-bulkload-example/AWS

   Cloning into 'aerospike-graph'...
   remote: Enumerating objects: 2345, done.
   remote: Counting objects: 100% (2345/2345), done.
   remote: Compressing objects: 100% (892/892), done.
   remote: Total 2345 (delta 765), reused 1543 (delta 543)
   Receiving objects: 100% (2345/2345), 3.45 MiB | 6.78 MiB/s, done.
   Resolving deltas: 100% (765/765), done.

   Example response

Review the directory structure

The AWS bulk loading directory contains:

• set_variables.sh: Configuration script defining AWS resource names, regions, and paths
• create_cluster.sh: Script to create an Aerospike cluster using Aerolab
• bulkload.sh: Script to create an EMR cluster and submit the bulk load job
• bucket-files/: Directory containing the bulk loader configuration
  • config/bulk-loader.properties: Bulk loader configuration file

You will edit these files in subsequent steps to configure your bulk loading pipeline.

Checklist

• I’ve configured Aerolab for AWS.
• I’ve set my default AWS region.
• I’ve cloned the repository.