Create Aerospike cluster

This page guides you through setting up networking, deploying an Aerospike cluster, and configuring access for EMR.

Set up networking

Many enterprise AWS accounts don’t have a default VPC, so this tutorial explicitly configures networking. You need a public subnet (with internet access) for the Aerospike cluster.

Use existing public subnet

If you already have a public subnet in your VPC, use it.

1. Find a public subnet in your VPC.

   List your subnets:

   aws ec2 describe-subnets --query 'Subnets[*].[SubnetId,VpcId,MapPublicIpOnLaunch,Tags[?Key==`Name`].Value|[0]]' --output table

   Look for a subnet where MapPublicIpOnLaunch is True, or one with “public” in its name. Set your subnet ID:

   SUBNET_ID=YOUR_SUBNET_ID
   echo "SUBNET_ID=$SUBNET_ID"

2. Get the VPC ID.

   VPC_ID=$(aws ec2 describe-subnets --subnet-ids $SUBNET_ID --query 'Subnets[0].VpcId' --output text)
   echo "VPC_ID=$VPC_ID"

Create new public subnet

If you only have private subnets, create a public subnet with internet access.

1. Get your VPC ID.

   List your VPCs and choose one:

   aws ec2 describe-vpcs --query 'Vpcs[*].[VpcId,Tags[?Key==`Name`].Value|[0]]' --output table

   Set your VPC ID:

   VPC_ID=YOUR_VPC_ID
   echo "VPC_ID=$VPC_ID"

2. Find a valid CIDR for your subnet.

   Check your VPC’s CIDR range and existing subnets:

   echo "VPC CIDR:"
   aws ec2 describe-vpcs --vpc-ids $VPC_ID --query 'Vpcs[0].CidrBlock' --output text
   echo "Existing subnets:"
   aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPC_ID" --query 'Subnets[*].CidrBlock' --output text

   Your subnet CIDR must start with the same numbers as your VPC CIDR and not overlap with existing subnets. Pick an unused number for the third section:

   Your VPC CIDR	Example subnet CIDR
   10.0.x.x	10.0.200.0/24
   172.31.x.x	172.31.200.0/24

3. Create a public subnet.

   Replace YOUR_CIDR with your chosen CIDR:

   SUBNET_ID=$(aws ec2 create-subnet \
     --vpc-id $VPC_ID \
     --cidr-block YOUR_CIDR \
     --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=aerospike-graph-public}]' \
     --query 'Subnet.SubnetId' --output text)
   echo "SUBNET_ID=$SUBNET_ID"

4. Enable auto-assign public IPs.

   aws ec2 modify-subnet-attribute --subnet-id $SUBNET_ID --map-public-ip-on-launch

5. Create an internet gateway (if your VPC doesn’t have one).

   IGW_ID=$(aws ec2 create-internet-gateway \
     --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=aerospike-graph-igw}]' \
     --query 'InternetGateway.InternetGatewayId' --output text)
   aws ec2 attach-internet-gateway --internet-gateway-id $IGW_ID --vpc-id $VPC_ID
   echo "IGW_ID=$IGW_ID"

   If you get “Resource already exists”, your VPC already has an IGW. Get its ID:

   IGW_ID=$(aws ec2 describe-internet-gateways --filters "Name=attachment.vpc-id,Values=$VPC_ID" --query 'InternetGateways[0].InternetGatewayId' --output text)
   echo "IGW_ID=$IGW_ID"

6. Create a route table with internet access.

   RTB_ID=$(aws ec2 create-route-table \
     --vpc-id $VPC_ID \
     --tag-specifications 'ResourceType=route-table,Tags=[{Key=Name,Value=aerospike-graph-rtb}]' \
     --query 'RouteTable.RouteTableId' --output text)
   aws ec2 create-route --route-table-id $RTB_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW_ID
   aws ec2 associate-route-table --route-table-id $RTB_ID --subnet-id $SUBNET_ID
   echo "RTB_ID=$RTB_ID"

Create a security group

1. Create a security group.

   SG_ID=$(aws ec2 create-security-group \
     --group-name aerospike-graph-sg \
     --description "Security group for Aerospike Graph tutorial" \
     --vpc-id $VPC_ID \
     --query 'GroupId' --output text)
   echo "SG_ID=$SG_ID"

2. Add inbound rules.

   # SSH access from your IP
   MY_IP=$(curl -s https://checkip.amazonaws.com)
   aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port 22 --cidr ${MY_IP}/32
   
   # Aerospike port from within the security group
   aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port 3000 --source-group $SG_ID

3. Save the IDs for the next steps.

   echo "Save these values:"
   echo "  SUBNET_ID=$SUBNET_ID"
   echo "  SG_ID=$SG_ID"

Update the scripts

The create_cluster.sh script sources set_variables.sh, so you only need to set your values in one place.

1. Update set_variables.sh with your networking values.

   Open set_variables.sh in your editor and update the network settings at the end:

   SUBNET_ID="YOUR_SUBNET_ID"
   SECURITY_GROUP="YOUR_SG_ID"

   Replace with your actual values from the previous step.

2. Update create_cluster.sh to use the variables.

   Open create_cluster.sh and replace the aerolab cluster create command with:

   aerolab cluster create --name "$name" \
       --count "$instance_count" \
       --instance-type "$aws_instance_type" \
       --aerospike-version "$aerospike_version" \
       --owner "$username" \
       --subnet-id "$SUBNET_ID" \
       --secgroup-id "$SECURITY_GROUP" \
       --start n

   Note

   This uses the $SUBNET_ID and $SECURITY_GROUP variables from set_variables.sh to create the Aerospike cluster.

3. Save both files.

Create the Aerolab cluster

1. Make the scripts executable.

   chmod +x create_cluster.sh bulkload.sh

2. Run the cluster creation script.

   ./create_cluster.sh

   Creating Cluster
   Using subnet ID subnet-XXXXX
   Using security group ID sg-XXXXX
   ...
   Connection succeeded, continuing installation...
   Instance UP, continuing
   CLUSTER EXPIRES: Wednesday, 10-Dec-25 20:11:08 PST (in: 29h56m47s)
   Done
   
   Starting Cluster
   Done

   Example response

   Note

   The cluster has an expiration time. To extend it, use aerolab cluster add expiry.

Configure the bulk loader properties

The bulk loader needs to know how to connect to your Aerospike cluster. You will update the properties file with your cluster’s private IP address.

1. Get the cluster private IP address.

   Use Aerolab to list your clusters and retrieve the private IP:

   aerolab cluster list

   ClusterName       NodeNo  ExpiresIn  State    PublicIP       PrivateIP     Owner       AsdVer
   my-graph-cluster  1       29h41m0s   running  44.243.177.28  172.31.0.110  myusername  8.0.0.7

   Example response

   Note the PrivateIP value (for example, 172.31.0.110).

2. Open bucket-files/config/bulk-loader.properties in your editor.

3. Update the Aerospike connection settings.

   Locate the core AGS connection section and update these properties:

   aerospike.client.host=10.0.1.123
   aerospike.client.port=3000
   aerospike.client.namespace=test

   Replace 10.0.1.123 with your cluster’s private IP address.

4. Update the data source paths.

   Locate the bulk loader source paths section and update the bucket name:

   aerospike.graphloader.vertices=s3://YOUR_BUCKET_NAME/vertices
   aerospike.graphloader.edges=s3://YOUR_BUCKET_NAME/edges
   aerospike.graphloader.temp-directory=s3://YOUR_BUCKET_NAME/temp

   Replace YOUR_BUCKET_NAME with your actual S3 bucket name from the previous section.

5. Save the file.

Note

The bulk loader uses the private IP address to communicate with the Aerospike cluster. This requires the EMR cluster to be in the same VPC and subnet, which is why we configure the subnet and security group settings.

Checklist

• I’ve set up networking (subnet and security group).
• I’ve created an Aerospike cluster on AWS.
• I’ve configured the bulk loader properties file.