Integration with AWS Kinesis

This page demonstrates how to integrate the Aerospike ESP Outbound Connector with AWS Kinesis Data Streams using the API Gateway, enabling reliable, real-time streaming of Aerospike data to AWS for downstream processing.

The integration uses three components, each with a defined role:

Aerospike ESP Outbound Connector: Pushes change notifications or data out of Aerospike.
AWS API Gateway: Acts as a secure interface between Aerospike and AWS services.
AWS Kinesis Data Streams: Serves as the destination for real-time Aerospike data.

Prerequisites

An AWS account with appropriate permissions
Access to AWS Kinesis, API Gateway, and IAM services
An Aerospike database with XDR and change notifications configured
The ESP Outbound Connector installed and configured

AWS setup

The setup process takes place in the following stages:

Create a Kinesis data stream
Create an API in API Gateway
Configure a mapping template
Generate X-API-Key
Deploy the API

Stage 1 - Create a Kinesis data stream

Go to AWS Console → Kinesis.
Click Create Data Stream.
Choose On-Demand capacity mode.
Provide a stream name such as my_kinesis_stream.
Click Create data stream.

For more information, see the AWS Kinesis Documentation.

Stage 2 - Create an API in API Gateway

Go to AWS Console → API Gateway.
Select Create API → Choose REST API.
Define a new resource and methods (GET/POST)
Configure the POST method to integrate with AWS Kinesis PutRecord:
- Integration type: AWS Service
- Action Name: PutRecord
- AWS Region: your region (e.g., ap-south-1)
- IAM Role: Provide role with access to Kinesis

For more information, see the AWS API Gateway Documentation.

Stage 3 - Configure mapping template

Configure the mapping template under POST → Integration Request. This template enables API Gateway to forward data to the correct Kinesis stream:

{
  "StreamName": "$input.params('streamname')",
  "PartitionKey": "$input.params('aerospike-digest')",
  "Data": "$util.base64Encode($input.body)"
}

Stage 4 - Generate X-API-Key

Include this key in ESP connector headers for authentication.

In API Gateway, go to API Keys → Create a key.
Associate it with a Usage Plan.
Link this key and plan to your API.

Stage 5 - Deploy the API

In the API Gateway console, select your API and click Deploy
Choose a stage name (e.g., test, prod)
Note the Invoke URL - this will be used in your ESP configuration

ESP configuration

The following examples configure the ESP outbound connector to forward data from the Aerospike database to the API Gateway endpoint.

Create or modify the `aerospike-esp-outbound.yml` file:

destinations:
  aws-kinesis:
    urls:
      - https://<your-api-id>.execute-api.<region>.amazonaws.com/<stage>/<resource>
    headers:
      send-digest-headers: true
      additional-headers:
        Content-Type: application/json
        streamname: my_kinesis_stream
        x-api-key: <your-api-key>
    call-timeout: 10000
    connect-timeout: 2000
    max-connections-per-endpoint: 10

Configuration parameters

url: The API Gateway endpoint URL (Invoke URL from deployment)
Content-Type: Specifies the data format (e.g., application/json)
streamname: Name of the Kinesis stream to route data into
x-api-key: Authentication key generated in AWS API Gateway
send-digest-headers: When set to true, includes the aerospike-digest header used as the partition key

Test using cURL

The following example tests your endpoint using a cURL command:

curl -H "streamname:my_kinesis_stream" \
    -H "aerospike-digest:partition-1" \
    -H "x-api-key:your_api_key_here" \
    -H "Content-Type:application/json" \
    -X POST https://<api-id>.execute-api.<region>.amazonaws.com/<stage>/<resource> \
    -d '{"data":"value"}'

View Kinesis data

To verify if data has been received:

Navigate to your Kinesis stream → Data Viewer.
Enter the Shard ID and Sequence Number (from test log).
Click Get records.

Complete example configuration

The following is a complete example of aerospike-esp-outbound.yml configured for AWS Kinesis:

service:
  port:
    - 8901
  manage:
    port: 8902

logging:
  file: /var/log/aerospike-esp-outbound/aerospike-esp-outbound.log

destinations:
  aws-kinesis:
    urls:
      - https://abc123xyz.execute-api.ap-south-1.amazonaws.com/prod/kinesis
    protocol: HTTP_1_1
    headers:
      send-digest-headers: true
      additional-headers:
        Content-Type: application/json
        streamname: my_kinesis_stream
        x-api-key: your-api-key-here
    connection-ttl: 15000
    call-timeout: 10000
    connect-timeout: 2000
    max-connections-per-endpoint: 10
    health-check:
      enabled: true
      interval-seconds: 30
      call-timeout: 5000

routing:
  mode: static
  destination:
    - aws-kinesis

format:
  mode: json
  metadata-key: metadata

Additional considerations

Security: Ensure your API Gateway endpoint uses HTTPS and proper authentication.
Error handling: Monitor API Gateway logs and Kinesis metrics for any delivery issues.
Scaling: Consider using On-Demand capacity mode for Kinesis to handle variable workloads.
Partitioning: The aerospike-digest header is used as the partition key, ensuring records with the same digest go to the same shard.