In this blog post, we detail how to create an Amazon Managed Streaming for Apache Kafka (Amazon MSK) resource using AWS Identity and Access Management (AWS IAM) in roles and policies to authenticate user access. In the initial step, we establish an Aerospike Database cluster and insert sample messages into the database. Subsequently, we observe in real time how these messages are streamed to Amazon MSK using Aerospike's Kafka Source Connector.
Lab is fun
The Aerospike Quick Start is an automated deployment guide built by Aerospike and Amazon Web Services (AWS) Solutions Architects to get you started with Aerospike. It’s best suited for users who want to get started with Aerospike on the AWS Cloud using current best practices, high availability, and security-related features. More details about Aerospike Quick Start can be found here.
Serverless compute architectural pattern allows you to run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes. AWS Lambda and Google Cloud Functions in GCP are important services that are based on this pattern and can be extended to support event streaming pipelines. AWS Lambda is integrated with the majority of the services of the AWS big data ecosystem. Likewise for the integration of Google Cloud Functions with Google Cloud Platform. This presents an opportunity for you to integrate the Aerospike database with the broader cloud ecosystem, in order to build efficient cloud-native streaming pipelines.
How I select the optimal AWS instance type for running Aerospike
One of the most common conversations I have when providing consulting services for Aerospike customers running on AWS is about how to select the optimal AWS instance type.