CONNECT

Aerospike Connect for Event Stream Processing

Seamlessly connect Aerospike to HTTP-based systems to build efficient streaming pipelines.

Connect for Event Stream Processing

Easily integrate with HTTP-based systems in the cloud or on-premises.

Stream Change Notification with Aerospike Connect for Event Stream Processing (ESP)

Event stream processing (ESP) systems commonly rely on Change Data Capture (CDC) data patterns which inform external systems when records are inserted, modified, or deleted in a database. These CDC patterns (in Aerospike CDC is implemented as “Change Notification” but unlike CDC, may include several changes) are published as they occur via the Aerospike Cross-Datacenter Replication (XDR) feature.

diagram: Aerospike Connect for Event Stream Processing data pipeline

Benefits

Why Aerospike Connect for ESP

Connect for ESP integrates Aerospike Database 5 with HTTP-based systems whether in the cloud or on-premises via CDC change notification.

  • Support for serverless design

    Supports serverless processing design patterns, where CDCs are streamed to a Lambda function in AWS or Google Cloud Functions .
  • Easy integration

    Easily integrate the Aerospike Real-time Data Platform with HTTP-based systems either in the cloud or on-premises.
  • Supports efficient data exchange between systems

    Supports JSON and binary formats such as Avro and MessagePack.
  • Simplifies operations

    Removes the overhead associated with operationalizing Kafka or Pulsar production clusters for use cases that do not warrant it.
FEATURES

Key features of Aerospike Connect for ESP

Connect for ESP converts CDC change notifications over XDR into HTTP requests and streams them to downstream consumers.

Supports HTTP-based systems

Converts CDC Change Notification over XDR into HTTP/1 or HTTP/2 to enable connection with broader HTTP-based systems.

Efficient data exchange between systems

Serializes the change notification payload into a text format such as JSON, or a binary format, such as Avro or MessagePack.

Change notifications for big data ecosystems

Triggers AWS Lambda via API Gateway or Application Load Balancer, or Google Functions to process CDC change notifications and make it available to the broader big data ecosystem in AWS or GCP.

Enable message ordering

Ships LUT (last-update-time) of the record to enable downstream applications to build their own custom logic to order messages.

Delivery guarantee

Extends XDR’s at-least-once delivery guarantee to ensure zero message loss.

Ease of deployment

Deploys seamlessly either in the cloud or on-premises.

Message filtering

Can be used in conjunction with XDR filter expressions to filter out records before transmission.

Learn more about Connect for ESP

Aerospike Connect for Event Stream Processing (ESP) integrates with other systems in the cloud or on-prem that can ingest CDC change notification as HTTP requests.

Read solution brief
Use Cases

Aerospike Connect for Event Stream Processing powers use cases across architectures

Aerospike Connect for ESP makes it easy for enterprises to enable use cases requiring streaming data changes (Change Notification) from the Aerospike database to other HTTP-based systems.

Search

Export shortened Aerospike Change Notification (CDC) messages to Elasticsearch using its Document REST API. This pattern can be extended to make Aerospike data available for analysis in Splunk using their HTTP Event Collector.

Backup

Back up Aerospike data to S3 (via AWS Lambda) or Google Cloud Storage (via Google Cloud Functions) for archival purposes.

AI/ML data processing

Stream Aerospike data to AWS SageMaker (via AWS Lambda) or Google AI (via Google Cloud Functions) to accelerate your cloud native AI/ML pipelines.

Compliance

Encrypt Aerospike data for compliance using AWS KMS via Lambda, before ingesting it into your data lake.

Web application development

Develop a web application in the language of your choice to ingest and process Aerospike Change Notification (CDC).

Autoscale streaming connectors

Autoscale Aerospike streaming connectors for Kafka, JMS, or Pulsar to enable you to build a cloud-native streaming pipeline.

Resources

Learn more about Connect for Event Stream Processing.

Learn more about the Aerospike Connect product line