Modern data architectures don’t work in isolation. They are made up of multiple technologies each serving a different purpose in the data ecosystem. Apache Kafka® fits naturally as a messaging layer for real-time streaming data pipelines to reliably get data between many independent systems or applications.
One of Aerospike’s superpowers is fast data access, so it is no surprise that it is often part of a larger event-driven architecture consisting of one or more data stores, messaging systems and information processing systems,
With this in mind, in 2019 we released Aerospike Connect for Kafka to exchange data bi-directionally between Aerospike and transactional systems at the edge and core. Aerospike Connect for Kafka makes it easy to integrate an Aerospike database into a larger system’s architecture. Aerospike Connect for Kafka includes two connectors:
- Inbound for streaming data from Kafka into an Aerospike database.
The inbound Aerospike Kafka Sink Connector streams data from Kafka and persists it to an Aerospike database. Each Kafka record is converted to an Aerospike record and persisted to an Aerospike cluster. In standalone mode all work is performed in a single process. Distributed mode handles automatic balancing of work, supports automatic scaling, and offers fault tolerance both in the active tasks and for configuration and offset commit data.
- Outbound for updates from an Aerospike database to Kafka.
The outbound connector receives data from the Aerospike server via the Change Notification Framework. Each message contains either an updated/inserted record or a record deletion notification.
Aerospike supports JSON, Flat JSON, Avro, Kafka Avro and MessagePack message formats. A REST Client can be utilized as an easy and standard interface for the Aerospike database.
In partnership with Confluent, Aerospike is excited to sponsor Kafka Summit 2020, which kicks off next week with an action-packed virtual program led by prominent experts to learn more about Apache Kafka® — a highly flexible streaming platform for real time digital businesses.
On Monday August 24th, 11:30AM PT, we invite you to join us for a deep dive into Distributed Data Storage & Streaming for Real-time Decisioning using Kafka, Spark and Aerospike, delivered by Kiran Matty, Director Product Management at Aerospike. In this talk Kiran will cover how Aerospike Connect for Kafka and Spark is used for real time analytics including reviewing a real world trading application and an HPE COVID-19 project architecture.
Kiran is the Director of Product Management at Aerospike, where he manages ecosystem products that relate to AI/ML, Streaming, and Aerospike DB access. His experience spans big data and ML infrastructure, data security, and analytics. Prior to Aerospike, he was a product manager at Visa, Hortonworks, HPE, and Cisco. His interests include large scale distributed systems and AI/ML.
If you’re ready to take your Kafka skills to the next level join us at Kafka Summit, August 24-25,
