Skip to main content
Aerospike Vector is now live with LangChain integrationTell me more
Loading

Streaming from Aerospike to Google Cloud Pub/Sub

caution

This release of the Aerospike Connect for Google Cloud Pub/Sub is a Beta release and is not supported by Aerospike for production use. If you would like to participate in the beta program, contact Aerospike support. If you would like to engage with Aerospike in a Proof-of-Concept exercise, contact Aerospike sales at sales@aerospike.com.

Aerospike Connect for Google Cloud Pub/Sub makes Aerospike Database a part of your Google Pub/Sub-based, low-latency, streaming pipeline in a highly scalable and easily configurable manner. It allows you to use Aerospike Database at the edge for high-velocity ingestion of data that is subsequently ingested using Google Pub/Sub into Google Cloud Platform's big-data ecosystem for stream analytics.

  • Supports Pub/Sub Lite, in addition to Pub/Sub, to offer you a low-cost option
  • Provides options to route a set, namespace, or bin value to an existing Pub/Sub topic, and options to enable use cases that need filtering of records at the source.
  • Secures your data by supporting both TLS and authentication by means of your Google service account key.
  • Supports JSON, Flat JSON, and binary message formats such as Avro and MessagePack for higher compression.

System topology​

An instance of Aerospike Connect for Google Pub/Sub is called a Pub/Sub connector. You install, configure, and run Pub/Sub connectors on virtual machines (VM) in Google Cloud Platform, providing them with the credentials to connect to your Aerospike databases and Google Pub/Sub.

This diagram shows a connector publishing changes from an Aerospike database to Pub/Sub or Pub/Sub Lite

  1. Client applications make changes in your Aerospike database.
  2. The database sends notifications of changes to the connector by using Aerospike's cross-datacenter replication (XDR).
  3. The connector creates messages from the change notifications. The messages can be in one of several supported formats. The connector then publishes the messages to Pub/Sub. You can use Pub/Sub Lite, instead of Pub/Sub.
  4. Pub/Sub pushes the messages to subscribing applications.
  5. Pub/Sub acknowledges the messages that it receives.

Supported database events​

The connector receives data from an Aerospike database by means of a change-notification system which is based on Aerospike's Cross-Datacenter Replication (XDR). Each message contains a notification of one of these two types of events:

A record was inserted or updated
Each notification contains the full database record, including all, or a subset, of the record's bins.

Operations on CDT List or Map types are not supported. For example, if a list element is added or a map entry is removed, actions which cause a partial bin update, Aerospike retransmits the full corresponding record bin.

A record was deleted
Each notification contains the corresponding record digest, but not the corresponding record bin.

note

Note about both types of supported database events: There might be instances where records are re-ordered on the network or across connector instances, in which case a message for an older version of a record could be delivered after a message for a newer version of the same record.

Prerequisites​

  • Create a Linux virtual machine (VM), whether in Google Cloud Platform or another cloud platform, for hosting a connector instance. If you want to run more than one connector instance for faster throughput, create a Linux (VM) for each. Connector instances make full use of the system resources available to them, so it is suggested not to run more than one connector instance in a VM.
    Following are the Linux platforms and versions that Aerospike Connect for Google Cloud Pub/Sub supports:
    • RedHat Enterprise Linux 7 and later
    • CentOS 7 and later
    • Ubuntu 20.04, and later
    • Debian 8 (removed in Database 6.0), Debian 9 (removed in Database 6.3), Debian 10 (only on x86_64 hardware in Database 6.3), Debian 11 (introduced in Database 6.0), Debian 12 (introduced in Database 6.4)
  • Verify that Java 8 or later is installed on each VM. If you want the connector to authenticate to your Aerospike database through TLS 1.3, install Java 11 or later. Both Oracle and OpenJDK Java Runtime Environments are supported.
  • Verify that your digitally signed feature-key-file for your Aerospike database sets asdb-change-notification to true.
  • Verify that you have created topics and subscriptions in Pub/Sub or Pub/Sub Lite for consuming changes to data in your Aerospike database.
  • Set up Aerospike's change-notification system to send change notifications to the connector or connectors that you plan to run.

Procedure​

Follow these steps for each connector instance that you plan to run:

Log in to the VM that you created for the connector.

Download the installation package for the version of Linux that the VM is running. If you would like to participate in the beta program, contact Aerospike support. If you would like to engage with Aerospike in a Proof-of-Concept exercise, contact Aerospike sales at sales@aerospike.com.

Install the connector by running one of these commands:

  • On RedHat Enterprise Linux or CentOS: sudo rpm -i aerospike-pubsub-outbound-VERSION_NUMBER.noarch.rpm
  • On Debian and Ubuntu: sudo dpkg -i aerospike-pubsub-outbound-VERSION_NUMBER.deb

For the connector to authenticate to Pub/Sub or Pub/Sub Lite, follow these steps:

  • In Cloud Console or Cloud Shell in Google Cloud Platform, create a JSON file that contains your service account key and save that file on the VM.
  • In the VM, create the text file /etc/aerospike-pubsub-outbound/aerospike-pubsub-outbound.conf. This file has the same name, and is in the same directory, as the file that you create for configuring the properties of the connector. However, their extensions are different. The extension for this file is .conf, while the extension for the configuration file is .yml.
  • In that file, add the following line:
    GOOGLE_APPLICATION_CREDENTIALS="[path_to_the_JSON_file]"

Configure the connector.

Create the file /etc/aerospike-pubsub-outbound/aerospike-pubsub-outbound.yml and open it for editing.

  • If you are using Pub/Sub, place the following lines at the top of the file, where [projectID] is the unique identifier for your Pub/Sub project:
    publisher:
      type: pubsub
      project-id: [projectID&]
  • If you are using Pub/Sub Lite, place the following lines at the top of the file:
    type: pubsub-lite
    project-number: [project-number]
    cloud-zone: [cloud-zone]

where

Enable and start the connector by running this pair of commands:

sudo systemctl enable aerospike-pubsub-outbound
sudo systemctl start aerospike-pubsub-outbound

What to do next​

If you modify a configuration file for a connector, you must restart the connector after you finish making your changes in the file. Connectors read their configuration files only when they are started.

Last updated on