Install Event Stream Processing outbound connector

Prerequisites

Aerospike Database change notification configuration

Note

Streaming from Aerospike requires you to configure your Aerospike database to send change notifications. For the steps, see Configuring Change Notification for Use with Connectors.

Procedures

You can deploy the ESP outbound connector in three environments:

• Linux
• Kubernetes
• Docker

Installation prerequisites on Linux

Supported operating systems

The connector is supported on the following operating systems:

• RHEL/CentOS (8 and 9)
• Ubuntu (20.04, 22.04 and 24.04 LTS)
• Debian (11, 12 and 13)
• Amazon Linux 2023

Java Runtime Environment

This connector is a Java web-application that requires Java 11 or later. Both Oracle and OpenJDK Java Runtime Environments are supported.

Java 17 is supported in ESP version 1.1.0 and later.

Installation

The following steps apply to installing both the Debian (*.deb) and RHEL (*.rpm) packages. Significant differences in the steps are noted.

Install Java 11 or later version

The Aerospike Connect for ESP Outbound package does not include a Java runtime environment. Most supported platforms provide official JDK 11 packages. For platforms that do not, you can get Oracle’s JDK 11 builds from jdk.java.net/11/.

RHEL and CentOS

sudo yum install java-11-openjdk

Debian and Ubuntu

sudo apt-get install openjdk-11-jre

Amazon Linux

sudo dnf install java-11-openjdk

Install connector package

Download the connector package for your platform from Aerospike Enterprise downloads and install it. The connector is supported on both amd64 and arm64 architectures.

Set the protocol based on the change notification source. See protocol for details.

RHEL and CentOS

sudo rpm -i aerospike-esp-outbound-2.4.14-1.noarch.rpm

Debian and Ubuntu

sudo dpkg -i aerospike-esp-outbound-2.4.14.all.deb

Amazon Linux

sudo rpm -i aerospike-esp-outbound-2.4.14-1.noarch.rpm

Verify/update connector configuration

The configuration file for the connector is in /etc/aerospike-esp-outbound/. For details on how to configure the ESP outbound connector, see the Configuration Guide.

Start the connector

The connector package includes a Systemd service definition. The installation procedure creates an aerospike-esp-outbound service.

Enable the connector to start on system startup or reboot:

$ sudo systemctl enable aerospike-esp-outbound

To start the connector service run:

$ sudo systemctl start aerospike-esp-outbound

Deploy in Kubernetes

Configure the ESP outbound connector and apply the configuration with a Kubernetes ConfigMap. In this task, you create a folder, create and add files to the folder, and point to this folder when you create a ConfigMap.

1. Create a folder. The path and name of the directory do not matter.

   Example

   mkdir aerospike-esp-outbound

2. Copy any TLS truststore and keystore files into this directory.

   If you plan to use TLS to connect to the destinations of the messages sent by the ESP outbound connector (load balancers or connector instances), copy truststore and keystore files for those connections into this folder, so that the Kubernetes container has access to them.

3. Create the configuration file aerospike-esp-outbound.yml. You can create this file anywhere in your system. When you run a Docker container in the next step, it mounts the file into the container’s file system.

4. Create the ConfigMap.

   kubectl -n <k8s namespace> create configmap aerospike-esp-outbound-config \
     --from-file=<path-and-name-of-directory> --dry-run=client -o yaml | kubectl apply -f -

   • <k8s namespace>: Name of the Kubernetes namespace in which to create the ConfigMap.
   • <path-and-name-of-directory>: Path and name of the directory that contains the configuration file.

   If the command runs successfully, it displays a message that it created the ConfigMap:

   W0512 09:21:00.039900   12016 helpers.go:557] --dry-run is deprecated and can be replaced with --dry-run=client.
   configmap/aerospike-esp-outbound-config created

5. Deploy the ESP outbound connector in a container.

   These steps explain how to run the ESP outbound connector in its own pod. However, you can run the ESP outbound connector as a sidecar to the main container of an Aerospike database being run by the Aerospike Kubernetes Operator.

6. Create a Kubernetes deployment file.

   Here is a sample Kubernetes deployment file named aerospike-esp-outbound-deployment.yml. It deploys of two instances of ESP outbound connector as separate pods. You can run more than one instance of the ESP outbound connector in case one fails, the other continues receiving CN messages from your Aerospike cluster.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: aerospike-esp-outbound
     namespace: aerospike
   
   spec:
     replicas: 2
   
     selector:
       matchLabels:
         app: aerospike-esp-outbound
   
     template:
       metadata:
         labels:
           app: aerospike-esp-outbound
   
       spec:
         volumes:
           - name: aerospike-esp-outbound-config
             configMap:
               name: aerospike-esp-outbound-config
   
         containers:
           - name: aerospike-esp-outbound
             image: aerospike/aerospike-esp-outbound:latest
             volumeMounts:
               - mountPath: /etc/aerospike-esp-outbound
                 name: aerospike-esp-outbound-config

7. Use the following command to deploy the ESP outbound connector:

   kubectl apply -f aerospike-esp-outbound-deployment.yml

   Sample output:

   deployment.apps/aerospike-esp-outbound created

Install ESP outbound connector on Kubernetes with helm-charts

If your deployment situation allows the use of Helm charts, you can install the ESP outbound connector on a Kubernetes deployment using Helm.

1. Add the Aerospike Helm repository.

   helm repo add aerospike https://aerospike.github.io/helm-charts --force-update
   helm repo update

2. Create the file values.yaml with your ESP outbound connector configuration. The following example includes a basic ESP outbound connector configuration as part of the Helm installation.

   values.yaml

   replicaCount: 3
   
   image:
     repository: aerospike/aerospike-esp-outbound
     tag: "latest"
   
   connectorConfig:
     service:
       address: 0.0.0.0
       port: 8901
       manage:
         address: 0.0.0.0
         port: 8902
       io-threads: 100
       worker-threads: 100
       max-concurrent-requests: 100000
   
     routing:
       mode: static-multi-destination
       destinations:
         - dc1
   
     record-ordering:
       enable: false
       lut-cache-ttl-seconds: 30
   
     destinations:
       dc1:
         urls:
           - https://connector.aerospike.com:8443
         tls:
           trust-store:
             store-file: secrets/tls-certs/svc.cluster.local.truststore.jks
             store-password-file: secrets/tls-certs/storepass
         health-check:
           call-timeout: 5100
         connect-timeout: 5000
         connection-ttl: 300000
         max-connections-per-endpoint: 20
         call-timeout: 5100
         protocol: HTTP_2
         http2-max-concurrent-streams: 100
   
     logging:
       enable-console-logging: true
   
   connectorSecrets: []
   
   serviceAccount:
     create: true
     name: aerospike-esp-outbound

3. Install the ESP outbound connector by running the following Helm command.

   helm install aerospike-esp-outbound aerospike/aerospike-esp-outbound --values values.yaml --atomic --wait --debug --create-namespace --namespace aerospike

Deploy in Docker

Aerospike provides Docker images for the ESP outbound connector on Docker Hub.

1. Download and install Docker from here.

2. Pull the image for the ESP outbound connector from Docker Hub.

   docker pull container.aerospike.com/aerospike/aerospike-esp-outbound:latest

3. Create the configuration file aerospike-esp-outbound.yml. You can create this file anywhere in your system. When you run a Docker container in the next step, it mounts the file into the container’s file system.

   For descriptions of the configuration properties and a sample configuration file, see Configure event stream processing outbound connector.

4. Mount the configuration file in the container.

   sudo docker run \
    -v <path to local aerospike-esp-outbound.yml>:/etc/aerospike-esp-outbound/aerospike-esp-outbound.yml \
    container.aerospike.com/aerospike/aerospike-esp-outbound:latest

   <path to local aerospike-esp-outbound.yml>: The path (including the name) of the configuration file in your local filesystem.