Configure Event Stream Processing outbound connector

Configure the ESP outbound connector with the file aerospike-esp-outbound.yml.

Set up Aerospike Database

In your Aerospike database, you must configure Cross-Datacenter Replication (XDR) and enable change notification. You must also configure your ESP cluster to be a “connector” XDR datacenter. The namespace must point to this as an XDR remote datacenter.

See the change notification configuration parameters and example configuration.

Modify `aerospike-esp-outbound.yml`

Configure streaming from Aerospike for ESP by modifying the aerospike-esp-outbound.yml YAML configuration file.

The location of this file depends on how you plan to deploy, or have already deployed, the ESP outbound connector:

If you deploy in Docker, the file can be anywhere in your system as long as you include the path to it in the docker run command.
If you deploy in Kubernetes, this file is in a directory that also includes the deployment file.
if you deploy in a non-containerized environment, this file is located in the root-level directory, /etc/aerospike-esp-outbound/.

The configuration file has the following sections:

service Configures the connector’s listening ports, TLS, and network interface.
destinations Configures HTTP-based destinations where change notifications are sent.
bin-transforms Specifies the bin transformations to apply on the Aerospike record.
format Specifies the message format to use for the outbound messages sent to the message broker.
batching Specifies how to collect a group of Change Notification Records into a single batch of outbound messages sent to the HTTP endpoint.
routing Configures how incoming record updates and deletes from Aerospike are routed to the message broker.
namespaces Configures bin transforms, format, routing at namespace and set level.
logging Configures the destination and level for the connector logs.
port-based-config Allows alternate configurations based on which port received the change notification.
executor-thread-pool-size Size of the thread pool used to dispatch HTTP requests. The default value is twice the number of available processors.
record-ordering Configure ordering of incoming records sent to the outbound destination.

Cascading configuration values

You can configure the bin-transforms, format, and routing sections to include default values, values for each namespace, and values for each set within a namespace. The appropriate values are applied to each Aerospike record.

Here is an annotated example:

routing:
  mode: static
  destination: default <1>
‎‎
namespaces:
  users:
    routing:
      mode: static
      destination: users <2>
    format:
      mode: flat-json
      metadata-key: metadata
    sets:
      premium:
        routing:
          mode: static
          destination: premium <3>

Specifies that records in all namespaces are shipped to the default destination.
Specifies that records in the users namespace are shipped to the users destination.
Specifies that records in the premium set in the users namespace are shipped to the premium destination.

Example

A sample version of aerospike-esp-outbound.yml is as follows:

service:
  port:
    - 8901
    - 9901
  manage:
    port: 8902

logging:
  enable-console-logging: true

# Destinations
destinations:
  xdr-proxy-1:
    urls:
      - http://proxy1.example.com
    max-requests-queued-per-endpoint: 10240
    connection-ttl: 15000
    max-connections-per-endpoint: 100
    call-timeout: 10000
    connect-timeout: 2000
    health-check:
      call-timeout: 10000
    headers:
      authority-header-omit-default-port: true
      send-digest-header: true
      additional-headers:
        custom-header-1: custom-value-1
        custom-header-2: custom-header-2

  xdr-proxy-2:
    urls:
      - http://proxy2.example.com/

port-based-config:
  8901:
    routing:
      mode: static-multi-destination
      destinations: xdr-proxy-1
  9901:
    routing:
      mode: static-multi-destination
      destinations: xdr-proxy-2