Skip to content

Configure Aerospike and the Elasticsearch connector

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

In the payment service, your authorization service writes dispute records to Aerospike in the test namespace. The configuration in this section connects that write path to the support search path so that every change in Aerospike is published to the Elasticsearch connector.

Create both aerospike.conf and aerospike-elasticsearch-outbound.yml in your project directory. Neither file does anything on its own. When you compose the stack, Docker Compose mounts each file into its container as a read-only volume, so Aerospike Database and the Elasticsearch connector load your local configuration at startup.

  1. Create aerospike.conf to run an in-memory test namespace and send its changes through XDR connector mode to a destination named esDC. This is a local single-machine walkthrough with no remote datacenter connections. For production, adapt the configuration to your cluster topology, durability, and security requirements.

    aerospike.conf
    service {
        cluster-name elasticsearch-demo
        feature-key-file /etc/aerospike/features.conf
    }
    

    logging {
        console {
            context any info
        }
    }
    

    network {
        service {
            address any
            port 3000
        }
        heartbeat {
            mode mesh
            port 3002
            address any
            interval 150
            timeout 10
        }
        fabric {
            port 3001
            address any
        }
        info {
            port 3003
            address any
        }
    }
    

    namespace test {
        replication-factor 1
        storage-engine memory {
            data-size 1G
        }
        nsup-period 10
    }
    

    xdr {
        dc esDC {
            connector true
            node-address-port aerospike-elasticsearch-outbound 8080
            namespace test {
            }
        }
    }

    The xdr configuration is the change-notification subscription:

    • connector true marks the esDC destination as an Aerospike connector that receives change-notification messages. With the default connector false, the nodes in node-address-port define a remote Aerospike cluster instead.
    • node-address-port aerospike-elasticsearch-outbound 8080 sets the connector hostname and port where change notifications are delivered. Docker Compose resolves the service name aerospike-elasticsearch-outbound to the container’s internal IP.
    • namespace test { } enables shipping for the test namespace. Only records in listed namespaces are shipped.

    See Configure change notification for the complete XDR requirements.

  2. Create aerospike-elasticsearch-outbound.yml to receive XDR messages and write them to an Elasticsearch index named aerospike. This file defines the mapping between a dispute record and a searchable document: which index it lands in, how its document ID is derived, and how Aerospike bins relate to Elasticsearch fields.

    aerospike-elasticsearch-outbound.yml
    service:
        address: 0.0.0.0
        port: 8080
        manage:
            address: 0.0.0.0
            port: 8902
    

    logging:
        enable-console-logging: true
    

    routing:
        mode: static
        destination: aerospike
    

    doc-id:
        source: digest
    

    format:
        mode: custom
        batch-formatter-class: com.aerospike.connect.elasticsearch.outbound.formatter.ElasticsearchDefaultBatchFormatter
        payload-format:
            mode: flat-json
            metadata-key: metadata
    

    es-client:
        cluster-config:
            type: on-prem
            nodes:
                - host:
                      hostname: elasticsearch
                      port: 9200
                      scheme: http
        auth-config:
            type: none

    The configuration uses these choices:

    • service.manage.port: 8902 exposes the management and metrics endpoints over an unencrypted HTTP connection. Restrict or enable TLS on this port in production. See Connector settings for details.
    • Static routing sends every record to the aerospike index. doc-id.source: digest gives each document a stable ID derived from the Aerospike record digest, so changes to the same primary key replace the existing Elasticsearch document. See Doc ID for other ID sources.
    • batch-formatter-class selects the default Elasticsearch batch formatter built into the connector image. The flat-json payload puts Aerospike bins such as merchant, status, and memo at the top level so support agents can search them directly, with record metadata under metadata.
    • es-client connects to the self-managed Elasticsearch container over HTTP.