Record ordering

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

This page describes the recommended deployment pattern and configuration for keeping record versions in order with the ESP outbound connector and the optional XDR Proxy.

Prerequisites

Change notifications configured on the source Aerospike Database cluster
One ESP outbound connector instance per source node
An HTTP destination or XDR Proxy cluster to receive change notifications

How record ordering works

Cross-Datacenter Replication (XDR) change notifications can arrive out of order in some network configurations, causing older record versions to overwrite newer ones at the destination.

To prevent this, you can guarantee sequential record delivery by enabling a 1:1 pairing between each source node and its local ESP connector, alongside the connector’s record-ordering setting. Here is how they work together:

1:1 Node-to-connector pairing: Ensures each source node ships changes only to its co-hosted connector, meaning every version from that node reaches a single connector instance.
Record-ordering setting: Uses a Last Update Time (LUT) cache on each connector to filter out stale record versions. It also applies an in-flight lock per record, in which the connector dispatches one version at a time and waits for an HTTP success response before sending the next.

Under normal operation, this architecture ensures the destination receives record versions in strict, non-decreasing LUT order. Downstream HTTP load balancers and XDR Proxy hops do not break ordering once it is enforced at the source.

Data flow simulation

The following interactive diagram compares two deployment patterns. Launch the simulation and toggle between the two patterns with the switch at the top. Use the timeline scrubber and pause control to step through the event log at your own pace. The simulation starts in load-balanced mode (not recommended). After it finishes, toggle to 1:1 sidecar connectors to compare.

Load balancer in front of connectors (not recommended): A source node emits rapid updates to the same record. An ESP load balancer splits traffic across three connector instances, each with its own LUT cache. ESP 1 experiences network lag and takes a while to dispatch the update. Because its cache is independent from the other connectors, it ships a stale update after the other connectors have already advanced the destination. When ESP 1 finally dispatches, the destination receives an older LUT after newer ones were already applied, resulting in out-of-order delivery.

1:1 sidecar connectors (recommended): A three-node source cluster runs one connector per source node. In this run, Source 1 writes two successive versions of the same record (LUT 100, then LUT 101). Connector 1 dispatches the LUT 100 version, holds the LUT 101 version behind a lock until the destination returns HTTP 200, then ships it. Traffic passes through an HTTP load balancer and XDR Proxy on the way to the destination without reordering versions.

Load-balanced connectors

1:1 sidecar connectors

Launch the simulation to explore record ordering.

Routine update Stale update (lagging ESP) Record A / LUT 100 Record B / LUT 101 STOP: in-flight lock HTTP 200: success ack

Event log

Set up record ordering

Deploy one ESP connector per source node. Do not place a load balancer in front of the connectors on the source cluster, because different versions of the same record can land on different connector instances and arrive out of order at the destination.

On each source node, install the ESP outbound connector.

See Install event stream processing outbound connector for package, Docker, and Kubernetes installation options.
On each source node, create or edit /etc/aerospike-esp-outbound/aerospike-esp-outbound.yml.

Add a record-ordering block and configure a destination for your topology. See Example: direct HTTP destination or Example: XDR Proxy destination for complete files you can adapt.
On each source node, configure XDR to ship change notifications to the local connector.

In /etc/aerospike/aerospike.conf, add a datacenter block to the xdr section. Set connector to true and point node-address-port at the connector on the same host. In a sidecar deployment, use localhost and the port from service.port or service.ports in aerospike-esp-outbound.yml.
aerospike.conf
```
xdr {
  dc esp {
    connector    true
    node-address-port    localhost 8901

    namespace sessions {
    }
  }
}
```
Repeat this block on every node in the source cluster, listing only the local connector address on each node. For prerequisites, feature-key requirements, and additional XDR properties, see Configure change notification.

Restart the source node and start the connector on each host.

sudo systemctl restart aerospike
sudo systemctl enable aerospike-esp-outbound
sudo systemctl start aerospike-esp-outbound

Example: direct HTTP destination

Use the following pattern when each connector writes change notifications directly to an HTTP endpoint, such as an API Gateway or application server.

Architecture for ESP outbound record ordering with a direct HTTP destination.

aerospike-esp-outbound.yml listens on port 8901, enables record ordering, and routes all namespaces to a single HTTP destination.

service:
  ports:
    - 8901
  manage:
    address: 0.0.0.0
    port: 8902

record-ordering:
  enable: true
  lut-cache-ttl-seconds: 30

routing:
  mode: static-multi-destination
  destinations:
    - http-destination

destinations:
  http-destination:
    urls:
      - https://api.example.com/events
    protocol: HTTP_2
    max-connections-per-endpoint: 100
    call-timeout: 10000
    connect-timeout: 2000
    health-check:
      call-timeout: 3000

logging:
  file: /var/log/aerospike-esp-outbound/aerospike-esp-outbound.log

Set lut-cache-ttl-seconds to a value greater than the maximum delay a record can encounter between XDR dispatch and connector receipt. See Configure record ordering for LUT cache sizing and tuning.

Example: XDR Proxy destination

Use this pattern when replicating to a remote Aerospike Database cluster through an XDR Proxy cluster running in HTTP mode. The proxy cluster may sit behind a load balancer. A load balancer in front of the proxy is acceptable because ordering is enforced on the source cluster before traffic leaves each connector.

Architecture for ESP outbound record ordering with an XDR Proxy destination behind a load balancer.

aerospike-esp-outbound.yml ships ordered change notifications from each source node to the proxy cluster. Each connector on the source cluster uses the same destination URL, typically the load balancer in front of the proxy.

service:
  ports:
    - 8901
  manage:
    address: 0.0.0.0
    port: 8902

record-ordering:
  enable: true
  lut-cache-ttl-seconds: 30

routing:
  mode: static-multi-destination
  destinations:
    - xdr-proxy

destinations:
  xdr-proxy:
    urls:
      - https://xdr-proxy-lb.example.com:8443
    protocol: HTTP_2
    max-connections-per-endpoint: 20
    connection-ttl: 300000
    call-timeout: 10000
    connect-timeout: 5000
    health-check:
      call-timeout: 5100
      interval-seconds: 5
    tls:
      trust-store:
        store-file: /etc/aerospike-esp-outbound/certs/ca.truststore.jks
        store-password-file: /etc/aerospike-esp-outbound/certs/storepass

logging:
  file: /var/log/aerospike-esp-outbound/aerospike-esp-outbound.log

If your proxy cluster sits behind a Layer 4 (L4) load balancer, configure connection-ttl. This forces the connector to periodically open new connections, ensuring traffic redistributes evenly when you scale the proxy cluster.

By contrast, a Layer 7 (L7) load balancer inherently understands HTTP traffic and will redistribute requests automatically without requiring this setting.

Deploy as a Kubernetes sidecar

On Kubernetes, run the ESP connector as a sidecar in the same pod as each Aerospike Database container. The Aerospike Kubernetes Operator mounts the connector configuration from a ConfigMap and points XDR at localhost.

Create a directory with your connector configuration and any TLS files.
Terminal window
```
mkdir aerospike-esp-outbound
cp aerospike-esp-outbound.yml aerospike-esp-outbound/
```
Use one of the example configuration files above as a starting point. Ensure record-ordering is enabled and service.ports includes the port XDR will use.

Create a ConfigMap from the directory.

kubectl -n aerospike create configmap aerospike-esp-outbound-config \
  --from-file=aerospike-esp-outbound/ --dry-run=client -o yaml | kubectl apply -f -

Add the sidecar and mount the ConfigMap in your AerospikeCluster custom resource.

Mount the ConfigMap at /etc/aerospike-esp-outbound for the sidecar container. In aerospikeConfig, set connector: true and point node-address-ports at localhost and the connector listen port.

spec:
  storage:
    volumes:
      - name: aerospike-esp-outbound-config
        source:
          configMap:
            name: aerospike-esp-outbound-config
        sidecars:
          - containerName: esp-outbound
            path: /etc/aerospike-esp-outbound

  aerospikeConfig:
    xdr:
      dcs:
        - name: esp
          connector: true
          node-address-ports:
            - localhost 8901
          namespaces:
            - name: sessions

  podSpec:
    sidecars:
      - name: esp-outbound
        image: aerospike/aerospike-esp-outbound:latest
        ports:
          - containerPort: 8901
          - containerPort: 8902

Apply the updated custom resource.
Terminal window
```
kubectl apply -f aerospike-cluster.yaml
```

Verify

Verify that each source node is configured to ship XDR change notifications exclusively to its co-hosted local connector.

On each node, check that node-address-port in the XDR datacenter block points to localhost and the connector listen port. There should be no shared connector endpoint or load balancer on the source cluster.
Verify that record ordering is enabled in each connector configuration.

The record-ordering block should be present with enable: true.
For XDR Proxy destinations, verify that each connector ships to the proxy cluster in HTTP mode.

Check that destination URLs point at the proxy or its load balancer and that protocol is HTTP_1_1 or HTTP_2.
Check ordering at runtime.
- Write the same record twice in quick succession on the source cluster.
- Compare LUT values at the destination against the order of updates on the source.
- The destination should not contain an older LUT after a newer one for the same record digest.

Next steps

Configure record ordering for LUT cache behavior and tuning
Install and configure XDR Proxy
ESP outbound connector overview