--- title: "Record ordering" description: "Recommended architecture for maintaining record order with the ESP outbound connector and XDR Proxy." --- # Record ordering > For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt) > > All documentation pages available in markdown. This page describes the recommended deployment pattern and configuration for keeping record versions in order with the [ESP outbound connector](https://aerospike.com/docs/connectors/streaming/esp) and the optional [XDR Proxy](https://aerospike.com/docs/database/manage/xdr/proxy). ## Prerequisites - [Change notifications](https://aerospike.com/docs/connectors/streaming/common/change-notification) configured on the source Aerospike Database cluster - One [ESP outbound connector](https://aerospike.com/docs/connectors/streaming/esp/inbound-install) instance per source node - An HTTP destination or [XDR Proxy](https://aerospike.com/docs/database/manage/xdr/proxy) cluster to receive change notifications ## How record ordering works Cross-Datacenter Replication (XDR) change notifications can arrive out of order in some network configurations, causing older record versions to overwrite newer ones at the destination. To prevent this, you can guarantee sequential record delivery by enabling a 1:1 pairing between each source node and its local ESP connector, alongside the connector’s record-ordering setting. Here is how they work together: - 1:1 Node-to-connector pairing: Ensures each source node ships changes only to its co-hosted connector, meaning every version from that node reaches a single connector instance. - Record-ordering setting: Uses a Last Update Time (LUT) cache on each connector to filter out stale record versions. It also applies an in-flight lock per record, in which the connector dispatches one version at a time and waits for an HTTP success response before sending the next. Under normal operation, this architecture ensures the destination receives record versions in strict, non-decreasing LUT order. Downstream HTTP load balancers and XDR Proxy hops do not break ordering once it is enforced at the source. ## Data flow simulation The following interactive diagram compares two deployment patterns. Launch the simulation and toggle between the two patterns with the switch at the top. Use the timeline scrubber and pause control to step through the event log at your own pace. The simulation starts in load-balanced mode (not recommended). After it finishes, toggle to 1:1 sidecar connectors to compare. **Load balancer in front of connectors (not recommended):** A source node emits rapid updates to the same record. An ESP load balancer splits traffic across three connector instances, each with its own LUT cache. ESP 1 experiences network lag and takes a while to dispatch the update. Because its cache is independent from the other connectors, it ships a stale update after the other connectors have already advanced the destination. When ESP 1 finally dispatches, the destination receives an older LUT after newer ones were already applied, resulting in out-of-order delivery. **1:1 sidecar connectors (recommended):** A three-node source cluster runs one connector per source node. In this run, Source 1 writes two successive versions of the same record (LUT 100, then LUT 101). Connector 1 dispatches the LUT 100 version, holds the LUT 101 version behind a lock until the destination returns HTTP 200, then ships it. Traffic passes through an HTTP load balancer and XDR Proxy on the way to the destination without reordering versions. Load-balanced connectors 1:1 sidecar connectors Launch the simulation to explore record ordering. Timeline 0.0s Launch Simulation Routine update Stale update (lagging ESP) Record A / LUT 100 Record B / LUT 101 STOP: in-flight lock HTTP 200: success ack Event log ## Set up record ordering Deploy one ESP connector per source node. Do not place a load balancer in front of the connectors on the source cluster, because different versions of the same record can land on different connector instances and arrive out of order at the destination. 1. On each source node, install the ESP outbound connector. See [Install event stream processing outbound connector](https://aerospike.com/docs/connectors/streaming/esp/inbound-install) for package, Docker, and Kubernetes installation options. 2. On each source node, create or edit `/etc/aerospike-esp-outbound/aerospike-esp-outbound.yml`. Add a `record-ordering` block and configure a destination for your topology. See [Example: direct HTTP destination](#example-direct-http-destination) or [Example: XDR Proxy destination](#example-xdr-proxy-destination) for complete files you can adapt. 3. On each source node, configure XDR to ship change notifications to the local connector. In `/etc/aerospike/aerospike.conf`, add a datacenter block to the `xdr` section. Set `connector` to `true` and point `node-address-port` at the connector on the same host. In a sidecar deployment, use `localhost` and the port from `service.port` or `service.ports` in `aerospike-esp-outbound.yml`. aerospike.conf ```text xdr { dc esp { connector true node-address-port localhost 8901 namespace sessions { } } } ``` Repeat this block on every node in the source cluster, listing only the local connector address on each node. For prerequisites, feature-key requirements, and additional XDR properties, see [Configure change notification](https://aerospike.com/docs/connectors/streaming/common/change-notification). 4. Restart the source node and start the connector on each host. Terminal window ```shell sudo systemctl restart aerospike sudo systemctl enable aerospike-esp-outbound sudo systemctl start aerospike-esp-outbound ``` ## Example: direct HTTP destination Use the following pattern when each connector writes change notifications directly to an HTTP endpoint, such as an API Gateway or application server. ![Architecture for ESP outbound record ordering with a direct HTTP destination.](https://aerospike.com/docs/_astro/esp_record_ordering_architecture.BsLdKZnQ_1W8Saa.png) `aerospike-esp-outbound.yml` listens on port `8901`, enables record ordering, and routes all namespaces to a single HTTP destination. aerospike-esp-outbound.yml ```yaml service: ports: - 8901 manage: address: 0.0.0.0 port: 8902 record-ordering: enable: true lut-cache-ttl-seconds: 30 routing: mode: static-multi-destination destinations: - http-destination destinations: http-destination: urls: - https://api.example.com/events protocol: HTTP_2 max-connections-per-endpoint: 100 call-timeout: 10000 connect-timeout: 2000 health-check: call-timeout: 3000 logging: file: /var/log/aerospike-esp-outbound/aerospike-esp-outbound.log ``` Set `lut-cache-ttl-seconds` to a value greater than the maximum delay a record can encounter between XDR dispatch and connector receipt. See [Configure record ordering](https://aerospike.com/docs/connectors/streaming/esp/configure/record-ordering) for LUT cache sizing and tuning. ## Example: XDR Proxy destination Use this pattern when replicating to a remote Aerospike Database cluster through an [XDR Proxy](https://aerospike.com/docs/database/manage/xdr/proxy) cluster running in HTTP mode. The proxy cluster may sit behind a load balancer. A load balancer in front of the proxy is acceptable because ordering is enforced on the source cluster before traffic leaves each connector. ![Architecture for ESP outbound record ordering with an XDR Proxy destination behind a load balancer.](https://aerospike.com/docs/_astro/xdr_proxy_record_ordering_architecture.BE2ylXcX_28ylTW.png) `aerospike-esp-outbound.yml` ships ordered change notifications from each source node to the proxy cluster. Each connector on the source cluster uses the same destination URL, typically the load balancer in front of the proxy. aerospike-esp-outbound.yml ```yaml service: ports: - 8901 manage: address: 0.0.0.0 port: 8902 record-ordering: enable: true lut-cache-ttl-seconds: 30 routing: mode: static-multi-destination destinations: - xdr-proxy destinations: xdr-proxy: urls: - https://xdr-proxy-lb.example.com:8443 protocol: HTTP_2 max-connections-per-endpoint: 20 connection-ttl: 300000 call-timeout: 10000 connect-timeout: 5000 health-check: call-timeout: 5100 interval-seconds: 5 tls: trust-store: store-file: /etc/aerospike-esp-outbound/certs/ca.truststore.jks store-password-file: /etc/aerospike-esp-outbound/certs/storepass logging: file: /var/log/aerospike-esp-outbound/aerospike-esp-outbound.log ``` If your proxy cluster sits behind a Layer 4 (L4) load balancer, configure `connection-ttl`. This forces the connector to periodically open new connections, ensuring traffic redistributes evenly when you scale the proxy cluster. By contrast, a Layer 7 (L7) load balancer inherently understands HTTP traffic and will redistribute requests automatically without requiring this setting. ::: note Using the ESP load balancer in the XDR Proxy architecture causes extra latencies. ::: ## Deploy as a Kubernetes sidecar On Kubernetes, run the ESP connector as a sidecar in the same pod as each Aerospike Database container. The Aerospike Kubernetes Operator mounts the connector configuration from a ConfigMap and points XDR at `localhost`. 1. Create a directory with your connector configuration and any TLS files. Terminal window ```shell mkdir aerospike-esp-outbound cp aerospike-esp-outbound.yml aerospike-esp-outbound/ ``` Use one of the [example configuration files](#example-direct-http-destination) above as a starting point. Ensure `record-ordering` is enabled and `service.ports` includes the port XDR will use. 2. Create a ConfigMap from the directory. Terminal window ```shell kubectl -n aerospike create configmap aerospike-esp-outbound-config \ --from-file=aerospike-esp-outbound/ --dry-run=client -o yaml | kubectl apply -f - ``` 3. Add the sidecar and mount the ConfigMap in your `AerospikeCluster` custom resource. Mount the ConfigMap at `/etc/aerospike-esp-outbound` for the sidecar container. In `aerospikeConfig`, set `connector: true` and point `node-address-ports` at `localhost` and the connector listen port. AerospikeCluster custom resource (excerpt) ```yaml spec: storage: volumes: - name: aerospike-esp-outbound-config source: configMap: name: aerospike-esp-outbound-config sidecars: - containerName: esp-outbound path: /etc/aerospike-esp-outbound aerospikeConfig: xdr: dcs: - name: esp connector: true node-address-ports: - localhost 8901 namespaces: - name: sessions podSpec: sidecars: - name: esp-outbound image: aerospike/aerospike-esp-outbound:latest ports: - containerPort: 8901 - containerPort: 8902 ``` 4. Apply the updated custom resource. Terminal window ```shell kubectl apply -f aerospike-cluster.yaml ``` ## Verify 1. Verify that each source node is configured to ship XDR change notifications exclusively to its co-hosted local connector. On each node, check that `node-address-port` in the XDR datacenter block points to `localhost` and the connector listen port. There should be no shared connector endpoint or load balancer on the source cluster. 2. Verify that record ordering is enabled in each connector configuration. The `record-ordering` block should be present with `enable: true`. 3. For XDR Proxy destinations, verify that each connector ships to the proxy cluster in HTTP mode. Check that destination URLs point at the proxy or its load balancer and that `protocol` is `HTTP_1_1` or `HTTP_2`. 4. Check ordering at runtime. - Write the same record twice in quick succession on the source cluster. - Compare LUT values at the destination against the order of updates on the source. - The destination should not contain an older LUT after a newer one for the same record digest. ## Next steps - [Configure record ordering](https://aerospike.com/docs/connectors/streaming/esp/configure/record-ordering) for LUT cache behavior and tuning - [Install and configure XDR Proxy](https://aerospike.com/docs/database/manage/xdr/proxy/install) - [ESP outbound connector overview](https://aerospike.com/docs/connectors/streaming/esp)