---
title: "Configure Aerospike Connect for Elasticsearch"
description: "Guide to configuring Aerospike Connect for Elasticsearch streaming via XDR and the outbound configuration file."
---

# Configure Aerospike Connect for Elasticsearch

> For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt)
> 
> All documentation pages available in markdown.

Configuring streaming from Aerospike to Elasticsearch involves setting up your Aerospike database to send change notifications and modifying the configuration file `/etc/aerospike-elasticsearch-outbound/aerospike-elasticsearch-outbound.yml`.

## Set Up Aerospike Database

In your Aerospike database, you must configure Cross-Datacenter Replication (XDR) and enable change notification. You must also configure your Elasticsearch outbound cluster to be a “connector” XDR datacenter and the namespace must point to this as xdr remote datacenter.

See the [change notification configuration parameters and example config](https://aerospike.com/docs/connectors/elasticsearch/configure/change-notification).

## Modify the Config File

You configure streaming from Aerospike to elasticsearch by modifying the `/etc/aerospike-elasticsearch-outbound/aerospike-elasticsearch-outbound.yml` YAML configuration file.

### Configuration levels

There are two levels at which various configuration can be defined:

| Level | Description |
| --- | --- |
| Root | This type of property can be configured only once at the root level of the configuration file. |
| All | This type of property can be configured at the root level, namespace level and/or at any of the sets’ level of that namespace. Generally, this type of configuration is applied to the Aerospike record. The set level configuration takes the higher precedence. If absent, namespace level configuration is used for the record of that particular set or if set is absent. If configuration is also absent at the namespace level, the root level configuration is used for the given record. |

### Configuration sections

The configuration file has the following sections:

| Section | Description | Configuration level |
| --- | --- | --- |
| [`service`](https://aerospike.com/docs/connectors/elasticsearch/configure/service) | Configures the connector’s listening ports, TLS, and network interface. | Root |
| [`es-client`](https://aerospike.com/docs/connectors/elasticsearch/configure/es-client) | Configures Elasticsearch client properties for the target elasticsearch cluster. | Root |
| [`doc-id`](https://aerospike.com/docs/connectors/elasticsearch/configure/doc-id) | Specifies how to generate Elasticsearch’s document id. | All |
| [`batching`](https://aerospike.com/docs/connectors/elasticsearch/configure/batching) | Specifies how to batch Aerospike records to send them to Elasticsearch in a single HTTP request. | All |
| [`bin-transforms`](https://aerospike.com/docs/connectors/elasticsearch/configure/bin-transforms) | Specifies the bin transformations to apply on the Aerospike record. | All |
| [`format`](https://aerospike.com/docs/connectors/elasticsearch/configure/format) | Specifies the message format to use for the outbound messages sent to the Elasticsearch cluster. | All |
| [`record-ordering`](https://aerospike.com/docs/connectors/elasticsearch/configure/record-ordering) | Configures ordering of outbound records. | All |
| [`routing`](https://aerospike.com/docs/connectors/elasticsearch/configure/routing) | Configures how incoming record updates/deletes from Aerospike are routed to the Elasticsearch cluster. | All |
| [`namespaces`](https://aerospike.com/docs/connectors/elasticsearch/configure/namespaces) | Configures bin transforms, format, routing at namespace and set level. | Root |
| [`logging`](https://aerospike.com/docs/connectors/elasticsearch/configure/logging) | Configures the destination and level for the connector’s logs. | Root |
| [`bulk-request-config`](https://aerospike.com/docs/connectors/elasticsearch/configure/bulk-request) | Configures the properties of Elasticsearch’s Bulk API. | All |
| [`ignore-error-codes`](https://aerospike.com/docs/connectors/elasticsearch/configure/error-codes) | Specifies a set of error codes to ignore from Elasticsearch. | All |

::: caution
If the route is not specified for an Aerospike record, then the record is skipped and logged at the _warn_ level.
:::

## Configuration Example

An example Aerospike Connect for Elasticsearch configuration:

```yaml
service:

  address: 0.0.0.0

  # Optional TLS section

  tls:

    port: 8080

    mutual-auth: true # For tcp server.

    # Mutual auth is triggered only if this property is non-empty in http proxy server.

    allowed-peer-names:

      - asd.aerospike.com

    trust-store:

      store-type: PEM

      certificate-files: tls/ca.aerospike.com.crt

    key-store:

      store-type: PEM

      store-file: tls/connector.aerospike.com.key.encrypted.pem

      store-password-file: tls/storepass

      certificate-chain-files: tls/connector.aerospike.com.crt

  manage:

    address: 0.0.0.0

    port: 8902

logging:

  file: /var/log/aerospike-elasticsearch-outbound/aerospike-elasticsearch-outbound.log

# Aerospike record routing to an Elasticsearch index.

routing:

  mode: static

  destination: aerospike

doc-id:

  # The digest is used as Elasticsearch's doc-id unless overridden at the child level.

  source: digest

es-client:

  cluster-config:

    type: on-prem

    nodes:

      - host:

          hostname: 192.168.123.234

          port: 9200

          scheme: https

  auth-config:

    # We support 3 types of authentication methods which is explained in the other file. Using api-key method here.

    type: api-key

    api-key-id-file: /tmp/api-key-id-file

    api-key-secret-file: /tmp/api-key-secret-file

  # Tls config to be used if Elasticsearch is configured to use TLS.

  tls-config:

    trust-store:

      store-file: tls/ca.aerospike.com.truststore.jks

      store-password-file: tls/storepass

  elasticsearch-request-config:

    # There are several other options which are defined in the separate file.

    connect-timeout: 10000

batching:

  # Overriding one of the batching parameter. Defaults are mentioned in the separate file.

  batch-interval: 1000

namespaces:

  customTransformer:

    batching:

      # We are explicitly disabling batching here as we are using class, not a batch-formatter-class. This means that it only

      # gets one record as an input. We get multiple records when batching is configured. This disables batching at

      # customTransformer namespace unless some specific set overrides it for itself.

      enabled: false

    format:

      mode: custom

      class: com.aerospike.connect.elasticsearch.outbound.TestElasticsearchOutboundCustomFormatter

    custom-transformer:

      class: com.aerospike.connect.outbound.TestOutboundCustomTransformer

    sets:

      ignoreErrorCodeSet:

        batching:

          enabled: false

        format:

          mode: custom

          class: com.aerospike.connect.elasticsearch.outbound.TestElasticsearchOutboundIgnoreErrorCodeCustomFormatter

        # We will consider the record to be successfully processed if Elasticsearch returns error code 400.

        ignore-error-codes:

          - 400

  root:

    sets:

      binTransformsSet:

        bin-transforms:

          map:

            original: transformed

          transforms:

            - uppercase

      docIdBinValueSet:

        doc-id:

          # Using bin-value type to generate a doc id. The default is digest at top.

          source: bin-value

          bin-name: es_doc_id

          failure-strategy: fail

      updateOperationSet:

        # Example of bulk-request-config for the update operation. There are more supported config options which are

        # defined on a separate page.

        bulk-request-config:

          ignore-aerospike-delete: true

          aerospike-write-operation-mapping:

            operation-type: update

            doc-as-upsert: true

      createOperationSet:

        doc-id:

          source: static

          value: UhP3zQBGIGbdqFIV5pqzdZB6rKA=

        # Example for bulk-request-config for create operation. There are more supported config options which are

        # defined on a separate page. Create operation means that this will succeed only if the doc doesn't exist in

        # Elasticsearch. That's why we have used static doc-id for this set. Static doc-id won't be used practically

        # by anyone. This is just for illustrative purpose.

        bulk-request-config:

          aerospike-write-operation-mapping:

            operation-type: create
```

## Mapping between Aerospike and Elasticsearch models

| RDBMS | Aerospike | Elasticsearch |
| --- | --- | --- |
| Database | Namespace | Index |
| Table | Set | Index (Database 6.0.0+) |
| Row | Record | Document |
| Column | Bin | Fields |
|  | Partitions | Shards |
| Schema |  | Mappings |
| Primary key | Primary key | document\_id |

_Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries._