---
title: "Configure bulk request for Aerospike Connect for Elasticsearch"
description: "Configure bulk request settings for the Aerospike Connect for Elasticsearch outbound connector."
---

# Configure bulk request for Aerospike Connect for Elasticsearch

> For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt)
> 
> All documentation pages available in markdown.

The `bulk-request-config` section of the `aerospike-elasticsearch-outbound.yaml` specifies the configuration parameters for [BulkRequest](https://github.com/elastic/elasticsearch-java/blob/v8.4.2/java-client/src/main/java/co/elastic/clients/elasticsearch/core/BulkRequest.java). The Elasticsearch outbound connector always sends data to Elasticsearch in a BulkRequest. The connector uses the following parameter with its default values unless you override them or write your own custom batch-formatter/formatter.

| Option | Required | Default | Description |
| --- | --- | --- | --- |
| pipeline | No | Elasticsearch’s default | A string value defining an Elasticsearch’s `pipeline` field to use for pre-processing incoming documents. |
| refresh | No | Elasticsearch’s default | Whether or not to refresh the affected shards to make this operation visible to search. This parameter defines an Elasticsearch’s `refresh` field. the [Refresh configuration options](https://aerospike.com/docs/connectors/elasticsearch/configure/bulk-request/#refresh-configuration-options) section describes all possible options. |
| require-alias | No | Elasticsearch’s default | Boolean field defining an Elasticsearch’s `require_alias` field for all incoming documents. |
| shard-routing | No | Elasticsearch’s default | Specific shard routing value. [The shard routing configuration options section](#shard-routing-configuration-options) lists all possible options examples. |
| timeout | No | Elasticsearch’s default | String value defining Elasticsearch’s `timeout` field. [The Timeout configuration options section](#timeout-configuration-options) lists all possible options examples. |
| wait-for-active-shards | No | Elasticsearch’s default | Number of shard copies that must be active before proceeding with the bulk operation. See [Wait for active shards configuration options section](#wait-for-active-shards-configuration-options) for configuration options. |
| aerospike-write-operation-mapping | No | Operation type `index` | Mapping of an Aerospike XDR’s write operation to Elasticsearch operation. See [XDR Write operation mapping section](#xdr-write-operation-mapping) for more details. |
| if-primary-term | No | Elasticsearch’s default | Integer value defining the constraint that it should only perform the operation if the document has this primary term. |
| if-seq-no | No | Elasticsearch’s default | Integer value defining the constraint that it should only perform the operation if the document has this sequence number. |
| version | No | Elasticsearch’s default | Long value defining the explicit version number for concurrency control. The specified version must match the current version of the document for the request to succeed. |
| version-type | No | Elasticsearch’s default | Specific version type. See [Version type configuration options](#version-type-configuration-options) for configuration options. |
| ignore-aerospike-delete | No | false | Boolean flag that asks not to process Aerospike’s delete record to avoid document deletion from Elasticsearch. |

## Refresh configuration options

An optional `refresh` field can use one of the three possible options:

| Option | Description |
| --- | --- |
| true | Refresh the relevant primary and replica shards (not the whole index) immediately after the operation occurs, so that the updated document appears in search results immediately. This should ONLY be done after careful thought and verification that it does not lead to poor performance, both from an indexing and a search standpoint. |
| false | Take no refresh related actions. The changes made by this request will be made visible at some point after the request returns. |
| wait\_for | Wait for the changes made by the request to be made visible by a refresh before replying. This doesn’t force an immediate refresh, rather, it waits for a refresh to happen. |

## Shard Routing configuration options

This field has the following two properties:

| Property | Required | Default | Description |
| --- | --- | --- | --- |
| source | Yes |  | Source for generating Elasticsearch’s `shard-routing` value. |
| failure-strategy | No | `USE_DIGEST` | What to do if Aerospike can’t generate the `shard-routing` value using the specified source. While not all sources are compatible with this configuration parameter, you can still generate the `shard-routing` value using those sources. |

## Source values

The source field accepts the following values:

| Value | Description | Can configure failure-strategy |
| --- | --- | --- |
| none | The destination system will auto-generate the value or use null. | No |
| system-default | The connector sets a different default source. In this case, it is set to source none. | No |
| namespace | Use the record’s namespace as Elasticsearch’s `shard-routing` value. | No |
| set | Use the record’s set name as Elasticsearch’s `shard-routing` value. | Yes |
| digest | Use the record’s digest as Elasticsearch’s `shard-routing` value. | No |
| user-key | Use the record’s user-key as Elasticsearch’s `shard-routing` value. | Yes |
| bin-value | Use the record’s bin value of the given bin name as Elasticsearch’s `shard-routing` value. | Yes |
| static | Use a static value as Elasticsearch’s `shard-routing` value. | No |

## Failure Strategy values

While not all sources are compatible with this configuration parameter, you can still generate the `shard-routing` value using those sources.

The failure-strategy field accepts the following values:

| Value | Description | Details |
| --- | --- | --- |
| USE\_DIGEST | Use the record’s digest as Elasticsearch’s `shard-routing` value. | This value is always available. |
| FAIL | Send the temporary error to XDR so that it will retry the record. |  |
| IGNORE | Send the permanent error to XDR so that it will not retry the record. | In this situation, the record is never shipped to the destination system. |

## Examples

You can use any of the failure-strategy values wherever applicable. The following examples show usage of all failure-strategy values at different places.

### Source: none

```
…bulk-request-config:  shard-routing:    source: none…
```

### Source: system-default

```
…bulk-request-config:  shard-routing:    source: system-default…
```

### Source: namespace

```
…bulk-request-config:  shard-routing:    source: namespace…
```

### Source: set

```
…bulk-request-config:  shard-routing:    source: set    failure-strategy: USE_DIGEST…
```

### Source: digest

```
…bulk-request-config:  shard-routing:    source: digest…
```

### Source: user-key

```
…bulk-request-config:  shard-routing:    source: user-key    failure-strategy: FAIL…
```

### Source: bin-value

```
…bulk-request-config:  shard-routing:    source: bin-value    bin-name: color    failure-strategy: IGNORE…
```

### Source: static

```
…bulk-request-config:  shard-routing:    source: static    value: dummy_doc_id…
```

## Timeout configuration options

An optional operation `timeout` field supports following duration units:

| Unit | Example |
| --- | --- |
| `m` - Minutes | 1m |
| `s` - Seconds | 10s |
| `ms` - Milliseconds | 200ms |
| `micros` - Microseconds | 10000micros |
| `nanos` - Nanoseconds | 1000000nanos |

### Example

```yaml
bulk-request-config:

  ...

  timeout: 200ms

  ...

namespaces:

  players:

    bulk-request-config:

      ...

      timeout: 3s

      ...
```

## Wait for active shards configuration options

An optional `wait-for-active-shards` field requires two sub-properties:

| Name | Value |
| --- | --- |
| kind | Defines how a timeout value should be interpreted. Valid values are `option` (predefined options by Elasticsearch) and `count` (an active shard count). |
| value | The actual value allowed as per the kind property. It should be one of `all` or `index-setting` when kind is an `option` and a number when kind is a `count`. |

### Example

```yaml
bulk-request-config:

  ...

  # All shards must be active before proceeding the operation.

  wait-for-active-shards:

    kind: option

    value: all

  ...

namespaces:

  players:

    bulk-request-config:

      ...

      # Follow index set wait-for-active-shard setting.

      wait-for-active-shards:

        kind: option

        value: index-setting

      ...

    sets:

      gujarati:

        bulk-request-config:

          ...

          # 2 shards must be active before proceeding the operation.

          wait-for-active-shards:

            kind: count

            value: 2

          ...
```

## Version type configuration options

An optional `version-type` field can use one of the four possible options:

| Option | Description |
| --- | --- |
| internal | The default versioning scheme provided by Elasticsearch that starts at 1 and increments with each update and delete. |
| external | The user defines their own version number. The value provided must be a numeric, long value greater than or equal to 0, and less than around 9.2e+18. |
| external\_gte | The given version should be equal or higher than the version of the stored document. If there is no existing document the operation will succeed as well. |
| force | This option is deprecated by Elasticsearch because it can cause primary and replica shards to diverge. |

## XDR write operation mapping

This configuration property has a required sub-property `operation-type` which defines how to map XDR’s write operation to Elasticsearch operations. See [Create and index optional parameters](https://aerospike.com/docs/connectors/elasticsearch/configure/bulk-request/#create-and-index-optional-parameters) for supported optional configuration parameters. There are three options:

| Elasticsearch Operation | Description |
| --- | --- |
| `create` | Map Aerospike’s `write` operation to Elasticsearch’s `create` operation. |
| `index` | Map Aerospike’s `write` operation to Elasticsearch’s `index` operation. |
| `update` | Map Aerospike’s `write` operation to Elasticsearch’s `update` operation. See [Update optional parameters](https://aerospike.com/docs/connectors/elasticsearch/configure/bulk-request/#update-optional-parameters) for supported optional configuration parameters. |

### Create and index optional parameters

| Parameter | Description |
| --- | --- |
| dynamic-template | Define a custom mappings that can be applied to dynamically added fields based on the matching condition. See [dynamic templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html) for more details. |

#### Example

```yaml
bulk-request-config:

  ...

  aerospike-write-operation-mapping:

    operation-type: index

    # Set dynamic-templates for operation-type index.

    dynamic-templates: [

      {

        strings_as_ip: "{\"match_mapping_type\":\"string\",\"match\":\"ip*\",\"runtime\":{\"type\":\"ip\"}}"

      }

    ]

  ...

namespaces:

  players:

    bulk-request-config:

      ...

      aerospike-write-operation-mapping:

        operation-type: create

        # Set dynamic-templates for operation-type create.

        dynamic-templates: [

          {

            integers: "{\"match_mapping_type\":\"long\",\"mapping\":{\"type\":\"integer\"}}"

          },

          {

            strings: "{\"match_mapping_type\":\"string\",\"mapping\":{\"type\":\"text\",\"fields\":{\"raw\":{\"type\":\"keyword\",\"ignore_above\":256}}}}"

          }

        ]

      ...
```

### Update optional parameters

| Parameter | Description |
| --- | --- |
| retry-on-conflict | An integer value specifying how many times an update should be retried in the case of a version conflict. |
| doc-as-upsert | Instead of sending a partial doc plus an upsert doc, you can set doc\_as\_upsert to true to use the contents of doc as the upsert value. |

#### Example

```yaml
bulk-request-config:

  ...

  aerospike-write-operation-mapping:

    operation-type: update

    # Set retry-on-conflict and doc-as-upsert for operation-type update.

    retry-on-conflict: 3

    doc-as-upsert: true

  ...
```

## Example

```yaml
...

bulk-request-config:

  pipeline: testPipeline

  refresh: wait_for

  require-alias: true

  shard-routing:

    source: digest

  timeout:

    kind: time

    value: 2m

  wait-for-active-shards:

    kind: option

    value: index-setting

  aerospike-write-operation-mapping:

    operation-type: create

    dynamic-templates: [

      {

        strings_as_ip: "{\"match_mapping_type\":\"string\",\"match\":\"ip*\",\"runtime\":{\"type\":\"ip\"}}"

      }

    ]

  if-primary-term: 2

  if-seq-no: 3

  version: 4

  version-type: external_gte

  ignore-aerospike-delete: true

...
```