Configure formats for Aerospike Connect for Elasticsearch

Overview

This page describes the format section of the aerospike-elasticsearch-outbound.yaml file which specifies the serialization format of the data output.

Aerospike record metadata

Every Aerospike record has a metadata associated with it. Following is the list of metadata fields that XDR sends to connector with every record:

Metadata	Type	Description	Present in
msg	string	Write/Delete operation.	Delete and Write
namespace	string	Namespace of the Aerospike record.	Delete and Write
set	string	Set of the Aerospike record.	Delete and Write
userKey	long, double, bytes or string	User key of the Aerospike record.	Write (only if user key is stored on server)
digest	bytes	Digest of the Aerospike record.	Delete and Write
gen	int	Generation of the Aerospike record.	Write
lut	int	Last update time of the Aerospike record.	Write
exp	int	Expiry of the Aerospike record.	Write
durable	boolean	Whether the delete is durable.	Delete

Caution

• HyperLogLog bins are forwarded as is, as bytes or as Base64-encoded strings for string-based formats.

Caution

Elasticsearch has strict checks for data-types of all the documents of an index. All Aerospike record bins should be converted to Elasticsearch document fields according to the index’s mapping/schema. The connector reports Elasticsearch’s mapper_parsing_exception in case of a mapping violation.

The only supported format is custom. The payload-format of this custom format can either be JSON or FlatJSON.

Example

This example uses ElasticsearchDefaultBatchFormatter, the default batch formatter built into the connector image. To write your own formatter instead, see Format transformer.

...
format:
   mode: custom
   payload-format:
      mode: flat-json
   batch-formatter-class: com.aerospike.connect.elasticsearch.outbound.formatter.ElasticsearchDefaultBatchFormatter
...