Configure message format for Pulsar Outbound
Format
The format section specifies the serialization format of the data output. The supported formats are:
- Avro - data will be converted to Avro format
- Flat JSON - data will be converted to Flat JSON format
- JSON - data will be converted to JSON format
- MessagePack - data will be converted to MessagePack format
- Custom - data will be converted to a custom format
WARNING
- If the Aerospike record cannot be serialized as per the format configuration then the record will be skipped.
- HyperLogLog bins will be forwarded as is, as bytes or Base64 encoded strings for string based formats.
JSON
Specifies that the data be serialized to JSON
format with the type information of bins.
| Option | Required | Default | Expected value | Description |
|---|---|---|---|---|
| mode | yes | json | Selects JSON format. | |
| stringify-map-keys | no | true | Whether the numeric keys in CDT maps should be converted to strings. See Stringify Map Keys |
Stringify Map Keys
Aerospike allows many data types as keys in collection data types (CDTs) Maps, but JSON only allows strings as keys in objects. As of now the Pulsar connector only permits for conversion of numeric types to string keys when converting Aerospike CDT maps.
An Aerospike CDT map with any other data type as key will fail to be parsed
in the Pulsar connector. Also, an Aerospike CDT map with numeric key, but
stringify-map-keys set to false will fail to be parsed in the Pulsar connector.
Example
format:
mode: json
Flat JSON
Specifies that the data be serialized to Flat JSON format. The type information of bins is not part of the output.
| Option | Required | Default | Expected value | Description |
|---|---|---|---|---|
| mode | yes | flat-json | Selects Flat JSON format. | |
| metadata-key | yes | The Aerospike record metadata will be inserted into this top level key. | ||
| stringify-map-keys | no | true | Whether the numeric keys in CDT maps should be converted to strings. See Stringify Map Keys |
Example
format:
mode: flat-json
metadata-key: metadata
MessagePack
Specifies that the data be serialized to MessagePack format
with the type information of bins.
| Option | Required | Expected value | Description |
|---|---|---|---|
| mode | yes | message-pack | Selects message pack format. |
Example
format:
mode: message-pack
Avro
Specifies that the data be serialized as Avro format. The Avro schema can be a map or a record. In the case of a map, just specifying the type of the map values is sufficient. In the case of a record, the exact field names and types need to be specified.
| Option | Required | Default | Expected value | Description |
|---|---|---|---|---|
| mode | yes | avro | Selects avro format. | |
| schema | no | The schema of the data. | ||
| schema-file | no | The file containing the schema of the data. | ||
| stringify-map-keys | no | true | Whether the numeric keys in CDT maps should be converted to strings. See Stringify Map Keys |
One of schema or schema-file has to be specified.
Aerospike record metadata
The Aerospike record metadata has the following fields
| Metadata | Type | Description | Present in |
|---|---|---|---|
| msg | string | Write/Delete operation. | Delete and Write |
| namespace | string | Namespace of the Aerospike record. | Delete and Write |
| set | string | Set of the Aerospike record. | Write |
| userKey | long, double, bytes or string | The user key of the Aerospike record. | Write (only if user key is stored on server) |
| digest | bytes | The digest of the Aerospike record. | Delete and Write |
| gen | int | The generation of the Aerospike record. | Write |
| lut | int | The last update time of the Aerospike record. | Write |
| exp | int | The expiry of the Aerospike record. | Write |
| durable | boolean | Whether the delete is durable. | Delete |
Custom
Specifies that the data be serialized with user provided custom code. See Format Transformer.
Example
format:
mode: custom
class: com.aerospike.connect.outbound.example.Formatter