Skip to main content
Aerospike Vector is now live with LangChain integrationTell me more
Loading

MessagePack Format for Outbound

MessagePack Serialization Format​

This document specifies the MessagePack data serialization format used by the Aerospike Connect for Kafka. It describes how Aerospike change notifications are represented using MessagePack. Each message represents an event that occurred in an Aerospike database cluster. Different kinds of messages are used to describe events such as record creation/update and record deletions.

MessagePack is a binary data serialization format. It's compact and simple form is designed for efficient transmission of data over the wire. MessagePack parser implementations exist for a wide variety of popular programming languages. The official MessagePack specification can be found at https://msgpack.org/. The message format used by the Aerospike Connect is compliant with the MessagePack spec as published on 2017-08-09.

Message format​

Each message consists of a MessagePack array with 3 elements:

+---------------+------------+------------------+
| Version [int] | Type [int] | Payload [varies] |
+---------------+------------+------------------+
  • Version: 1
  • Message Types:
    • WRITE: 1
    • DELETE: 2
  • Payload: Message payload depending on message type - see below.

Write Message Type Payload​

A WRITE message is a MessagePack array with 4 parts:

+-----+----------------------+-----------------------+----------------+--------------+
| Key | Generation [int|nil] | Expiry Time [int|nil] | LUT [long|nil] | Bins [array] |
+-----+----------------------+-----------------------+----------------+--------------+
  • Key: Record key - see below for MessagePack format. The record key in WRITE messages always contains the namespace and key digest, and may contain the set name and user key components.
  • Generation: Record generation, changed on each record update. [1]
  • Expiry Time: Time when the record will expire in seconds since Unix epoch; zero means record will not expire. [1]
  • LUT: Time when the record was last updated, in milliseconds since the Unix epoch. Will be available whenever the Aerospike server ships last-update time. [1], [2]
  • Bins: List of record bins - see below for MessagePack format for each bin.

[1] Breaking Change Generation, Expiry, LUT are null when they are not shipped by the Aerospike server. LUT has been upcast from an Integer to a Long.

[2] When the Aerospike server does not ship lut, then the following versions of these outbound connectors ship lut as zero:

  • JMS outbound connector, versions earlier than 3.0.0
  • Kafka outbound connector, versions earlier than 4.0.0
  • Pulsar outbound connector, versions earlier than 2.0.0

Breaking Change Writes are shipped with this array type in the following versions of the outbound connectors

  • JMS outbound connector, versions earlier than 3.0.0
  • Kafka outbound connector, versions earlier than 4.0.0
  • Pulsar outbound connector, versions earlier than 2.0.0
+-----+------------------+-------------------+-----------+--------------+
| Key | Generation [int] | Expiry Time [int] | LUT [int] | Bins [array] |
+-----+------------------+-------------------+-----------+--------------+

Delete Message Type Payload​

A DELETE message is a MessagePack array with 5 parts:

+-----+-------------+----------------------+------------------+----------------+
| Key | Flags [int] | Generation [int|nil] | Expiry [int|nil] | LUT [long|nil] |
+-----+-------------+-------------------+--+------------------+----------------+
  • Key: Record key - see below for MessagePack format. The record key in DELETE messages only contains the namespace and key digest, but not the set name or user key components.
  • Flags:
    • DURABLE_DELETE: 0x01
  • Generation: Record generation, changed on each record update. [1]
  • Expiry Time: Time when the record will expire in seconds since Unix epoch; zero means record will not expire. Will be available whenever the Aerospike server ships expiry for delete. [1]
  • LUT: Time when the record was last updated, in milliseconds since the Unix epoch. Will be available whenever the Aerospike server ships last-update time. [1]

[3] Breaking Change: Generation, Expiry, LUT are null when they are not shipped by the Aerospike server. LUT has been upcast from an Integer to a Long.

Breaking Change Deletes are shipped with this array type in the following versions of the outbound connectors

  • JMS outbound connector, versions earlier than 3.0.0
  • Kafka outbound connector, versions earlier than 4.0.0
  • Pulsar outbound connector, versions earlier than 2.0.0
+-----+--------------+
| Key | Flags [int]  |
+-----+--------------+

Key Format​

A record key consists of a MessagePack array with 4 elements. Only the namespace and digest are included in every message. The set name and user key are optional and may be nil.

+-----------------+---------------+--------------+----------------------------+
| Namespace [str] | Set [str|nil] | Digest [bin] | User Key [str|int|bin|nil] |
+-----------------+---------------+--------------+----------------------------+
  • Namespace: Aerospike namespace
  • Set: Set name within the namespace (optional)
  • Digest: 160-bit record digest.
  • User Key: The integer, string or bytes value used by the application to uniquely address the record. Will be included in the message only if it is stored on the server.

Bins Format​

Record bins are sent as a MessagePack array; each bin has the following format:

+------------+------------+-------------+----------------+
| Name [str] | Type [int] | Flags [int] | Value [varies] |
+------------+------------+-------------+----------------+
  • Name: Name of the bin.
  • Type: Type of the bin's value:
    • INTEGER: 1
    • DOUBLE: 2
    • STRING: 3
    • BLOB: 4
    • JAVA OBJ: 7 (Serialized Java Objects)
    • BOOLEAN: 17
    • MAP: 19
    • LIST: 20
    • GEOJSON: 23
  • Flags: Type specific flags:
    • MAP: Map Order:
      • Unordered: 0
      • Key-Ordered: 1
      • Key-Value-Ordered: 3
    • LIST: List Order:
      • Unordered: 0
      • Ordered: 1
    • Other: zero (0)
  • Value: The MessagePack format of the value depends on the bin's type:
    • INTEGER: int
    • DOUBLE: float
    • BOOLEAN: bool
    • STRING: str
    • BLOB: bin
    • JAVA OBJ: bin
    • MAP: map
    • LIST: array
    • GEOJSON: str

Map and list elements can be of any allowed bin data type, including nested lists and maps. Within maps and lists only, serialized Java Objects and GeoJSON strings are represented using the MessagePack ext format type, with the ext type being the same as the corresponding bin data type.

Batch MessagePack Serialization Format​

If batching is configured then the records in the batch will be a MessagePack array of MessagePack formatted records.

If the keys of a batch are configured to be concatenated, then the payload key of the batch will be a MessagePack array of MessagePack formatted keys.

Last updated on