Configure routing to a Kafka destination

The routing section of the /etc/aerospike-kafka-outbound/aerospike-kafka-outbound.yml controls how Aerospike records are routed to a destination topic in Apache Kafka.

A source connector can route messages to one or more topics. You can use the static routing mode to specify in advance the exact names of the topics to route messages to. You can also use three different dynamic routing modes to have the source connector determine the names of topics dynamically, based on information in records.

Here is a list of the routing modes:

Static mode

static Always routes to the specified topic. This is the default mode.
skip Skip dispatch of record to Kafka and ack success to XDR.

Dynamic modes

namespace Uses the namespace of each Aerospike record as the name of the topic to publish messages to.
set Uses the set of each Aerospike record as the name of the topic to publish to. For records without a set, you can specify a fallback route that uses the “static” routing mode.
bin Uses the value of a bin in each Aerospike record as the name of the topic to publish to. Only string, blob, and integer bin types are supported.

When you use one of the dynamic modes, you can transform the names of namespaces, sets, and bins by changing their case, removing whitespace, or using regular expressions to replace characters.

Static routing

Here are the configuration options:

Option	Required	Expected value	Description
`mode`	yes	`static`	Specifies to write messages to the topic that is named in the `destination` option.
`destination`	yes		Name of the destination topic.

Example

routing:
  mode: static
  destination: name-of-topic

Skip routing

For skip routing the configuration options are

Option	Required	Expected value	Description
`mode`	yes	`skip`	Skip dispatch of record to Kafka and ack success to XDR.

Example

routing:
  mode: skip

Routing by namespace names

Here are the configuration options:

Option	Required	Expected value	Description
`mode`	yes	`namespace`	Specifies to write to topics that have names that match names of namespaces in record metadata.
`default`	no		Default destination topic to use in case the namespace name is missing in the record or the destination topic is not found.
`transforms`	no		List of transformations to apply to the namespace name. See the Transforming dynamically derived names section for details.

Example

routing:
  mode: namespace
  default: default-topic
  transforms:
    - trim
    - regex:
        pattern: '(.*):(.*)'
        replacement: '$2:$1'
    - regex:
        pattern: '$'
        replacement: ':please'
    - uppercase

Routing by set names

Here are the configuration options:

Option	Required	Expected value	Description
`mode`	yes	`set`	Specifies to write to topics that have names that match names of sets in record metadata.
`default`	no		Default destination topic to use in case the set name is missing in the record or the destination topic is not found.
`transforms`	no		List of transformations to apply to the set name. See the Transforming dynamically derived names section for details.

Example

routing:
  mode: set
  transforms:
    - trim
    - regex:
        pattern: '(.*):(.*)'
        replacement: '$2:$1'
    - regex:
        pattern: '$'
        replacement: ':please'
    - uppercase

Routing by bin values

Here are the configuration options:

Option	Required	Expected value	Description
`mode`	yes	`bin`	Specifies to write to topics that have names that match values in the specified bin.
`bin`	yes		Name of the bin to pick value from.
`default`	yes		Default destination topic to use in case the bin is missing in the record or the destination topic is not found.
`transforms`	no		List of transformations to apply to the bin value. See the Transforming dynamically derived names section for details.

Example

routing:
  mode: bin
  bin: category
  default: test-topic
  transforms:
    - trim
    - regex:
        pattern: '[^A-Za-z0-9]'
        replacement: '-'
    - lowercase

Transforming dynamically derived names

You can configure a list of transforms that will be applied, in order, to the record’s set name, namespace path, or bin value to derive the name of the destination topic.

The following transformations are supported:

lowercase Converts to lowercase.
uppercase Converts to uppercase
trim Trims leading and trailing whitespace.
regex Matches against a regex pattern and replaces all occurrences with a replacement. The regex and replacement use Java regex syntax.

Example

The following transform configuration trims the route, replaces all non-alphanumeric characters with ’-’, and then converts the result to lowercase.

routing:
  mode: bin
  bin: category
  default: test-topic
  transforms:
    - trim
    - regex:
        pattern: '[^A-Za-z0-9]'
        replacement: '-'
    - lowercase

The following routing modes are available.

static Always route to a static topic.
namespace Use the name of the namespace of the Aerospike record as the name of the destination topic.
set Use the name of the set of the Aerospike record as the name of the destination topic.
bin Sets the route based on the value of a bin in the record. Only string, blob and integer bin-types are supported.

Static routing

For static routing, the configuration options are:

Option	Required	Expected value	Description
`mode`	yes	`static`	Selects static routing configuration.
`destination`	yes		Name of the destination topic.

Here is an example use of static routing:

routing:
  mode: static
  destination: name-of-topic

Set-name routing

For record set-name routing, the configuration options are:

Option	Required	Expected value	Description
`mode`	yes	`set`	Selects set name routing configuration.
`default`	no		Default destination topic to use in case the set name is missing in the record or the destination topic is not found.
`transforms`	no		List of transformations to apply to the set name. See the Transforms section for details.

Example

routing:
  mode: set
  transforms:
    - trim
    - regex:
        pattern: '(.*):(.*)'
        replacement: '$2:$1'
    - regex:
        pattern: '$'
        replacement: ':please'
    - uppercase

Namespace-name routing

For record namespace-name routing, the configuration options are:

Option	Required	Expected value	Description
`mode`	yes	`namespace`	Selects namespace name routing configuration.
`default`	no		Default destination topic to use in case the namespace name is missing in the record or the destination topic is not found.
`transforms`	no		List of transformations to apply to the namespace name. See the Transforms section for details.

Example

routing:
  mode: namespace
  default: default-topic
  transforms:
    - trim
    - regex:
        pattern: '(.*):(.*)'
        replacement: '$2:$1'
    - regex:
        pattern: '$'
        replacement: ':please'
    - uppercase

Bin-value routing

For routing according to bin values, the configuration options are:

Option	Required	Expected value	Description
`mode`	yes	`bin`	Selects bin based routing.
`bin`	yes		Name of the bin to pick value from.
`default`	yes		Default destination topic to use in case the bin is missing in the record or the destination topic is not found.
`transforms`	no		List of transformations to apply to the bin value. See the Transforms section for details.

Example

routing:
  mode: bin
  bin: category
  default: test-topic
  transforms:
    - trim
    - regex:
        pattern: '[^A-Za-z0-9]'
        replacement: '-'
    - lowercase

Custom Routing

Record can be also be routed with custom code. See Routing Transform.

Example

routing:
  mode: custom
  class: com.aerospike.connect.outbound.example.GenerationRouter

Transforms

You can configure a list of transforms that will be applied, in order, to the record’s set name, namespace path or bin value in order to derive the destination topic.

Currently, the following transforms are supported:

lowercase - converts the bin value to lowercase.
uppercase - converts the bin value to uppercase.
trim - trim leading and trailing whitespace.
regex - match against a regex pattern and replace all occurrences with a replacement. The regex and replacement uses Java regex syntax.

Example

The following transform configuration trims the route, replaces all non-alphanumeric characters with ’-’, and then converts the result to lowercase.

routing:
  mode: bin
  bin: category
  default: test-topic
  transforms:
    - trim
    - regex:
        pattern: '[^A-Za-z0-9]'
        replacement: '-'
    - lowercase