Configuring the inbound Kafka connections

Overview

This page describes how to configure the Aerospike section of the aerospike-kafka-inbound.yml file. This file specifies the connection properties that the connector must use to connect to your Aerospike database. This process takes place in the following stages:

1. List the seed nodes
2. Specify the cluster name
3. (Optional) Specify the rack ID
4. (Optional) Configure TLS
5. (Optional) Configure auth credentials
6. (Optional) Configure communication settings
7. (Optional) Configure throttle connections
8. (Optional) Configure timeouts
9. (Optional) Configure boolean particle type

List the seed nodes

Use the required seeds stanza to list the nodes in the Aerospike database cluster that you want the connector to connect to. By connecting to a seed node, the connector can discover all of the nodes in the cluster. The connector iterates through the list of nodes until it successfully connects to one of them, then it discovers the other nodes in the cluster.

The connector is an Aerospike Smart Client, and stores in memory the partition maps of each node in the cluster, so that it can send updates directly to the appropriate nodes. For more information, see Smart Client.

Property	Required	Default	Description
port	no	3000	The port to use when making connections to the Aerospike database.
tls-name	no		The TLS name of the Aerospike database.

Example

Following is an example with the seeds stanza at the top of the file:

aerospike:
  seeds:
    - 192.168.50.1:
        port: 3000
        tls-name: red
    - 192.168.50.2
  socket-timeout: 30000
  total-timeout: 1000
  use-bool-bin: false

Specify cluster name

If the database cluster is configured with a name, specify the name.

Use the cluster-name property to specify the name of the Aerospike database that you are streaming data to. The connector can run for a long time, during which nodes of the cluster can be removed from the cluster permanently or for maintenance. The IP addresses of those nodes could be reassigned to nodes that are in other Aerospike clusters or in non-Aerospike clusters. By specifying the name of the Aerospike cluster that you are streaming data to, you ensure that the connector always communicates only with nodes that are part of that cluster.

The value must match the value that is set by the configuration property cluster-name in the configuration file for the Aerospike database. For information about setting this property, see cluster-name in "Configuration parameter reference"

Example

Following is an example with the cluster-name property appearing after the seeds stanza:

aerospike:
  seeds:
    - 192.168.50.1:
        port: 3000
        tls-name: red
    - 192.168.50.2
  cluster-name: east
  socket-timeout: 30000
  total-timeout: 1000
  use-bool-bin: false

(Optional) Specify rack ID

If the Aerospike cluster is organized into racks, it may make sense to configure Aerospike's rack awareness feature which allows reads from partitions in the same logical rack to be prioritized, reducing both latency and network use. If a rack ID is specified in the Kafka connector , it reads from partitions in the same rack, even if the local partition is a replica partition. This is achieved using the smart client mechanism - the Aerospike client has detailed data about the placement of all partitions in a cluster.

For the Kafka inbound connector this will usually not make a difference as writes are always to the master partition. If a custom transformation is used and if reads are made by the custom transformer, they are made using the rack awareness feature. Under these circumstances, taking advantage of rack awareness could be beneficial.

note

Starting with Database 7.2, the active rack feature allows you to dynamically designate a particular rack-id to hold all master partition copies. For instructions on how to enable active rack, see Designate an active rack.

Example

Following is an example with the rack-id property set. If any reads are made, they will be made from partitions in rack 1. Also included is an example of use of tls-name from the previous step:

aerospike:
  seeds:
    - 192.168.50.1:
        port: 3000
        tls-name: red
    - 192.168.50.2
  cluster-name: east
  rack-id: 1

(Optional) Configure TLS

You can use the tls stanza to secure connections from the connector to your Aerospike database with Transport Layer Security (TLS).

This stanza is more complex than the seeds stanza in the first step. Following is an example up front.

aerospike:
  seeds:
    - 192.168.50.1:
        port: 3000
        tls-name: red
    - 192.168.50.2
  cluster-name: east
  tls:
    key-store:
      store-file: /path/to/store/file
      store-password-file: /path/to/store/password/file
      key-password-file: /path/to/key/password/file
      store-type: JKS
    trust-store:
      store-file: /path/to/store/file
      store-password-file: /path/to/store/password/file
      key-password-file: /path/to/key/password/file
      store-type: JKS
    ciphers:
      - TLS_RSA_WITH_3DES_EDE_CBC_SHA
    revoke-certificates:
      - 12345678

Following are descriptions of the main properties in the tls stanza:

Property	Required	Default	Description
key-store	yes		The key store containing the Aerospike client certificate for mutual authentication. See "Configuring key-store and trust-store".
trust-store	no	Default Java trust store.	The trust store containing trusted CA certificate for Aerospike database certificate. See "Configuring key-store and trust-store".
ciphers	no	default ciphers allowed by the JVM	Allowed list of TLS ciphers that clients can use for secure connections.
revoke-certificates	no		List of certificate serial numbers to reject.

Configuring key-store and trust-store

key-store takes these properties. trust-store can either take these properties or its default value.

Property	Required	Default	Description
store-file	yes		Store file
store-password-file	yes		File that contains the password to the store.
key-password-file	no		File that contains the password for the key.
store-type	no	JKS	Keystore type. Valid values are JKS, JCEKS, PKCS12, PKCS11, DKS, Windows_MY, and BKS.

Example

Following is an example of a tls stanza with the default value for trust-store.

tls:
  key-store:
    store-file: /path/to/store/file
    store-password-file: /path/to/store/password/file
    key-password-file: /path/to/key/password/file
    store-type: JKS
  trust-store: default

(Optional) Configure auth credentials

Use the optional credentials stanza to provide the authentication credentials that you want the connector to use to connect to nodes in the Aerospike database cluster.

Property	Required	Default	Description
username	yes		Username.
password-file	yes		A file from which the password is read. Everything after the first newline is ignored. Trailing spaces in the first line are not ignored.
auth-mode	no	internal	The authentication mode. Valid values are internal, external, external-insecure and pki. internal - user credentials are validated internally by the Aerospike cluster, using a hashed password. external - user credentials are validated externally (e.g. using LDAP) by the Aerospike cluster. TLS is required between the clusters for this mode, as the user password is sent in clear text. external-insecure - user credentials are validated externally (e.g. using LDAP) by the Aerospike cluster. TLS is not required for this mode, but since the user password is sent in clear text, this mode is not recommended for production systems. pki - user credentials are validated from the client TLS certificate. No user name or password needs to be configured. This mode requires that the server be configured with TLS mutual auth and that the client have a valid TLS certificate. Requires Database 5.7 or later.

Example

Following is an example with the credentials stanza. Also included is the example from previous step.

aerospike:
  seeds:
    - 192.168.50.1:
        port: 3000
        tls-name: red
    - 192.168.50.2
  cluster-name: east
  tls:
    key-store:
      store-file: /path/to/store/file
      store-password-file: /path/to/store/password/file
      key-password-file: /path/to/key/password/file
      store-type: JKS
    trust-store:
      store-file: /path/to/store/file
      store-password-file: /path/to/store/password/file
      key-password-file: /path/to/key/password/file
      store-type: JKS
    ciphers:
      - TLS_RSA_WITH_3DES_EDE_CBC_SHA
    revoke-certificates:
      - 12345678
  credentials:
    username: admin
    password-file: /path/to/password/file.txt
    auth-mode: internal

(Optional) Communication settings

Use the optional services stanza to configure optional settings for the connector to use when communicating with nodes of your Aerospike database.

Property	Required	Default	Description
ip-map	no	No translation	If the connector is outside of the network in which your Aerospike database is running, you can use this property to map node IP addresses visible to the connector to destination IP addresses within your network.
use-services-alternate	no	false	Use if alternate-access-address is set in the configuration file for your Aerospike database. See the reference for alternate-access-address for more information about this property.

Example

Following is an example of the services stanza mapping two node IP addresses as exposed outside of an Aerospike database's network to IP addresses for those nodes within the network. Also included is the example from previous step.

aerospike:
  seeds:
    - 192.168.50.1:
        port: 3000
        tls-name: red
    - 192.168.50.2
  cluster-name: east
  tls:
    key-store:
      store-file: /path/to/store/file
      store-password-file: /path/to/store/password/file
      key-password-file: /path/to/key/password/file
      store-type: JKS
    trust-store:
      store-file: /path/to/store/file
      store-password-file: /path/to/store/password/file
      key-password-file: /path/to/key/password/file
      store-type: JKS
    ciphers:
      - TLS_RSA_WITH_3DES_EDE_CBC_SHA
    revoke-certificates:
      - 12345678
  credentials:
    username: admin
    password-file: /path/to/password/file.txt
    auth-mode: internal
  services:
    ip-map:
      192.168.50.1: 192.168.60.1
      192.168.50.2: 192.168.60.2

(Optional) Configure throttle connections

Use the optional performance stanza to throttle the number of connections the sink connector can open per node in your Aerospike database. You can also throttle the number of threads that the connector runs on nodes that it connects to.

Property	Required	Default	Description
max-connections-per-node	no	300	Maximum number of connections allowed per Aerospike database node
event-loop-size	no	Number of processors on a node.	Number of threads that the client launches on a node.

Example

Following is an example of the performance stanza setting a maximum of 310 connections per node and specifying to launch four threads per node. Also included is the example from previous step.

aerospike:
  seeds:
    - 192.168.50.1:
        port: 3000
        tls-name: red
    - 192.168.50.2
  cluster-name: east
  tls:
    key-store:
      store-file: /path/to/store/file
      store-password-file: /path/to/store/password/file
      key-password-file: /path/to/key/password/file
      store-type: JKS
    trust-store:
      store-file: /path/to/store/file
      store-password-file: /path/to/store/password/file
      key-password-file: /path/to/key/password/file
      store-type: JKS
    ciphers:
      - TLS_RSA_WITH_3DES_EDE_CBC_SHA
    revoke-certificates:
      - 12345678
  credentials:
    username: admin
    password-file: /path/to/password/file.txt
    auth-mode: internal
  services:
    ip-map:
      192.168.50.1: 192.168.60.1
      192.168.50.2: 192.168.60.2
  performance:
    max-connections-per-node: 310
    event-loop-size: 4

(Optional) Configure timeouts

Specify timeouts applied to the Aerospike transactions.

Property	Required	Default	Description
socket-timeout	no	30000 (30 seconds)	Socket idle timeout in milliseconds when processing a database command.
total-timeout	no	1000 (1 second)	Total transaction timeout in milliseconds.

Example

Following is an example of the specifying socket-timeout and total-timeout.

aerospike:
  seeds:
    - 192.168.50.1:
        port: 3000
        tls-name: red
    - 192.168.50.2
  socket-timeout: 30000
  total-timeout: 1000

(Optional) Configure boolean particle type

Specify particle types of boolean bins.

• If true, boolean is stored on the Aerospike server with a boolean particle type (introduced in Aerospike Database 5.6).
• If false, boolean is stored on the Aerospike server with an integer particle type (1 or 0). Must be false for server versions earlier than 5.6 which do not support boolean bins.

Property	Required	Default	Description
use-bool-bin	no	false	Set this property to true if you would like boolean bins stored with boolean particle type in the Aerospike server.

Example

Following is an example specifying use-bool-bin.

aerospike:
  seeds:
    - 192.168.50.1:
        port: 3000
        tls-name: red
    - 192.168.50.2
  use-bool-bin: false