Skip to main content
Loading

Configuring the Properties of Connections from the Connector to an Aerospike Database

The aerospike section of the aerospike-kafka-inbound.yml file configures the connection properties that the connector must use when connecting to your Aerospike database.

Carry out these steps in the aerospike section of the /etc/aerospike-kafka-inbound/aerospike-kafka-inbound.yml file:

List the seed nodesโ€‹

Use the required seeds stanza to list the nodes in the Aerospike database cluster that you want the connector to connect to. By connecting to a seed node, the connector can discover all of the nodes in the cluster. The connector iterates through the list of nodes until it successfully connects to one of them, then it discovers the other nodes in the cluster.

The connector is an Aerospike Smart Client, and stores in memory the partition maps of each node in the cluster, so that it can send updates directly to the appropriate nodes. For more information about Aerospike Smart Clients, see Smart Client.

PropertyRequiredDefaultDescription
portno3000The port to use when making connections to the Aerospike database.
tls-namenoThe TLS name of the Aerospike database.

Exampleโ€‹

Following is an example with the seeds stanza at the top of the file:

aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
socket-timeout: 30000
total-timeout: 1000
use-bool-bin: false

Specify cluster nameโ€‹

If the database cluster is configured with a name, specify the name.

Use the optional cluster-name property to specify the name of the Aerospike database that you are streaming data to. The connector can run for a long time, during which nodes of the cluster can be removed from the cluster permanently or for maintenance. The IP addresses of those nodes could be reassigned to nodes that are in other Aerospike clusters or in non-Aerospike clusters. By specifying the name of the Aerospike cluster that you are streaming data to, you ensure that the connector always communicates only with nodes that are part of that cluster.

The value must match the value that is set by the configuration property cluster-name in the configuration file for the Aerospike database. For information about setting this property, see cluster-name in "Configuration parameter reference"

Exampleโ€‹

Following is an example with the cluster-name property appearing after the seeds stanza:

aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
cluster-name: east
socket-timeout: 30000
total-timeout: 1000
use-bool-bin: false

(Optional) Specify rack IDโ€‹

If the Aerospike cluster is organized into racks, under certain circumstances it may make sense to make use of Aerospike's rack awareness feature.

Aerospike's rack awareness feature allows reads from partitions in the same logical rack to be prioritised, reducing both latency and network use. Network use reduction can save significant money in a cloud environment. If the Kafka Connector itself has a rack id specified then it will read from partitions in the same rack, even if the local partition is a replica partition. This is achieved using the smart client mechanism - the Aerospike client has detailed data about the placement of all partitions in a cluster.

For the Kafka inbound connector this will usually not make a difference as writes will always be to the master partition. If a custom transformation is made use of however, then if reads are made by the custom transformer they will be made using the rack awareness feature. Under these circumstances, taking advantage of rack awareness could be beneficial for the reasons given above.

Exampleโ€‹

Following is an example with the rack-id property set. If any reads are made, they will be made from partitions in rack 1. Also included is an example of use of tls-name from the previous step:

aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
cluster-name: east
rack-id: 1

(Optional) TLSโ€‹

You can use the tls stanza to secure connections from the connector to your Aerospike database with Transport Layer Security (TLS).

This stanza is more complex than the seeds stanza in the first step. Following is an example up front.

aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
cluster-name: east
tls:
key-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
trust-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
ciphers:
- TLS_RSA_WITH_3DES_EDE_CBC_SHA
revoke-certificates:
- 12345678

Following are descriptions of the main properties in the tls stanza:

PropertyRequiredDefaultDescription
key-storeyesThe key store containing the Aerospike client certificate for mutual authentication. See "Configuring key-store and trust-store".
trust-storenoDefault Java trust store.The trust store containing trusted CA certificate for Aerospike database certificate. See "Configuring key-store and trust-store".
ciphersnodefault ciphers allowed by the JVMAllowed list of TLS ciphers that clients can use for secure connections.
revoke-certificatesnoList of certificate serial numbers to reject.

Configuring key-store and trust-storeโ€‹

key-store takes these properties. trust-store can either take these properties or its default value.

PropertyRequiredDefaultDescription
store-fileyesStore file
store-password-fileyesFile that contains the password to the store.
key-password-filenoFile that contains the password for the key.
store-typenoJKSKeystore type. Valid values are JKS, JCEKS, PKCS12, PKCS11, DKS, Windows_MY, and BKS.

Exampleโ€‹

Following is an example of a tls stanza with the default value for trust-store.

tls:
key-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
trust-store: default

(Optional) Auth credentialsโ€‹

Use the optional credentials stanza to provide the authentication credentials that you want the connector to use to connect to nodes in the Aerospike database cluster.

PropertyRequiredDefaultDescription
usernameyesUsername.
password-fileyesA file from which the password is read. Everything after the first newline is ignored. Trailing spaces in the first line are not ignored.
auth-modenointernalThe authentication mode. Valid values are internal, external, external-insecure and pki.
  • internal - user credentials are validated internally by the Aerospike cluster, using a hashed password.
  • external - user credentials are validated externally (e.g. using LDAP) by the Aerospike cluster. TLS is required between the clusters for this mode, as the user password is sent in clear text.
  • external-insecure - user credentials are validated externally (e.g. using LDAP) by the Aerospike cluster. TLS is not required for this mode, but since the user password is sent in clear text, this mode is not recommended for production systems.
  • pki - user credentials are validated from the client TLS certificate. No user name or password needs to be configured. This mode requires that the server be configured with TLS mutual auth and that the client have a valid TLS certificate. Requires Database 5.7 or later.

Exampleโ€‹

Following is an example with the credentials stanza. Also included is the example from previous step.

aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
cluster-name: east
tls:
key-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
trust-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
ciphers:
- TLS_RSA_WITH_3DES_EDE_CBC_SHA
revoke-certificates:
- 12345678
credentials:
username: admin
password-file: /path/to/password/file.txt
auth-mode: internal

(Optional) Communication settingsโ€‹

Use the optional services stanza to configure optional settings for the connector to use when communicating with nodes of your Aerospike database.

PropertyRequiredDefaultDescription
ip-mapnoNo translationIf the connector is outside of the network in which your Aerospike database is running, you can use this property to map node IP addresses visible to the connector to destination IP addresses within your network.
use-services-alternatenofalseUse if alternate-access-address is set in the configuration file for your Aerospike database. See the reference for alternate-access-address for more information about this property.

Exampleโ€‹

Following is an example of the services stanza mapping two node IP addresses as exposed outside of an Aerospike database's network to IP addresses for those nodes within the network. Also included is the example from previous step.

aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
cluster-name: east
tls:
key-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
trust-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
ciphers:
- TLS_RSA_WITH_3DES_EDE_CBC_SHA
revoke-certificates:
- 12345678
credentials:
username: admin
password-file: /path/to/password/file.txt
auth-mode: internal
services:
ip-map:
192.168.50.1: 192.168.60.1
192.168.50.2: 192.168.60.2

(Optional) Throttle connectionsโ€‹

Use the optional performance stanza to throttle the number of connections the sink connector can open per node in your Aerospike database. You can also throttle the number of threads that the connector runs on nodes that it connects to.

PropertyRequiredDefaultDescription
max-connections-per-nodeno300Maximum number of connections allowed per Aerospike database node
event-loop-sizenoNumber of processors on a node.Number of threads that the client launches on a node.

Exampleโ€‹

Following is an example of the performance stanza setting a maximum of 310 connections per node and specifying to launch four threads per node. Also included is the example from previous step.

aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
cluster-name: east
tls:
key-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
trust-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
ciphers:
- TLS_RSA_WITH_3DES_EDE_CBC_SHA
revoke-certificates:
- 12345678
credentials:
username: admin
password-file: /path/to/password/file.txt
auth-mode: internal
services:
ip-map:
192.168.50.1: 192.168.60.1
192.168.50.2: 192.168.60.2
performance:
max-connections-per-node: 310
event-loop-size: 4

(Optional) Timeoutsโ€‹

Specify timeouts applied to the Aerospike transactions.

PropertyRequiredDefaultDescription
socket-timeoutno30000 (30 seconds)Socket idle timeout in milliseconds when processing a database command.
total-timeoutno1000 (1 second)Total transaction timeout in milliseconds.

Exampleโ€‹

Following is an example of the specifying socket-timeout and total-timeout.

aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
socket-timeout: 30000
total-timeout: 1000

(Optional) Boolean particle typeโ€‹

Specify particle types of boolean bins.

  • If true, boolean is stored on the Aerospike server with a boolean particle type (introduced in Aerospike Database 5.6).
  • If false, boolean is stored on the Aerospike server with an integer particle type (1 or 0). Must be false for server versions earlier than 5.6 which do not support boolean bins.
PropertyRequiredDefaultDescription
use-bool-binnofalseSet this property to true if you would like boolean bins stored with boolean particle type in the Aerospike server.

Exampleโ€‹

Following is an example specifying use-bool-bin.

aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
use-bool-bin: false