Configuring the Properties of Connections from the Connector to an Aerospike Database
The aerospike
section of the aerospike-kafka-inbound.yml
file configures the connection properties that the connector must use when connecting to your Aerospike database.
Carry out these steps in the aerospike
section of the /etc/aerospike-kafka-inbound/aerospike-kafka-inbound.yml
file:
List the seed nodesโ
Use the required seeds
stanza to list the nodes in the Aerospike database cluster that you want the connector to connect to. By connecting to a seed node, the connector can discover all of the nodes in the cluster. The connector iterates through the list of nodes until it successfully connects to one of them, then it discovers the other nodes in the cluster.
The connector is an Aerospike Smart Client, and stores in memory the partition maps of each node in the cluster, so that it can send updates directly to the appropriate nodes. For more information about Aerospike Smart Clients, see Smart Client.
Property | Required | Default | Description |
---|---|---|---|
port | no | 3000 | The port to use when making connections to the Aerospike database. |
tls-name | no | The TLS name of the Aerospike database. |
Exampleโ
Following is an example with the seeds
stanza at the top of the file:
aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
socket-timeout: 30000
total-timeout: 1000
use-bool-bin: false
Specify cluster nameโ
If the database cluster is configured with a name, specify the name.
Use the optional cluster-name
property to specify the name of the Aerospike database that you are streaming data to. The connector can run for a long time, during which nodes of the cluster can be removed from the cluster permanently or for maintenance. The IP addresses of those nodes could be reassigned to nodes that are in other Aerospike clusters or in non-Aerospike clusters. By specifying the name of the Aerospike cluster that you are streaming data to, you ensure that the connector always communicates only with nodes that are part of that cluster.
The value must match the value that is set by the configuration property cluster-name
in the configuration file for the Aerospike database. For information about setting this property, see cluster-name
in "Configuration parameter reference"
Exampleโ
Following is an example with the cluster-name
property appearing after the seeds
stanza:
aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
cluster-name: east
socket-timeout: 30000
total-timeout: 1000
use-bool-bin: false
(Optional) Specify rack IDโ
If the Aerospike cluster is organized into racks, under certain circumstances it may make sense to make use of Aerospike's rack awareness
feature.
Aerospike's rack awareness feature allows reads from partitions in the same logical rack to be prioritised, reducing both latency and network use. Network use reduction can save significant money in a cloud environment. If the Kafka Connector itself has a rack id specified then it will read from partitions in the same rack, even if the local partition is a replica partition. This is achieved using the smart client mechanism - the Aerospike client has detailed data about the placement of all partitions in a cluster.
For the Kafka inbound connector this will usually not make a difference as writes will always be to the master partition. If a custom transformation
is made use of however, then if reads are made by the custom transformer they will be made using the rack awareness feature. Under these circumstances, taking advantage of rack awareness could be beneficial for the reasons given above.
Exampleโ
Following is an example with the rack-id
property set. If any reads are made, they will be made from partitions in rack 1. Also included is an example of use of tls-name from the previous step:
aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
cluster-name: east
rack-id: 1
(Optional) TLSโ
You can use the tls
stanza to secure connections from the connector to your Aerospike database with Transport Layer Security (TLS).
This stanza is more complex than the seeds
stanza in the first step. Following is an example up front.
aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
cluster-name: east
tls:
key-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
trust-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
ciphers:
- TLS_RSA_WITH_3DES_EDE_CBC_SHA
revoke-certificates:
- 12345678
Following are descriptions of the main properties in the tls
stanza:
Property | Required | Default | Description |
---|---|---|---|
key-store | yes | The key store containing the Aerospike client certificate for mutual authentication. See "Configuring key-store and trust-store ". | |
trust-store | no | Default Java trust store. | The trust store containing trusted CA certificate for Aerospike database certificate. See "Configuring key-store and trust-store ". |
ciphers | no | default ciphers allowed by the JVM | Allowed list of TLS ciphers that clients can use for secure connections. |
revoke-certificates | no | List of certificate serial numbers to reject. |
Configuring key-store
and trust-store
โ
key-store
takes these properties. trust-store
can either take these properties or its default value.
Property | Required | Default | Description |
---|---|---|---|
store-file | yes | Store file | |
store-password-file | yes | File that contains the password to the store. | |
key-password-file | no | File that contains the password for the key. | |
store-type | no | JKS | Keystore type. Valid values are JKS , JCEKS , PKCS12 , PKCS11 , DKS , Windows_MY , and BKS . |
Exampleโ
Following is an example of a tls
stanza with the default value for trust-store
.
tls:
key-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
trust-store: default
(Optional) Auth credentialsโ
Use the optional credentials
stanza to provide the authentication credentials that you want the connector to use to connect to nodes in the Aerospike database cluster.
Property | Required | Default | Description |
---|---|---|---|
username | yes | Username. | |
password-file | yes | A file from which the password is read. Everything after the first newline is ignored. Trailing spaces in the first line are not ignored. | |
auth-mode | no | internal | The authentication mode. Valid values are internal , external , external-insecure and pki .
|
Exampleโ
Following is an example with the credentials
stanza. Also included is the example from previous step.
aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
cluster-name: east
tls:
key-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
trust-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
ciphers:
- TLS_RSA_WITH_3DES_EDE_CBC_SHA
revoke-certificates:
- 12345678
credentials:
username: admin
password-file: /path/to/password/file.txt
auth-mode: internal
(Optional) Communication settingsโ
Use the optional services
stanza to configure optional settings for the connector to use when communicating with nodes of your Aerospike database.
Property | Required | Default | Description |
---|---|---|---|
ip-map | no | No translation | If the connector is outside of the network in which your Aerospike database is running, you can use this property to map node IP addresses visible to the connector to destination IP addresses within your network. |
use-services-alternate | no | false | Use if alternate-access-address is set in the configuration file for your Aerospike database. See the reference for alternate-access-address for more information about this property. |
Exampleโ
Following is an example of the services
stanza mapping two node IP addresses as exposed outside of an Aerospike database's network to IP addresses for those nodes within the network. Also included is the example from previous step.
aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
cluster-name: east
tls:
key-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
trust-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
ciphers:
- TLS_RSA_WITH_3DES_EDE_CBC_SHA
revoke-certificates:
- 12345678
credentials:
username: admin
password-file: /path/to/password/file.txt
auth-mode: internal
services:
ip-map:
192.168.50.1: 192.168.60.1
192.168.50.2: 192.168.60.2
(Optional) Throttle connectionsโ
Use the optional performance
stanza to throttle the number of connections the sink connector can open per node in your Aerospike database. You can also throttle the number of threads that the connector runs on nodes that it connects to.
Property | Required | Default | Description |
---|---|---|---|
max-connections-per-node | no | 300 | Maximum number of connections allowed per Aerospike database node |
event-loop-size | no | Number of processors on a node. | Number of threads that the client launches on a node. |
Exampleโ
Following is an example of the performance
stanza setting a maximum of 310 connections per node and specifying to launch four threads per node. Also included is the example from previous step.
aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
cluster-name: east
tls:
key-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
trust-store:
store-file: /path/to/store/file
store-password-file: /path/to/store/password/file
key-password-file: /path/to/key/password/file
store-type: JKS
ciphers:
- TLS_RSA_WITH_3DES_EDE_CBC_SHA
revoke-certificates:
- 12345678
credentials:
username: admin
password-file: /path/to/password/file.txt
auth-mode: internal
services:
ip-map:
192.168.50.1: 192.168.60.1
192.168.50.2: 192.168.60.2
performance:
max-connections-per-node: 310
event-loop-size: 4
(Optional) Timeoutsโ
Specify timeouts applied to the Aerospike transactions.
Property | Required | Default | Description |
---|---|---|---|
socket-timeout | no | 30000 (30 seconds) | Socket idle timeout in milliseconds when processing a database command. |
total-timeout | no | 1000 (1 second) | Total transaction timeout in milliseconds. |
Exampleโ
Following is an example of the specifying socket-timeout
and total-timeout
.
aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
socket-timeout: 30000
total-timeout: 1000
(Optional) Boolean particle typeโ
Specify particle types of boolean bins.
- If
true
, boolean is stored on the Aerospike server with a boolean particle type (introduced in Aerospike Database 5.6). - If
false
, boolean is stored on the Aerospike server with an integer particle type (1 or 0). Must be false for server versions earlier than 5.6 which do not support boolean bins.
Property | Required | Default | Description |
---|---|---|---|
use-bool-bin | no | false | Set this property to true if you would like boolean bins stored with boolean particle type in the Aerospike server. |
Exampleโ
Following is an example specifying use-bool-bin
.
aerospike:
seeds:
- 192.168.50.1:
port: 3000
tls-name: red
- 192.168.50.2
use-bool-bin: false