Configuring Streaming from Aerospike to JMS

Configuring streaming from Aerospike to JMS involves setting up your Aerospike database to send change notifications and modifying the configuration file for the JMS outbound connector: /etc/aerospike-jms-outbound/aerospike-jms-outbound.yml. "Outbound" means that the connector receives change notifications from Aerospike, translates them into JSON messages, and places the messages on topics or queues in a JMS message broker. This file is located on the system where you installed the connector package.

Setting Up Aerospike Database

In your Aerospike database, you must configure Cross-Datacenter Replication (XDR) and enable change notification. You must also configure your JMS cluster to be a "connector" XDR datacenter and the namespace must point to this as xdr remote datacenter.

See the change notification configuration parameters and example config.

Modifying aerospike-jms-outbound.yml

You configure streaming from Aerospike to JMS by modifying the /etc/aerospike-jms-outbound/aerospike-jms-outbound.yml YAML configuration file.

The configuration file has the following sections:

• service Configures the connector's listening ports, TLS, and network interface.
• jms Configures connection properties for the target JMS message broker.
• bin-transforms Specifies the bin transformations to apply on the Aerospike record.
• format Specifies the message format to use for the outbound messages sent to the message broker.
• batching Specifies how to collect a group of Change Notification Records into a single batch of outbound messages sent to the message broker.
• delivery-mode Chooses between persistent and non-persistent delivery of messages to the message broker.
• routing Configures how incoming record updates/deletes from Aerospike are routed to the message broker.
• namespaces Configures bin transforms, format, routing at namespace and set level.
• logging Configures the destination and level for the connectors logs.

caution

If the route or format is not specified for an Aerospike record, then the record is skipped and logged at the warn level.

Cascading Configuration Values

You can configure the bin-transforms, format, and routing sections to include default values, values for each particular namespace, and values for each particular set within a namespace. For each Aerospike record, the appropriate values are applied.

Here is an annotated example:

routing:
  mode: static
  destination: default <1>
‎‎
namespaces:
  users:
    routing:
      mode: static
      destination: users <2>
    format:
      mode: flat-json
      metadata-key: metadata
    sets:
      premium:
        routing:
          mode: static
          destination: premium <3>

1. Specifies that records in all namespaces are to be shipped to the default topic.
2. Specifies that records in the users namespace are to be shipped to the users topic.
3. Specifies that records in the premium set in the users namespace are to be shipped to the premium topic.

Example

A sample version of /etc/aerospike-jms-outbound/aerospike-jms-outbound.yml for connecting to IBM MQ is shown below:

service:
  port: 8080

logging:
  file: /var/log/aerospike-jms-outbound/aerospike-jms-outbound.log

jms:
  factory: com.ibm.mq.jms.MQConnectionFactory
  spec: 2.0
  config:
    hostName: 10.0.2.5
    port: 1414
    queueManager: QM1
    transportType: 1
    channel: DEV.APP.SVRCONN
  credentials:
    username: admin
    password-file: /path/to/password/file.txt

delivery-mode: persistent

format: json

bin-transforms:
  map:
   yellow: red
  transforms:
   - uppercase

routing:
  mode: bin
  type: queue
  bin: category
  default: test-queue
  transforms:
    - trim
    - regex:
        pattern: '[^A-Za-z0-9]'
        replacement: '-'
    - lowercase

namespaces:
  users:
    routing:
      mode: static
      type: queue
      destination: users
    format:
      mode: flat-json
      metadata-key: metadata
    sets:
      premium:
        routing:
          mode: static
          type: queue
          destination: premium
        bin-transforms:
          map:
            gold: platinum