Skip to content
Visit booth 3171 at Google Cloud Next to see how to unlock real-time decisions at scaleMore info

Specify JMS queues and topics

Overview

This page describes how to configure the queues and topics sections of the /etc/aerospike-jms-inbound/aerospike-jms-inbound.yml file. These sections specify the properties of the JMS queues and topics from which you want the JMS inbound connector to consume messages.

Example

Following is an example of a topics section that configures one topic. The name of the topic is employees. A queues section includes the same properties.

topics:
  employees:
    session-ratio: 1
    jms-transaction-size: 16
    jms-message-receive-timeout: 100
    aerospike-operation:
      type: write
      max-retries: 2
      total-timeout: 0
      record-exists-action: update
      send-key: false
    mapping:
      bins:
        all-value-fields: false
        type: multi-bins
        map:
          binOne:
            source: value-field
            field-name: field-one
      key-field:
        source: value-field
        field-name: employeeID
      namespace:
        mode: static
        value: east
      set:
        mode: dynamic
        source: value-field
        field-name: subLocation
      ttl:
        mode: dynamic
        source: value-field
        field-name: ttl
    parsing: json

This table shows how, using this example configuration, a JMS inbound connector maps JSON messages from the JMS message broker to Aerospike records:

JSONAerospike NamespaceAerospike SetAerospike KeyAerospike BinTTL
{"field-one":"one","subLocation":"BNG", "employeeID": E123, ttl: "100H"}eastBNGE123Bin(“binOne”, one)100H
{"field-one":["one"],"subLocation":"MYS", "employeeID": E456, ttl: "10M"}eastMYSE456Bin(“binOne”, List(“one”))10M
{"field-one":{"one": "one"},"subLocation":"DLH", "employeeID": E789, ttl: "1D"}eastDLHE789Bin(“binOne”, Map(“one” to “one”))1D
{"field-one":"one","subLocation":"MUM", "employeeID": E987, ttl: "100"}eastMUME987Bin(“binOne”, one)100

Procedure

Listing queues and topics to configure

The configuration file must include at least one of these sections:

PropertyRequiredDescription
queuesnoA list of the queues to consume messages from.
topicsnoA list of the topics to consume messages from.

The first level in a topics or queues section is the name of a topic or queue. For example, suppose you want to configure three topics: employees, offices, and buildings. Your topics section would, in outline, look like this:

topics:
  employees:
    ...
  offices:
    ...
  buildings:
    ...

Configuring queues and topics

For each topic or a queue that you name in their corresponding section, you can set the following properties:

OptionRequiredDefaultDescription
session-rationo1The ratio of total JMS sessions to use for this source. See Distributing JMS sessions across queues and topics for details.
jms-transaction-sizeno16The number of JMS messages to consume in a single JMS transaction.
jms-message-receive-timeoutno100The number of milliseconds to wait for a JMS message before timing out. A value of zero signifies that there is no timeout.
aerospike-operationnowriteSpecifies how to apply writes and deletes to your Aerospike database. See Configuring Aerospike operations to perform on records for details.
mappingin some casesThe mapping of the JMS message to an Aerospike record. It is required if you do not implement message-transformer for the topic/queue or implement a message-transformer with InboundMessage<K, M>. See Map JMS messages to Aerospike records for details.
parsingyesThe JMS message parsing config. See Specifying the format of inbound messages for details.
message-transformernoThe configuration parameters for a message transformer. See Aerospike JMS Inbound Message Transformer for details.

Distributing JMS sessions across queues and topics

You can set this property for each topic or queue that you specify in the queues and topics sections. However, to set it for an individual topic or queue, you must first know the values for all of the queues and topics that you plan to configure in /etc/aerospike-jms-inbound/aerospike-jms-inbound.yml.

To calculate the values of this property for all the queues and topics that you want to use:

  1. Taking the total number of JMS sessions as a whole (1), represent the whole as the fraction n/n, where n represents the number of equal divisions of the whole that you plan to distribute among your queues and topics.

    The value of n must be greater than or equal to the number of your queues and topics, and less than or equal to the number of JMS sessions. The number of JMS sessions is the product of two values that you set in your JMS message broker: max-connections and max-sessions-per-connection. For example, if you set max-connections to 4 and max-sessions-per-connection to 2, then the number of JMS sessions is 8. Therefore, if the number of your topics or queues is 4, n can be greater than or equal to 4, and less than or equal to 8.

    For example, suppose that you plan to distribute sessions across two topics and one queue. You smallest n you could use would be 3 because you have three sources. The fractional representation of the whole would be 3/3. (You don’t have to use a value for n that equals the number of your queues and topics, however.)

  2. Split this fraction across your queues and topics.

    For example, you have two topics (T1 and T2) and one queue (Q1) and the fraction is 3/3. Each source must be allocated some portion of the sessions. So, you give each source 1/3.

  3. Take the numerator of each fraction.

  4. Specify the numerators as the values of session-ratio for your queues and topics.

    To continue with the example from step 2, this table shows the value of session-ratio for each source:

    SourceTypesession-ratio
    T1Topic1
    T2Topic1
    Q1Queue1

    You might choose to distribute the sessions unequally across your queues and topics. For example, if you have topics T1 and T2, and queue Q1, you could use a value of 5 for n, making 5/5, and then set these values for session-ratio:

    SourceTypesession-ratio
    T1Topic2
    T2Topic2
    Q1Queue1

    In this case, topics T1 and T2 would each be allocated two-fifths of the sessions, while topic T3 would be allocated one-fifth.

    If the number of JMS sessions is not evenly divisible by n, then the remaining sessions are distributed randomly to your queues and topics at startup.

Configure Aerospike operations

Use the aerospike-operation option to specify how to apply writes and deletes to your Aerospike database.

OptionRequiredDefaultDescription
typenowriteSpecifies the type of command to perform in Aerospike. Valid values are: write, delete.
max-retriesno2Specifies the maximum number of times to retry a transaction.
total-timeoutno0Specifies the number of milliseconds to wait before a transaction time. A value of 0 indicates that there is no time limit.
record-exists-actionnoupdateSpecifies the action to take if a record already exists and type is set to write. Valid values are update, update-only, replace, replace-only, create-only.
send-keynofalseSpecifies to send a user-defined key, in addition to a hash digest. Applies only when the value for the property type is write.
durable-deletenofalseSpecifies to leave a tombstone for the record. Applies only when the value for the property type is delete.
ignore-error-codesnoempty setConsider a record as successfully processed if it throws AerospikeException with the given ResultCode.

Examples 

# An aerospike write operation with all relevant options.
aerospike-operation:
  type: write
  max-retries: 3
  total-timeout: 0
  record-exists-action: update-only
  send-key: true
  ignore-error-codes:
    - 22
    - 13
# An aerospike delete operation with all relevant options.
aerospike-operation:
  type: delete
  max-retries: 3
  total-timeout: 0
  durable-delete: true
  ignore-error-codes:
    - 22
    - 13

Map JMS messages to Aerospike records

Use the mapping option to specify how to map JMS messages in a topic or queue to Aerospike records.

OptionRequiredDescription
namespaceyesSpecifies whether the name of the namespace is set in the configuration file or in messages. See Specify namespace, set, and ttl for details.
setnoSpecifies whether the name of the set is set in the configuration file or in messages. See Specify namespace, set, and ttl for details.
ttlnoSpecifies whether the time to live is set in the configuration file or in messages. See Specify namespace, set, and ttl for details.
binsyesMapping of incoming message to Aerospike bins. See Mapping fields in messages to bins in Aerospike records for details.
key-fieldno*The Field selection configuration value for the Aerospike record key. See Key-field and Digest config for details.
digestno*The Field selection configuration value for the Aerospike record digest. The field has to be a valid Aerospike RIPEMD-160 digest. The digest field should be plain bytes or a Base64 encoded string. See Key-field and Digest config for details.

Key-field and Digest config

Use these two options to generate key-field or digest.

OptionRequiredDescription
sourceyesThe source of the value.
field-namenoName of the field to extract the value from. Only applies when source is value-field.

Possible values for the source option

ValueDescription
keyUse the whole of the Pulsar record key as the value.
valueUse the whole of the Pulsar record value as the value.
value-fieldExtract a value from the Pulsar record value.

Specify namespace, set, and ttl

Configuration to obtain values for namespace, set, and ttl requires definitions for the following properties:

OptionRequiredDescription
modeyesHow the value of a given field should be obtained. The possible options are static and dynamic.

Static: Specify the values in the configuration file

To set a static or unchanging name for the namespace, name for the set, or value for the time to live for an Aerospike record, you can use this option:

OptionRequiredDescription
valueyesThe static name or value.

This example sets the name of a namespace to east for all records that an inbound connector creates from the messages that it receives:

mapping:
  ...
  namespace:
    mode: static
    value: east
  ...

This example sets the time to live for all records that an inbound connector creates from the messages that it receives:

mapping:
  ...
  ttl:
    mode: static
    value: -1
  ...

Choose from among the following values when setting the time to live for Aerospike records:

ValueDescription
-2Specifies not to reset the time to live when a record is updated.
-1Specifies that the time to live never expires.
0Specifies to set the time to live to the default that is configured for the namespace in the Aerospike database.
Integer greater than 0Specifies by default the time to live in seconds. However, you can use one of the following suffixes to specify a different unit of time: M for minutes, H for hours, D for days.

Dynamic: Specify message fields from which to obtain values

Select a value dynamically from the JMS message.

OptionRequiredDescription
sourceyesThe source of the value.
field-namenoName of the field to extract the value from. Only applies when source is value-field.

Possible values for the source option

ValueDescription
keyUse the whole of the Pulsar record key as the value.
valueUse the whole of the Pulsar record value as the value.
value-fieldExtract a value from the Pulsar record value.

Example 

mapping:
  ...
  namespace:
    mode: dynamic
    source: value-field
    field-name: category
  set:
    mode: dynamic
    source: value-field
    field-name: brand
  ttl:
    mode: dynamic
    source: value-field
    field-name: validForSec
  ...

Mapping fields in messages to bins in Aerospike records

For topics and queues that you list in their corresponding section, you must map fields from messages to bins in Aerospike records.

OptionRequiredDescription
typeyesNumber of bins that exist in Aerospike records that correspond to messages in the topic/queue. Valid values are single-bin, multi-bins.

Single bin

Use these two options to specify the source of the value to write to single-bin Aerospike records.

OptionRequiredDescription
sourceyesThe source of the value.
field-namenoName of the field to extract the value from. Only applies when source is value-field.

Possible values for the source option

ValueDescription
keyUse the whole of the Pulsar record key as the value.
valueUse the whole of the Pulsar record value as the value.
value-fieldExtract a value from the Pulsar record value.

Examples 

# A single bin Aerospike record with the bin value extracted from the JMS
# record field "name".
bins:
  type: single-bin
  source: value-field
  field-name: name
# A single bin Aerospike record with the bin value equal to the whole of the
# JMS message value.
bins:
  type: single-bin
  source: value

Multiple bins

Use these options to specify the source of the values to write to Aerospike records that are comprised of multiple bins.

OptionRequiredDefaultDescription
all-value-fieldsnofalseIndicates if all fields from the JMS message value should be converted to Aerospike record bins.
mapnoMap of the Aerospike record bin name to the source of the bin value. Use this option only when all-value-fields is false.
sourceyesSource of the value. See the table below for the possible values. Used with the map option
field-namenoName of the field to extract the value from. Only applies when source is value-field. Used with the map option.

Possible values for the source option

ValueDescription
keyUse the whole of the Pulsar record key as the value.
valueUse the whole of the Pulsar record value as the value.
value-fieldExtract a value from the Pulsar record value.

Examples 

# Write all the JMS message value fields as Aerospike record bins.
bins:
  type: multi-bins
  all-value-fields: true
# Extract the values of Aerospike bins "name" and "city" from the JMS fields `firstName` and `residence`.
bins:
  type: multi-bins
  map:
    name:
      source: value-field
      field-name: firstName
    city:
      source: value-field
      field-name: residence
# Use the whole of the JMS message value as the Aerospike bin "metadata".
bins:
  type: multi-bins
  map:
    metadata:
      source: value

Specifying the format of inbound messages

The message parsing config. Valid values -

ValueDescription
jsonSpecifies that the message is a JMS TextMessage in JSON format.
msgpackSpecifies that the message is a JMS BytesMessage in MsgPack format.
mapSpecifies that the message is a JMS MapMessage.
java-serializedSpecifies that the message is a JMS ObjectMessage in Java-serialized object format. The serialized object should be in the class path. The JAR containing the class file and its dependencies should be put in “/opt/aerospike-jms-inbound/usr-lib`.
Feedback

Was this page helpful?

What type of feedback are you giving?

What would you like us to know?

+Capture screenshot

Can we reach out to you?