Skip to main content
We are excited to be a part of AWS re:Invent 2024. Visit us at booth #1844 in Las Vegas.More info
Loading

Specifying the JMS Queues and Topics to Consume from

Overview​

This page describes how to configure the queues and topics sections of the /etc/aerospike-jms-inbound/aerospike-jms-inbound.yml file. These sections specify the properties of the JMS queues and topics from which you want the JMS inbound connector to consume messages.

Example​

Following is an example of a topics section that configures one topic. The name of the topic is employees. A queues section includes the same properties.

topics:
  employees:
    session-ratio: 1
    jms-transaction-size: 16
    jms-message-receive-timeout: 100
    aerospike-operation:
      type: write
      max-retries: 2
      total-timeout: 0
      record-exists-action: update
      send-key: false
    mapping:
      bins:
        all-value-fields: false
        type: multi-bins
        map:
          binOne:
            source: value-field
            field-name: field-one
      key-field:
        source: value-field
        field-name: employeeID
      namespace:
        mode: static
        value: east
      set:
        mode: dynamic
        source: value-field
        field-name: subLocation
      ttl:
        mode: dynamic
        source: value-field
        field-name: ttl
    parsing: json

This table shows how, using this example configuration, a JMS inbound connector maps JSON messages from the JMS message broker to Aerospike records:

JSONAerospike NamespaceAerospike SetAerospike KeyAerospike BinTTL
{"field-one":"one","subLocation":"BNG", "employeeID": E123, ttl: "100H"}eastBNGE123Bin("binOne", one)100H
{"field-one":["one"],"subLocation":"MYS", "employeeID": E456, ttl: "10M"}eastMYSE456Bin("binOne", List("one"))10M
{"field-one":{"one": "one"},"subLocation":"DLH", "employeeID": E789, ttl: "1D"}eastDLHE789Bin("binOne", Map("one" to "one"))1D
{"field-one":"one","subLocation":"MUM", "employeeID": E987, ttl: "100"}eastMUME987Bin("binOne", one)100

Procedure​

Listing queues and topics to configure​

The configuration file must include at least one of these sections:

PropertyRequiredDescription
queuesnoA list of the queues to consume messages from.
topicsnoA list of the topics to consume messages from.

The first level in a topics or queues section is the name of a topic or queue. For example, suppose you want to configure three topics: employees, offices, and buildings. Your topics section would, in outline, look like this:

topics:
  employees:
    ...
  offices:
    ...
  buildings:
    ...

Configuring queues and topics​

For each topic or a queue that you name in their corresponding section, you can set the following properties:

OptionRequiredDefaultDescription
session-rationo1The ratio of total JMS sessions to use for this source. See "Distributing JMS sessions across queues and topics" for details.
jms-transaction-sizeno16The number of JMS messages to consume in a single JMS transaction.
jms-message-receive-timeoutno100The number of milliseconds to wait for a JMS message before timing out. A value of zero signifies that there is no timeout.
aerospike-operationnowriteSpecifies how to apply writes and deletes to your Aerospike database. See "Configuring Aerospike operations to perform on records" for details.
mappingin some casesThe mapping of the JMS message to an Aerospike record. It is required if you do not implement message-transformer for the topic/queue or implement a message-transformer with InboundMessage<K, M>. See "Configuring how to map JMS messages to Aerospike records" for details.
parsingyesThe JMS message parsing config. See "Specifying the format of inbound messages" for details.
message-transformernoThe configuration parameters for a message transformer. See "Aerospike JMS Inbound Message Transformer" for details.

Distributing JMS sessions across queues and topics​

You can set this property for each topic or queue that you specify in the queues and topics sections. However, to set it for an individual topic or queue, you must first know the values for all of the queues and topics that you plan to configure in /etc/aerospike-jms-inbound/aerospike-jms-inbound.yml.

To calculate the values of this property for all the queues and topics that you want to use:

  1. Taking the total number of JMS sessions as a whole (1), represent the whole as the fraction n/n, where n represents the number of equal divisions of the whole that you plan to distribute among your queues and topics.

    The value of n must be greater than or equal to the number of your queues and topics, and less than or equal to the number of JMS sessions. The number of JMS sessions is the product of two values that you set in your JMS message broker: max-connections and max-sessions-per-connection. For example, if you set max-connections to 4 and max-sessions-per-connection to 2, then the number of JMS sessions is 8. Therefore, if the number of your topics or queues is 4, n can be greater than or equal to 4, and less than or equal to 8.

    For example, suppose that you plan to distribute sessions across two topics and one queue. You smallest n you could use would be 3 because you have three sources. The fractional representation of the whole would be 3/3. (You don't have to use a value for n that equals the number of your queues and topics, however.)

  2. Split this fraction across your queues and topics.

    For example, you have two topics (T1 and T2) and one queue (Q1) and the fraction is 3/3. Each source must be allocated some portion of the sessions. So, you give each source 1/3.

  3. Take the numerator of each fraction.

  4. Specify the numerators as the values of session-ratio for your queues and topics.

To continue with the example from step 2, this table shows the value of session-ratio for each source:

SourceTypesession-ratio
T1Topic1
T2Topic1
Q1Queue1

You might choose to distribute the sessions unequally across your queues and topics. For example, if you have topics T1 and T2, and queue Q1, you could use a value of 5 for n, making 5/5, and then set these values for session-ratio:

SourceTypesession-ratio
T1Topic2
T2Topic2
Q1Queue1

In this case, topics T1 and T2 would each be allocated two-fifths of the sessions, while topic T3 would be allocated one-fifth.

If the number of JMS sessions is not evenly divisible by n, then the remaining sessions are distributed randomly to your queues and topics at startup.

info

If you set session-ratio for one of the queues and topics that you plan to use, then you must set it for all of those queues and topics.

Configure Aerospike operations​

Use the aerospike-operation option to specify how to apply writes and deletes to your Aerospike database.

OptionRequiredDefaultDescription
typenowriteSpecifies the type of operation to perform in Aerospike. Valid values are: write, delete.
max-retriesno2Specifies the maximum number of times to retry a transaction.
total-timeoutno0Specifies the number of milliseconds to wait before a transaction time. A value of 0 indicates that there is no time limit.
record-exists-actionnoupdateSpecifies the action to take if a record already exists and type is set to write. Valid values are update, update-only, replace, replace-only, create-only.
send-keynofalseSpecifies to send a user-defined key, in addition to a hash digest. Applies only when the value for the property type is write.
durable-deletenofalseSpecifies to leave a tombstone for the record. Applies only when the value for the property type is delete.
ignore-error-codesnoempty setConsider a record as successfully processed if it throws AerospikeException with the given ResultCode.

Examples​

# An aerospike write operation with all relevant options.
aerospike-operation:
  type: write
  max-retries: 3
  total-timeout: 0
  record-exists-action: update-only
  send-key: true
  ignore-error-codes:
    - 22
    - 13
# An aerospike delete operation with all relevant options.
aerospike-operation:
  type: delete
  max-retries: 3
  total-timeout: 0
  durable-delete: true
  ignore-error-codes:
    - 22
    - 13

Map JMS messages to Aerospike records​

Use the mapping option to specify how to map JMS messages in a topic or queue to Aerospike records.

OptionRequiredDescription
namespaceyesSpecifies whether the name of the namespace is set in the configuration file or in messages. See Specify namespace, set, and ttl for details.
setnoSpecifies whether the name of the set is set in the configuration file or in messages. See Specify namespace, set, and ttl for details.
ttlnoSpecifies whether the time to live is set in the configuration file or in messages. See Specify namespace, set, and ttl for details.
binsyesMapping of incoming message to Aerospike bins. See Mapping fields in messages to bins in Aerospike records for details.
key-fieldno*The Field selection configuration value for the Aerospike record key. See Key-field and Digest config for details.
digestno*The Field selection configuration value for the Aerospike record digest. The field has to be a valid Aerospike RIPEMD-160 digest. The digest field should be plain bytes or a Base64 encoded string. See Key-field and Digest config for details.
caution

Either key-field or digest has to be specified in a mapping.

Key-field and Digest config​

Use these two options to generate key-field or digest.

OptionRequiredDescription
sourceyesSource of the value.
field-namenoName of the field to extract the value from. Only applies when source is value-field.

Possible values for the source option​

ValueDescription
valueUse the whole of the JMS message value as the value.
value-fieldExtract a value from the JMS message value.

Specify namespace, set, and ttl​

Configuration to obtain values for namespace, set, and ttl requires definitions for the following properties:

OptionRequiredDescription
modeyesHow the value of a given field should be obtained. The possible options are static and dynamic.

Static: Specify the values in the configuration file​

To set a static or unchanging name for the namespace, name for the set, or value for the time to live for an Aerospike record, you can use this option:

OptionRequiredDescription
valueyesThe static name or value.

This example sets the name of a namespace to east for all records that an inbound connector creates from the messages that it receives:

mapping:
  ...
  namespace:
    mode: static
    value: east
  ...

This example sets the time to live for all records that an inbound connector creates from the messages that it receives:

mapping:
  ...
  ttl:
    mode: static
    value: -1
  ...

Choose from among the following values when setting the time to live for Aerospike records:

ValueDescription
-2Specifies not to reset the time to live when a record is updated.
-1Specifies that the time to live never expires.
0Specifies to set the time to live to the default that is configured for the namespace in the Aerospike database.
Integer greater than 0Specifies by default the time to live in seconds. However, you can use one of the following suffixes to specify a different unit of time: M for minutes, H for hours, D for days.

Dynamic: Specify message fields from which to obtain values​

Select a value dynamically from the JMS message.

OptionRequiredDescription
sourceyesSource of the value.
field-namenoName of the field to extract the value from. Only applies when source is value-field.

Possible values for the source option​

ValueDescription
valueUse the whole of the JMS message value as the value.
value-fieldExtract a value from the JMS message value.

Example​

mapping:
  ...
  namespace:
    mode: dynamic
    source: value-field
    field-name: category
  set:
    mode: dynamic
    source: value-field
    field-name: brand
  ttl:
    mode: dynamic
    source: value-field
    field-name: validForSec
  ...

Mapping fields in messages to bins in Aerospike records​

For topics and queues that you list in their corresponding section, you must map fields from messages to bins in Aerospike records.

OptionRequiredDescription
typeyesNumber of bins that exist in Aerospike records that correspond to messages in the topic/queue. Valid values are single-bin, multi-bins.

Single bin​

Use these two options to specify the source of the value to write to single-bin Aerospike records.

OptionRequiredDescription
sourceyesSource of the value.
field-namenoName of the field to extract the value from. Only applies when source is value-field.

Possible values for the source option​

ValueDescription
valueUse the whole of the JMS message value as the value.
value-fieldExtract a value from the JMS message value.

Examples​

# A single bin Aerospike record with the bin value extracted from the JMS
# record field "name".
bins:
  type: single-bin
  source: value-field
  field-name: name
# A single bin Aerospike record with the bin value equal to the whole of the
# JMS message value.
bins:
  type: single-bin
  source: value

Multiple bins​

Use these options to specify the source of the values to write to Aerospike records that are comprised of multiple bins.

OptionRequiredDefaultDescription
all-value-fieldsnofalseIndicates if all fields from the JMS message value should be converted to Aerospike record bins.
mapnoMap of the Aerospike record bin name to the source of the bin value. Use this option only when all-value-fields is false.
sourceyesSource of the value. See the table below for the possible values. Used with the map option
field-namenoName of the field to extract the value from. Only applies when source is value-field. Used with the map option.

Possible values for the source option​

ValueDescription
valueUse the whole of the JMS message value as the value.
value-fieldExtract a value from the JMS message value.

Examples​

# Write all the JMS message value fields as Aerospike record bins.
bins:
  type: multi-bins
  all-value-fields: true
# Extract the values of Aerospike bins "name" and "city" from the JMS fields `firstName` and `residence`.
bins:
  type: multi-bins
  map:
    name:
      source: value-field
      field-name: firstName
    city:
      source: value-field
      field-name: residence
# Use the whole of the JMS message value as the Aerospike bin "metadata".
bins:
  type: multi-bins
  map:
    metadata:
      source: value

Specifying the format of inbound messages​

The message parsing config. Valid values -

ValueDescription
jsonSpecifies that the message is a JMS TextMessage in JSON format.
msgpackSpecifies that the message is a JMS BytesMessage in MsgPack format.
mapSpecifies that the message is a JMS MapMessage.
java-serializedSpecifies that the message is a JMS ObjectMessage in Java-serialized object format. The serialized object should be in the class path. The JAR containing the class file and its dependencies should be put in `/opt/aerospike-jms-inbound/usr-lib.
Last updated on