Deploying the Pulsar Inbound Connector

The Aerospike Pulsar connector implements the standard Sink interface of the Pulsar IO framework, and therefore runs as a Pulsar Function.

Prerequisites

Set up your Pulsar Functions workers in any way that meets your requirements. The Pulsar Functions documentation for configuring workers is here. The rest of this documentation refers to locations where the Aerospike Pulsar connector runs as "Pulsar Functions worker nodes".

Ensure that the feature pulsar-connector is enabled in the Aerospike feature-key file features.conf.

Ensure that the systems on which you plan to install instances of the Aerospike Pulsar inbound connector are running version 2.7.1 or later of Apache Pulsar.

Install Java 8 or later, if it is not already installed.

RHEL or CentOS

sudo yum install java-1.8.0-openjdk

Debian or Ubuntu

sudo apt-get install openjdk-8-jre

If authentication is enabled for your Pulsar message brokers, enable security for your Functions workers. The instructions in the Pulsar documentation are here.

Procedure

To set up and start using Aerospike Connect for Pulsar:

Place Aerospike Connect for Pulsar in the Pulsar connector directory.

You must copy the connector to every Pulsar Functions worker node.

Download the Aerospike Pulsar Inbound .nar file, and put the .nar file in the connector directory of Pulsar, usually located inside /usr/local/pulsar.

tip

By default, Pulsar extracts connectors into /tmp directory. You might want to change this default; otherwise, the connector will be deleted from a node whenever the node is rebooted. You can change the default extraction directory by changing the value of the configuration property narExtractionDirectory in functions_worker.yaml.

Create the configuration file for the connector.

Create the aerospike-pulsar-inbound.yml configuration file for the Aerospike Pulsar inbound connector on one of the nodes of your Pulsar cluster or Pulsar Function cluster. The location of the file does not matter. Specify the path to the file when you create and start the connector. Refer to Configuring the Pulsar Inbound Connector for more details.

Create and start the connector by running the /bin/pulsar-admin sinks create command.

As shown here, the pulsar-admin command includes only the flags that you need to specify when creating and starting the Aerospike Pulsar inbound connector.

You can run the command either from a Pulsar Functions worker node or, if you include the optional flag --admin-url, from a separate system on your network. The command shown here can be run from a Pulsar Functions worker node:

<pulsar-dir>/bin/pulsar-admin sinks create \
    --tenant <value> \
    --namespace <value> \
    --name "aerospike-pulsar-sink" \
    --sink-type aerospike \
    --inputs <value> \
    --sink-config-file <value> \
    --parallelism <value>

• <pulsar-dir>: The directory where Pulsar is installed.
• --tenant: The Pulsar tenant in which the connector is installed.
• --namespace: The Pulsar namespace (within the tenant) in which the connector is installed.
• --inputs: The Pulsar topic or topics to use as input to the connector. You can specify multiple topics as a comma-separated list. All topics that you list here must be listed in the aerospike-pulsar-inbound.yml configuration file.
• --sink-config-file: The path to a aerospike-pulsar-inbound.yml configuration file. If you are running the command on a system outside of the cluster, you must specify the path to a copy of the configuration file on that system.* --parallelism: The number of connector instances to run. For example, if you set the value to 3, three instances of the connector run in the cluster.
• --admin-url: The Pulsar Admin URL (webServiceUrl): Include this optional flag if you want to run the command outside of your cluster on a system that is in your network.

What to do next

To find out whether the connector is making updates, inserts, and deletes in your Aerospike database, you can browse your data in Aerospike by using any of three browsing tools. See "Data Browsing" for details.