Orchestrating Aerospike Clusters with Docker Compose

note

This guide is accurate with the Docker 19.03.2 release.

Overview

This section describes how to set up Docker Compose to orchestrate the deployment and scaling of an Aerospike cluster. Docker Compose is one component in the Docker tool chain that enables multi-container applications to be orchestrated.

All Flash kernel parameters

note

In an All-Flash deployment, the following Linux kernel parameters are required. enforce-best-practices verifies that these kernel parameters have the expected values.

• When running as non-root, you must prepare these values before running the Aerospike server:

/proc/sys/vm/dirty_bytes = 16777216
/proc/sys/vm/dirty_background_bytes = 1
/proc/sys/vm/dirty_expire_centisecs = 1
/proc/sys/vm/dirty_writeback_centisecs = 10

• When running as root, the server configures them automatically. Either way, if these parameters can't be correctly set (manually or automatically by the server), the node will not start.

Define the environment

1. Define the Aerospike service.

...
services:
    aerospikedb:
        image: aerospike/aerospike-server:latest
        networks:
        - aerospikenetwork
        deploy:
            replicas: 2
            endpoint_mode: dnsrr
        labels:
            com.aerospike.cluster: "MYPROJECT"
        command: [ "--config-file","/run/secrets/aerospike.conf"]
        secrets:
        - source: conffile
          target: aerospike.conf
          mode: 0440
...

Key points:

• Aerospike Database version is defined (for example, latest)
• A declaration to use the network aerospikenetwork. This ensures that the discovery process can connect to the Aerospike nodes to perform clustering.
• A deploy parameter that specifies initial cluster size.
• A deploy parameter that specifies the discovery endpoint as dnsrr which is DNS round-robin. This is a Compose 3.2 feature
• A label defines the cluster name (e.g. "com.aerospike.cluster=myproject").
• Append a command to use a custom configuration file.
• A secret is mounted that contains our aerospike.conf configuration file as read-only.

2. Define the meshworker discovery service.

...
meshworker:
        image: aerospike/aerospike-tools:latest
        networks:
        - aerospikenetwork
        depends_on:
        - aerospike
        entrypoint:
        - /run/secrets/discovery
        - "--servicename"
        - aerospikedb
        - "-i"
        - "5"
        - "-v"
        secrets:
        - source: discoveryfile
          target: discovery
          mode: 0750
...

Key points:

• Uses the aerospike/aerospike-tools image to trigger cluster reconfigurations.
• A declaration to use the network aerospikenetwork. This ensures that the discovery process can connect to the Aerospike nodes to perform clustering.
• The discovery script can be found at discovery.py
• Depends_on: this is ignored in swarm mode, but obeyed with single node with compose. Ensures this service starts after the aerospikedb service.
• Overwrite the entrypoint for aerospike/aerospike-tools container to run our discovery script:
  • Passes in various parameters like: the service name to resolve
  • The interval to poll the service name
  • And a verbose flag for debugging/logging
• Mounts the discoveryfile secret as an executable script.

3. Define the network. Starting with Compose v2, overlay networks can be defined as well.

...
networks:
    aerospikenetwork:
        driver: overlay
        attachable: true
...

Key points:

• Uses the overlay driver for this network.
• Turns on the attachable flag for easier debugging. Allows non-compose managed containers to also utilize this network.

4. Define secrets.

Secrets are used instead of volume mounts for two reasons:

1. Secrets are propagated internally by Docker, whereas volume mounts need to exist on the Docker host (or swarm node) in order to be utilized.
2. Long format declaration allows renaming and setting permissions.

...
secrets:
    conffile:
        file: ./aerospike.conf
    discoveryfile:
        file: ./discovery.py

Deploy Aerospike containers

1. Use stack/compose to deploy the Aerospike containers.

docker stack can be used in conjunction with your Compose file, and optionally, your Docker Swarm cluster.

$ docker stack deploy -c aerospike.yml aerospike

2. View the Docker container running Aerospike.

$ docker ps
docker ps
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS              PORTS               NAMES
360e1da2d43f        aerospike/aerospike-tools:latest    "/run/secrets/disc..."   3 hours ago         Up 3 hours                              aerospike_meshworker.1.zvs6cfhz0z8cebej5tdg4qdou
4d8fc0bd20f8        aerospike/aerospike-server:latest   "/entrypoint.sh --..."   3 hours ago         Up 3 hours          3000-3002/tcp       aerospike_aerospikedb.3.v7edr5cvnnxvtep81beam9lqh
...

3. View your service status.

docker service ls
ID                  NAME                    MODE                REPLICAS            IMAGE                               PORTS
rfee38b7u9mt        aerospike_meshworker    replicated          1/1                 aerospike/aerospike-tools:latest
ws864n7ivhy2        aerospike_aerospikedb   replicated          2/2                 aerospike/aerospike-server:latest

note

The name of the container consists of:

• Stack name by the docker stack deploy command, e.g. "aerospike"
• Service name from the yaml file e.g. "aerospike"
• Numeric ID indicates how many of the service has been started, e.g. "1"
• Task ID - the internal ID for every docker action

Run Aerospike Enterprise Edition

info

As of Database 6.1, a simple feature-key file is included. This feature-key file only allows deployment of a single-node cluster.

The discovery.py, aerospike_ee.yml and aerospike_ee.conf for EE edition usage can be found here.

1. Clone the repo aerospike/aerospike-docker-swarm

   $ git clone https://github.com/aerospike/aerospike-docker-swarm
   $ cd aerospike-docker-swarm

2. Copy the features.conf file to the current path, the same path as aerospike_ee.yml, aerospike_ee.conf and discovery.py files. :::note aerospike_ee.conf contains feature-key-file configuration under service stanza and it points to path /run/secrets/features.conf. The path /run/secrets/ stores all the secrets defined in aerospike_ee.yml.

   services:
     aerospikedb:
         image: aerospike/aerospike-server-enterprise:latest
         networks:
         - aerospikenetwork
         deploy:
             replicas: 3
             endpoint_mode: dnsrr
         labels:
             com.aerospike.cluster: "myproject"
         command: [ "--config-file","/run/secrets/aerospike_ee.conf"]
         secrets:
         - source: conffile
           target: aerospike_ee.conf
           mode: 0440
         - source: featurekeyfile
           target: features.conf
           mode: 0440
   ....
   ....
   secrets:
     conffile:
         file: ./aerospike_ee.conf
     discoveryfile:
         file: ./discovery.py
     featurekeyfile:
         file: ./features.conf

3. Deploy the stack.

   $ docker stack deploy -c aerospike_ee.yml aerospike

Use Service to scale the Aerospike cluster

Docker services can be scaled dynamically using docker service scale.

1. Verify that two Aerospike services (containers) are running.

$ docker service scale aerospike_aerospikedb=2

note

At this point, the discovery container should see the updated hostnames and automatically cluster the containers.

2. View the Aerospike cluster topology.

$ docker run --net aerospike_aerospikenetwork aerospike/aerospike-tools asadm -h aerospikedb -e info
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                                                             Node               Node              Ip      Build   Cluster        Cluster     Cluster         Principal   Client     Uptime
                                                                                .                 Id               .          .      Size            Key   Integrity                 .    Conns          .
aerospike_aerospikedb.1.o0eujlsuyc8bylgzxt3svr61b.aerospike_aerospikenetwork:3000   *BB90301000A4202   10.0.1.3:3000   C-3.14.0         2   41339CCF5C67   True        BB90301000A4202        5   02:50:39
aerospike_aerospikedb.3.v7edr5cvnnxvtep81beam9lqh.aerospike_aerospikenetwork:3000   BB90201000A4202    10.0.1.2:3000   C-3.14.0         2   41339CCF5C67   True        BB90301000A4202        5   02:50:39
Number of rows: 2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace                                                                                Node   Avail%   Evictions                 Master                Replica     Repl     Stop     Pending       Disk    Disk     HWM        Mem     Mem    HWM      Stop
        .                                                                                   .        .           .   (Objects,Tombstones)   (Objects,Tombstones)   Factor   Writes    Migrates       Used   Used%   Disk%       Used   Used%   Mem%   Writes%
        .                                                                                   .        .           .                      .                      .        .        .   (tx%,rx%)          .       .       .          .       .      .         .
test        aerospike_aerospikedb.1.o0eujlsuyc8bylgzxt3svr61b.aerospike_aerospikenetwork:3000   99         0.000     (0.000  ,0.000  )      (0.000  ,0.000  )      2        false    (0,0)       0.000 B    0       50      0.000 B    0       60     90
test        aerospike_aerospikedb.3.v7edr5cvnnxvtep81beam9lqh.aerospike_aerospikenetwork:3000   99         0.000     (0.000  ,0.000  )      (0.000  ,0.000  )      2        false    (0,0)       0.000 B    0       50      0.000 B    0       60     90
test                                                                                                       0.000     (0.000  ,0.000  )      (0.000  ,0.000  )                        (0,0)       0.000 B                    0.000 B
Number of rows: 3

Establish an Aerospike mesh cluster dynamically

If the discovery container was not used, you would need to manually mesh the aerospike containers.

While both Containers are now on the same network, the Docker Overlay network does not support multi-cast.

Add each node into the Aerospike cluster with the tip command.

$ docker run --net aerospike_aerospikenetwork aerospike/aerospike-tools asinfo -v "tip:host=$(docker inspect -f '{{.NetworkSettings.Networks.prod.IPAddress }}' aerospike.2.nl30psiorabhcu2rj1kpghq6c );port=3002" -h aerospike.1.i6httgeicb9ix0vdevw8ye7xt 

Additional information

• For namespaces using index-type flash Aerospike container must run in privileged mode.
• Swarm mode doesn't support running containers in privileged mode.
• See the Aerospike Docker Swarm GitHub repo.