Aerospike Lua Client

Problem

You are using Lua and would like to make calls to an Aerospike database, but Aerospike does not provide a Lua Client API.

You may be using Lua within application servers like Nginx, BarracudaDrive, Mako, OpenResty, Kepler, etc – similar to how Javascript is used inside Node.

Solution

The solution is conceptually simple: call the Aerospike C API from Lua. This example demonstrates how to wrap the Aerospike C client API so it can be called from Lua.

Getting the code

The source code for this example is available on GitHub: http://github.com/aerospike/client-lua.

Clone the GitHub repository using the following command:

git clone https://github.com/aerospike/client-lua.git

Getting the C Client

The example also requires the Aerospike C client, you should have one available. If you don’t, follow the instructions here.

Build Instructions

Run the build script located in the root directory of the project to build the library as_lua.so

For Linux

./build_linux.sh

For OS X

./build_osx.sh

The shared library “as_lua.so” has dependencies on these libraries:

  • aerospike
  • ssl
  • crypto
  • pthread
  • rt
  • lua
  • m

If you have installed the Aerospike C client correctly, these dependencies will also be installed correctly.

The shared library “as_lua.so” should be placed in the Lua library path, in one of these locations:

./as_lua.so

/usr/local/lib/lua/5.1/as_lua.so

/usr/lib/lua/5.1/as_lua.so
“`

Running a Lua application to calling Aerospike

This example includes a simple Lua program that exercises each function implemented in the library. You will find code in the file “main.lua” in the “test” subdirectory.

You’ll need the Lua development environment, and thus lua.h in the include path. You can include the Lua 5.0 or 5.1 environment from source, or install a Lua development package with your package manager.

Discussion

In this example we will wrap 5 basic Aerospike key value operations:

  • Connect – connect to an Aerospike cluster
  • Get – read a record from the Aerospike cluster
  • Put – write a record to the Aerospike cluster
  • Increment – increment a bin value in a record
  • Disconnect – disconnect from the Aerospike cluster

You can use this example to wrap other Aerospike operations in the Aerospike C library.

There is a lot of information at Lua users wiki that describes calling C from Lua. Chapter 24 of “Programming in Lua” describes in detail how this is done.

The functions that will be exposed to Lua need to be defined in the following code:

static const struct luaL_Reg as_client [] = {
  {"connect", connect},
  {"disconnect", disconnect},
  {"get", get},
  {"put", put},
  {"increment", increment},
  {NULL, NULL}
};

extern int luaopen_as_lua(lua_State *L){
 luaL_register(L, "as_lua", as_client);
 return 0;
}

This function is called by the require statement in Lua. When require is called Lua will look for a library named as_lua.so on its library path. Be sure to put as_lua.so in one of the folowing directories, depending on your Lua installation:

  • ./as_lua.so
  • /usr/local/lib/lua/5.1/as_lua.so
  • /usr/lib/lua/5.1/as_lua.so

The C function lua_open_aerospike is called by the Lua function require as_lua. At the start of you Lua application you should have code like this:

local as = require "as_lua"

Connect

To connect to an Aerospike cluster you need to supply one or more “seed nodes” and ports. When a client application connects to a cluster, one name or IP address and port is sufficient. The Aerospike intelligent client will discover all the nodes in the cluster and become a 1st class observer of them. Additional nodes can be supplied at connection time in case the node you have specified is actually down.

Our Lua function will take one seed node address and port, and return a handle to a cluster.

err, message, cluster = as.connect("localhost", 3000)
print("connected", err, message)

Connect must be called before you call any other Aerospike function and only needs to be called once at the beginning of you Lua process.

Connect will return 3 values, in the above example, the variables err and message will contain the error code and error message, and the variable cluster will be a handle to the Aerospike cluster. You will use cluster handle in subsequent Aerospike function calls.

The C code to implement this function:

static int connect(lua_State *L){
 const char *hostName = luaL_checkstring(L, 1);
 int port = lua_tointeger(L, 2);
 as_error err;

 // Configuration for the client.
 as_config config;
 as_config_init(&config);

 // Add a seed host for cluster discovery.
 config.hosts[0].addr = hostName;
 config.hosts[0].port = port;

 // The Aerospike client instance, initialized with the configuration.
 aerospike as;
 aerospike_init(&as, &config);

 // Connect to the cluster.
 aerospike_connect(&as, &err);

 /* Push the return */
 lua_pushnumber(L, err.code);
 lua_pushstring(L, err.message);
 lua_pushlightuserdata(L, &as);
 return 3;
}

Lua uses its own stack mechanism to pass parameters between Lua and C. You will note at the start of this function parameters are taken from the stack

const char *hostName = luaL_checkstring(L, 1);
int port = lua_tointeger(L, 2);

and at the end of the function the return values are pushed onto the stack

lua_pushnumber(L, err.code);
lua_pushstring(L, err.message);
lua_pushlightuserdata(L, &as);
return 3;

Note that there are 3 return values:

  • error number – this is the C Client API error number
  • error message – this is the C Client API error message
  • cluster pointer

Error numbers and messages are documented here, you can also find them in as_status.h

Disconnect

When your Lua application has completed using Aerospike, usually at the end of the Lua code, it should disconnect from the Aerospike cluster. This frees resources in the process, things like socket connections, file descriptors, etc. It should be called once at the end of your Lua process.

Our Lua function to disconnect from the cluster will take one parameter that is the cluster handle.

err, message = as.disconnect(cluster)
print("disconnected", err, message)

The C code to implement this function:

static int disconnect(lua_State *L){
 aerospike* as = lua_touserdata(L, 1);
 as_error err;
 aerospike_close(as, &err);
 lua_pushnumber(L, err.code);
 lua_pushstring(L, err.message);
 return 2;
}

You should be seeing a pattern emerging in the way the parameters are passed to the function and how values are returned. The actual code that interacts with Aerospike is trivial. Most of the work in this function is to obtain the pointer to the cluster and return 2 values to Lua

Get

To read a record from Aerospike you use the get function. The cluster handle, obtained from the connect function, is the first argument and is the reference the get function uses to interact with the cluster. You can call this function asn many times as you like.

The next 3 parameters are namespace, set and key, in that order. These values uniquely identify the record.

err, message, record = as.get(cluster, "test", "test", "peter001")
print("read record", err, message, record.uid, record.name, record.dob)

The get function returns 3 values. The 1st is the error code, the second is the error message if one exists, the 3rd is a table containing the record values.

This function reads all the Bins in the record and is the easiest to implement. An extension to this sample would offer the capability to read a subset of bins.

Put

To write data to Aerospike use the put function. The cluster handle, obtained from the connect function, is the first argument and is the reference the put function uses to interact with the cluster, and it can be called as many time as you like.

The next 3 parameters are namespace, set and key, in that order. These values uniquely identify the record.

The 5th parameter is the number of values in the 6th parameters which is a table containing the Bin names and values. This is the actual data you want to store in Aerospike. These will be overwrite values with the same name, and be added to existing records.

local bins = {}
bins["uid"] = "peter001"
bins["name"] = "Peter"
bins["dob"] = 19800101

err, message = as.put(cluster, "test", "test", "peter001", 3, bins)
print("saved record", err, message)

The put function returns 2 values. The 1st is the error code, the second is the error message if one exists.

This function illustrates the complexity of passing parameters and return values between Lua and C. The complexity is in obtaining the values in table parameter. The function add_bins_to_rec takes the passed-in table, containing the bin names and values, and creates bin values and populates a as_record structure. This structure is returned to the caller.

The tricky part is obtaining the values from the 5th parameter, which is a Lua table. The code manipulates the Lua stack to iterate through the elements in the table and creates Bin values.

static as_record add_bins_to_rec(lua_State *L, int index, int numBins)
{
      as_record rec;
      as_record_init(&rec, numBins);

   // Push another reference to the table on top of the stack (so we know
   // where it is, and this function can work for negative, positive and
   // pseudo indices
   lua_pushvalue(L, index);
   // stack now contains: -1 => table
   lua_pushnil(L);
   // stack now contains: -1 => nil; -2 => table
   while (lua_next(L, -2))
   {
       // stack now contains: -1 => value; -2 => key; -3 => table
       // copy the key so that lua_tostring does not modify the original
       lua_pushvalue(L, -2);
       // stack now contains: -1 => key; -2 => value; -3 => key; -4 => table
       const char *key = lua_tostring(L, -1);
       // add to record
       if (lua_isnumber(L, -2)){
        int intValue = lua_tointeger(L, -2);
        as_record_set_int64(&rec, key, intValue);
       } else if (lua_isstring(L, -2)){
        const char *value = lua_tostring(L, -2);
        as_record_set_str(&rec, key, value);
       }
       // pop value + copy of key, leaving original key
       lua_pop(L, 2);
       // stack now contains: -1 => key; -2 => table
   }

   // stack now contains: -1 => table (when lua_next returns 0 it pops the key
   // but does not push anything.)
   // Pop table
   lua_pop(L, 1);
   // Stack is now the same as it was on entry to this function
   return rec;
}

Increment

The increment function is quite similar to the put function in that it changes the value of one or more Bins in a record. It is useful when your application uses counters, and it also removes the need to read the value, increment the value in your application and rewrite it.

The cluster handle, obtained from the connect function, is the first argument and is the reference the increment function uses to interact with the cluster. You can call the increment function as many times as you like.

The next 3 parameters are namespace, set and key, in that order. These values uniquely identify the record.

The 5th parameters is the number of bin names in the 6th parameter which is a table containing the Bin names and increment values.

bins = {}
bins["counter"] = 1
err, message = as.increment(cluster, "test", "test", "peter003", 1, bins)
print("incremented record", err, message)