Skip to main content
We are excited to be a part of AWS re:Invent 2024. Visit us at booth #1844 in Las Vegas.More info
Loading

Filters

Overview​

This page describes how filters allow users to specify which records will be shipped by XDR. Users need to use the Aerospike Expressions to build these filters. It provides a very rich interface in which users can express very complex logic. Users can select records based on the metadata of the record as well as content of the bins. These filters will be evaluated when the record is about to be shipped to the destination DC.

A few use cases in which these filters can be used are :

  • GDPR - If a record needs to be shipped based on the user’s profile.
  • Conserve network bandwidth - These filters allow a fine-grained control on which records will get shipped.
  • Change notification - XDR is also used to notify external systems. These filters can be used to send the notifications only if the record meets specific criteria. For e.g., only if the account balance falls below a threshold.

Remember that the expressions are only meant to decide if a record will be shipped or not. It does not determine which bins get shipped based on the expression. The bin-policy still determines which bins get shipped.

Writing the filter​

As mentioned above, Aerospike Expressions are used to express these filters. Get yourself familiar with the Expressions API. Minimally, look at the options available under

The following is an example of a filter written in C language. It selects all the records with a bin named 'age' and the bin-type is an integer and the integer value is >= 21.

#include <stdio.h>
#include <citrusleaf/alloc.h>
#include <aerospike/as_exp.h>
int main()
{
        as_exp_build_b64(exp_b64,
                  as_exp_cmp_ge(as_exp_bin_int("age"), as_exp_int(21)));
        printf("%s\n", exp_b64);
        cf_free(exp_b64);
}

The following is the same example in Java

import java.util.Base64;
import com.aerospike.client.exp.Exp;
import com.aerospike.client.exp.Expression;
public class ExpressionTest {
    public static void main(String args[]) {
        Expression filter = Exp.build(Exp.ge(Exp.intBin("age"), Exp.val(21)));
        System.out.println(filter.getBase64());
    }
}

Dealing with deletes​

If a namespace is configured to ship using XDR, an operation which deletes a record will leave a tombstone in place of the original record (until XDR successfully ships the record). The filters apply to these tombstones as well in case of XDR shipping. Tombstones will have metadata but no storage data (bin-level data). An important thing to note is that the storage expressions (bin-level) will evaluate to false for a tombstone. Metadata checks will apply as usual. If the overall expression evaluates to false, the record will not get shipped.

In the above example where we are checking age >= 21, it will not ship record delete operations as the “age” bin is no longer present and the expression will evaluate to false. To ship record deletes as well as records with age >= 21, the above code should be modified as below.

#include <stdio.h>
#include <citrusleaf/alloc.h>
#include <aerospike/as_exp.h>

int main()
{
        as_exp_build_b64(exp_b64, as_exp_or(
                  as_exp_is_tombstone(),
                  as_exp_cmp_ge(as_exp_bin_int("age"), as_exp_int(21))));
        printf("%s\n", exp_b64);

        cf_free(exp_b64);
}

Setting the filter​

XDR filters use Aerospike's SMD infrastructure. The filter needs to be set using an info command on a single node in the cluster and the SMD layer is distributed across the rest of the cluster. Any new node that joins the cluster automatically gets the existing filters using SMD.

Filters in XDR should be set for each namespace in a DC using the xdr-set-filter info command. This command expects the filter in base64 representation. The above examples illustrate how to convert the expression into the corresponding base64 representation.

Use the command below to set the filter:

asinfo -v "xdr-set-filter:dc=DC1;namespace=test;exp=kxGRSJMEk1ECo2FnZRU="

Use 'exp=null' as below to remove an existing filter:

asinfo -v "xdr-set-filter:dc=DC1;namespace=test;exp=null"

Setting the filter using client code​

The latest C and Java client libraries provide an API to set the filter directly without needing to generate the base-64 output and setting it using the xdr-set-filter info command.

  • In C-client the API is aerospike_set_xdr_filter
  • In Java-client the API is setXDRFilter in AerospikeClient class.

Sample code to set the filter using C-client code will be as below:

#include <stdio.h>
#include <citrusleaf/alloc.h>
#include <aerospike/aerospike.h>
#include <aerospike/as_exp.h>
#include <aerospike/as_error.h>
#include <aerospike/as_config.h>

int main()
{
        as_config config;
        as_config_init(&config);
        if (!as_config_add_hosts(&config, "127.0.0.1", 3000)) {
                printf("Invalid host");
                exit(-1);
        }

        aerospike as;
        aerospike_init(&as, &config);

        as_error err;
        if (aerospike_connect(&as, &err) != AEROSPIKE_OK) {
                printf("Failed to connect : %d - %s", err.code, err.message);
                exit(-1);
        }

        as_exp_build_b64(exp_b64, as_exp_or(
                as_exp_is_tombstone(),
                as_exp_cmp_ge(as_exp_bin_int("age"), as_exp_int(21))));

        if (aerospike_set_xdr_filter(&as, &err, NULL, "DC1", "test", exp_b64) != AEROSPIKE_OK) {
                printf("Failed to set filter : %d - %s", err.code, err.message);
        }

        cf_free(exp_b64);

        return 0;
}

The following is the same example in Java

import com.aerospike.client.AerospikeClient;
import com.aerospike.client.AerospikeException;
import com.aerospike.client.command.ParticleType;
import com.aerospike.client.exp.Exp;
import com.aerospike.client.exp.Expression;

public class ExpressionTest {

    public static void main(String args[]) throws AerospikeException {

        AerospikeClient client = new AerospikeClient("127.0.0.1", 3000);

        Expression filter = Exp.build(Exp.or(Exp.isTombstone(), Exp.ge(Exp.intBin("age"), Exp.val(21))));

        try {
            client.setXDRFilter(null, "DC1", "test", filter);
        } catch (AerospikeException e) {
            System.out.println("Failed to set filter " + e.getMessage());
        }

        client.close();
    }
}

Getting the filter​

Use the command below to see which filter is set:

asinfo -v "xdr-get-filter:dc=DC1;namespace=test"

output: namespace=test:exp=or(is_tombstone(), ge(bin_int("age"), 21))

Note: In version 5.3, the above command used to display the filter in base-64 format.

In version 5.4 and above, use the command below to get the filter in base-64 format:

asinfo -v "xdr-get-filter:dc=DC1;namespace=test;b64=true"

output: namespace=test:exp=kxGRSJMEk1ECo2FnZRU=

Examples of filters​

  • Ships records from sets 'shipped_set1' and 'shipped_set2'. This filter also ships tombstones from these sets as metadata checks apply as usual for tombstones.

    as_exp_build_b64(exp_b64, as_exp_or(
          as_exp_cmp_eq(as_exp_set_name(), as_exp_str("shipped_set1")),
          as_exp_cmp_eq(as_exp_set_name(), as_exp_str("shipped_set2"))));
    printf("%s\n", exp_b64);
    cf_free(exp_b64);

  • Ships records only if 'bin1' is an integer bin.

    as_exp_build_b64(exp_b64,
          as_exp_cmp_ne(as_exp_bin_type("bin1"), as_exp_int(AS_BYTES_INTEGER)));
    printf("%s\n", exp_b64);
    cf_free(exp_b64);

  • Ships records only if 'country' is 'NL'

    as_exp_build_b64(exp4_b64
          as_exp_cmp_eq(as_exp_bin_str("country"), as_exp_str("NL")));
    printf("%s\n", exp_b64);
    cf_free(exp_b64);