Filters

This page describes how filters allow you to specify which records will be shipped by XDR. You use Aerospike Expressions to build these filters. Expressions provide a rich interface for expressing complex logic. You can select records based on the metadata of the record as well as the content of the bins. These filters are evaluated when a record is about to be shipped to the destination DC.

A few use cases for these filters:

• GDPR - If a record needs to be shipped based on the user’s profile.
• Conserve network bandwidth - Fine-grained control on which records get shipped.
• Change notification - XDR is also used to notify external systems. Filters can send notifications only if the record meets specific criteria. For example, only if the account balance falls below a threshold.

Remember that expressions only decide if a record will be shipped or not. They do not determine which bins get shipped. The bin-policy still determines which bins get shipped.

Writing the filter

Aerospike Expressions are used to build XDR shipping filters. Get yourself familiar with the Expressions API. Minimally, look at the options available under

• Metadata
• Storage

The following example builds a filter that selects all records with a bin named age where the bin type is integer and the value is >= 21. Each language builds the expression and prints its base64 representation, which you pass to the xdr-set-filter info command.

Java

Expression filter = Exp.build(
    Exp.ge(Exp.intBin("age"), Exp.val(21)));
System.out.println(filter.getBase64());

Python

from aerospike_helpers.expressions import GE, IntBin

filter = GE(IntBin("age"), 21).compile()
b64 = client.get_expression_base64(filter)
print(b64)

Go

filter := as.ExpGreaterEq(as.ExpIntBin("age"), as.ExpIntVal(21))
b64, _ := filter.Base64()
fmt.Println(b64)

C

as_exp_build_b64(filter_b64,
    as_exp_cmp_ge(as_exp_bin_int("age"), as_exp_int(21)));
printf("%s\n", filter_b64);
as_exp_destroy_b64(filter_b64);

C#

Expression filter = Exp.Build(
    Exp.GE(Exp.IntBin("age"), Exp.Val(21)));
Console.WriteLine(filter.GetBase64());

Node.js

const exp = Aerospike.exp;

const filter = exp.ge(exp.binInt('age'), exp.int(21));
const b64 = client.expressionToBase64(filter);
console.log(b64);

Dealing with deletes

If a namespace is configured to ship using XDR, a command which deletes a record leaves a tombstone in place of the original record until XDR successfully ships the record. Filters apply to these tombstones as well. Tombstones have metadata but no storage data (bin-level data). Storage expressions (bin-level) evaluate to false for a tombstone. Metadata checks apply as usual. If the overall expression evaluates to false, the record is not shipped.

In the above example where we check age >= 21, it does not ship record delete operations because the age bin is no longer present and the expression evaluates to false. To ship record deletes as well as records with age >= 21, modify the filter to also check for tombstones:

Java

Expression filter = Exp.build(
    Exp.or(
        Exp.isTombstone(),
        Exp.ge(Exp.intBin("age"), Exp.val(21))));
System.out.println(filter.getBase64());

Python

from aerospike_helpers.expressions import Or, GE, IntBin, IsTombstone

filter = Or(
    IsTombstone(),
    GE(IntBin("age"), 21)
).compile()
b64 = client.get_expression_base64(filter)
print(b64)

Go

filter := as.ExpOr(
    as.ExpIsTombstone(),
    as.ExpGreaterEq(as.ExpIntBin("age"), as.ExpIntVal(21)))
b64, _ := filter.Base64()
fmt.Println(b64)

C

as_exp_build_b64(filter_b64, as_exp_or(
    as_exp_is_tombstone(),
    as_exp_cmp_ge(as_exp_bin_int("age"), as_exp_int(21))));
printf("%s\n", filter_b64);
as_exp_destroy_b64(filter_b64);

C#

Expression filter = Exp.Build(
    Exp.Or(
        Exp.IsTombstone(),
        Exp.GE(Exp.IntBin("age"), Exp.Val(21))));
Console.WriteLine(filter.GetBase64());

Node.js

const exp = Aerospike.exp;

const filter = exp.or(
    exp.isTombstone(),
    exp.ge(exp.binInt('age'), exp.int(21)));
const b64 = client.expressionToBase64(filter);
console.log(b64);

Setting the filter

XDR filters use Aerospike’s SMD infrastructure. The filter needs to be set using an info command on a single node in the cluster and the SMD layer distributes it across the rest of the cluster. Any new node that joins the cluster automatically gets the existing filters using SMD.

Setting the filter using info commands

Filters in XDR should be set for each namespace in a DC using the xdr-set-filter info command. This command expects the filter in base64 representation. The above examples illustrate how to convert the expression into the corresponding base64 representation.

Use the command below to set the filter:

asinfo -v "xdr-set-filter:dc=DC1;namespace=test;exp=kxGRSJMEk1ECo2FnZRU="

Use exp=null to remove an existing filter:

asinfo -v "xdr-set-filter:dc=DC1;namespace=test;exp=null"

Setting the filter using asadm

You can set and view XDR filters from within asadm using the asinfo command in privileged mode. This runs the info command across the cluster.

Admin+> asinfo -v "xdr-set-filter:dc=DC1;namespace=test;exp=kxGRSJMEk1ECo2FnZRU="

To remove an existing filter:

Admin+> asinfo -v "xdr-set-filter:dc=DC1;namespace=test;exp=null"

To view the filters currently set, use show config xdr filter:

Admin> show config xdr filter
~~~~~~~~~~~~~~~~~~~~~~~XDR Filters (2023-02-16 22:55:02 UTC)~~~~~~~~~~~~~~~~~~~~~~~
Namespace|Datacenter|  Base64 Expression|                                Expression
bar      |dc2       |null               |null
test     |dc2       |kxGRSJMEk1ECo2FnZRU|or(is_tombstone(), ge(bin_int("age"), 21))
Number of rows: 2

Setting the filter using client code

Aerospike client libraries provide an API to set the filter directly without generating the base64 output and calling the xdr-set-filter info command manually.

Java

AerospikeClient.setXDRFilter sets the filter in one call. Pass null as the filter to remove it.

import com.aerospike.client.AerospikeClient;
import com.aerospike.client.exp.Exp;
import com.aerospike.client.exp.Expression;

AerospikeClient client = new AerospikeClient("127.0.0.1", 3000);

Expression filter = Exp.build(
    Exp.or(
        Exp.isTombstone(),
        Exp.ge(Exp.intBin("age"), Exp.val(21))));

client.setXDRFilter(null, "DC1", "test", filter);
client.close();

Python

client.set_xdr_filter sets the filter. The expression must be compiled first.

import aerospike
from aerospike_helpers.expressions import Or, GE, IntBin, IsTombstone

client = aerospike.client({"hosts": [("127.0.0.1", 3000)]}).connect()

filter = Or(
    IsTombstone(),
    GE(IntBin("age"), 21)
).compile()

client.set_xdr_filter("DC1", "test", filter)
client.close()

Go

Client.SetXDRFilter sets the filter. Pass nil as the filter to remove it.

import as "github.com/aerospike/aerospike-client-go/v6"

client, _ := as.NewClient("127.0.0.1", 3000)

filter := as.ExpOr(
    as.ExpIsTombstone(),
    as.ExpGreaterEq(as.ExpIntBin("age"), as.ExpIntVal(21)))

client.SetXDRFilter(nil, "DC1", "test", filter)
client.Close()

C

aerospike_set_xdr_filter sets the filter using a base64 string.

#include <aerospike/aerospike.h>
#include <aerospike/as_exp.h>

as_config config;
as_config_init(&config);
as_config_add_hosts(&config, "127.0.0.1", 3000);

aerospike as;
aerospike_init(&as, &config);

as_error err;
aerospike_connect(&as, &err);

as_exp_build_b64(filter_b64, as_exp_or(
    as_exp_is_tombstone(),
    as_exp_cmp_ge(as_exp_bin_int("age"), as_exp_int(21))));

aerospike_set_xdr_filter(&as, &err, NULL, "DC1", "test", filter_b64);
as_exp_destroy_b64(filter_b64);

C#

AerospikeClient.SetXDRFilter sets the filter in one call.

using Aerospike.Client;

AerospikeClient client = new AerospikeClient("127.0.0.1", 3000);

Expression filter = Exp.Build(
    Exp.Or(
        Exp.IsTombstone(),
        Exp.GE(Exp.IntBin("age"), Exp.Val(21))));

client.SetXDRFilter(null, "DC1", "test", filter);
client.Close();

Node.js

client.setXDRFilter sets the filter. Pass the expression array directly.

const Aerospike = require('aerospike');
const exp = Aerospike.exp;

const client = await Aerospike.connect({ hosts: '127.0.0.1:3000' });

const filter = exp.or(
    exp.isTombstone(),
    exp.ge(exp.binInt('age'), exp.int(21)));

await client.setXDRFilter(filter, 'DC1', 'test');
client.close();

Getting the filter

Use the command below to see which filter is set:

asinfo -v "xdr-get-filter:dc=DC1;namespace=test"

output: namespace=test:exp=or(is_tombstone(), ge(bin_int("age"), 21))

Note: In version 5.3.0, the above command used to display the filter in base-64 format.

In version 5.4.0 and above, use the command below to get the filter in base-64 format:

asinfo -v "xdr-get-filter:dc=DC1;namespace=test;b64=true"

output: namespace=test:exp=kxGRSJMEk1ECo2FnZRU=

Examples of filters

Ship records from specific sets

Ship records from sets shipped_set1 and shipped_set2. This filter also ships tombstones from these sets because metadata checks (including set name) apply as usual for tombstones.

Java

Expression filter = Exp.build(
    Exp.or(
        Exp.eq(Exp.setName(), Exp.val("shipped_set1")),
        Exp.eq(Exp.setName(), Exp.val("shipped_set2"))));
System.out.println(filter.getBase64());

Python

from aerospike_helpers.expressions import Or, Eq, SetName

filter = Or(
    Eq(SetName(), "shipped_set1"),
    Eq(SetName(), "shipped_set2")
).compile()

Go

filter := as.ExpOr(
    as.ExpEq(as.ExpSetName(), as.ExpStringVal("shipped_set1")),
    as.ExpEq(as.ExpSetName(), as.ExpStringVal("shipped_set2")))

C

as_exp_build_b64(filter_b64, as_exp_or(
    as_exp_cmp_eq(as_exp_set_name(), as_exp_str("shipped_set1")),
    as_exp_cmp_eq(as_exp_set_name(), as_exp_str("shipped_set2"))));
printf("%s\n", filter_b64);
as_exp_destroy_b64(filter_b64);

C#

Expression filter = Exp.Build(
    Exp.Or(
        Exp.EQ(Exp.SetName(), Exp.Val("shipped_set1")),
        Exp.EQ(Exp.SetName(), Exp.Val("shipped_set2"))));
Console.WriteLine(filter.GetBase64());

Node.js

const exp = Aerospike.exp;

const filter = exp.or(
    exp.eq(exp.setName(), exp.str('shipped_set1')),
    exp.eq(exp.setName(), exp.str('shipped_set2')));

Ship records by bin type

Ship records only if bin1 is an integer bin.

Java

import com.aerospike.client.command.ParticleType;

Expression filter = Exp.build(
    Exp.eq(
        Exp.binType("bin1"),
        Exp.val(ParticleType.INTEGER)));
System.out.println(filter.getBase64());

Python

import aerospike
from aerospike_helpers.expressions import Eq, BinType

filter = Eq(
    BinType("bin1"),
    aerospike.AS_BYTES_INTEGER
).compile()

Go

import ParticleType "github.com/aerospike/aerospike-client-go/v6/types/particle_type"

filter := as.ExpEq(
    as.ExpBinType("bin1"),
    as.ExpIntVal(int64(ParticleType.INTEGER)))

C

as_exp_build_b64(filter_b64,
    as_exp_cmp_eq(as_exp_bin_type("bin1"), as_exp_int(AS_BYTES_INTEGER)));
printf("%s\n", filter_b64);
as_exp_destroy_b64(filter_b64);

C#

using Aerospike.Client;

Expression filter = Exp.Build(
    Exp.EQ(
        Exp.BinType("bin1"),
        Exp.Val((long)ParticleType.INTEGER)));
Console.WriteLine(filter.GetBase64());

Node.js

const exp = Aerospike.exp;

// ParticleType.INTEGER = 1
const filter = exp.eq(exp.binType('bin1'), exp.int(1));

Ship records by bin value

Ship records only if the string bin country equals NL.

Java

Expression filter = Exp.build(
    Exp.eq(Exp.stringBin("country"), Exp.val("NL")));
System.out.println(filter.getBase64());

Python

from aerospike_helpers.expressions import Eq, StrBin

filter = Eq(StrBin("country"), "NL").compile()

Go

filter := as.ExpEq(
    as.ExpStringBin("country"),
    as.ExpStringVal("NL"))

C

as_exp_build_b64(filter_b64,
    as_exp_cmp_eq(as_exp_bin_str("country"), as_exp_str("NL")));
printf("%s\n", filter_b64);
as_exp_destroy_b64(filter_b64);

C#

Expression filter = Exp.Build(
    Exp.EQ(Exp.StringBin("country"), Exp.Val("NL")));
Console.WriteLine(filter.GetBase64());

Node.js

const exp = Aerospike.exp;

const filter = exp.eq(exp.binStr('country'), exp.str('NL'));