Scan Operations

Description

Aerospike Scan Operations provide the ability to scan all record of a namespace and set in an Aerospike database.

Usage

Before you can execute a scan, you first need to define a scan using as_scan. See as_scan for details on defining scans.

Once you have a scan defined, then you can execute the scan using either:

• aerospike_scan_foreach() — Execute a scan on the database, then process the results.
• aerospike_scan_background() — Send a scan to the database, and not wait for completed. The scan is given an id, which can be used to query the scan status.

When aerospike_scan_foreach() is executed, it will process the results and create records on the stack. Because the records are on the stack, they will only be available within the context of the callback function.

When aerospike_scan_background() is executed, the client will not wait for results from the database. Instead, the client will be given a scan_id, which can be used to query the scan status on the database via aerospike_scan_info().

Walk-through

First, we build a scan using as_scan. The scan will be on the "test" namespace and "demo" set. We will select only bins "a" and "b" to be returned for each record.

as_scan scan;
as_scan_init(&scan, "test", "demo");
 
as_scan_select_inita(&scan, 2);
as_scan_select(&scan, "a");
as_scan_select(&scan, "B");

Now that we have a scan defined, we want to execute it using aerospike_scan_foreach().

if (aerospike_scan_foreach(&as, &err, NULL, &scan, callback, NULL) != AEROSPIKE_OK) {
    printf("error(%d) %s at [%s:%d]", err.code, err.message, err.file, err.line);
}

The callback provided to the function above is implemented as:

bool callback(const as_val* val, void* udata)
{
    if (!val) {
        return false; // Scan complete.
    }
 
    as_record* rec = as_record_fromval(val);
    // Process record
    // Do not call as_record_destroy() because the calling function will do that for you.
    return true;
}

When you are finished with the scan, you should destroy the resources allocated to it:

as_scan_destroy(&scan);

Collaboration diagram for Scan Operations:

Data Structures
struct	as_scan

Typedefs
typedef bool(*	aerospike_scan_foreach_callback) (const as_val *val, void *udata)
typedef bool(*	as_async_scan_listener) (as_error *err, as_record *record, void *udata, as_event_loop *event_loop)

Functions
AS_EXTERN as_status	aerospike_scan_async (aerospike *as, as_error *err, const as_policy_scan *policy, as_scan *scan, uint64_t *scan_id, as_async_scan_listener listener, void *udata, as_event_loop *event_loop)
AS_EXTERN as_status	aerospike_scan_background (aerospike *as, as_error *err, const as_policy_scan *policy, const as_scan *scan, uint64_t *scan_id)
AS_EXTERN as_status	aerospike_scan_foreach (aerospike *as, as_error *err, const as_policy_scan *policy, as_scan *scan, aerospike_scan_foreach_callback callback, void *udata)
AS_EXTERN as_status	aerospike_scan_info (aerospike *as, as_error *err, const as_policy_info *policy, uint64_t scan_id, as_scan_info *info)
AS_EXTERN as_status	aerospike_scan_node (aerospike *as, as_error *err, const as_policy_scan *policy, as_scan *scan, const char *node_name, aerospike_scan_foreach_callback callback, void *udata)
AS_EXTERN as_status	aerospike_scan_node_async (aerospike *as, as_error *err, const as_policy_scan *policy, as_scan *scan, uint64_t *scan_id, const char *node_name, as_async_scan_listener listener, void *udata, as_event_loop *event_loop)
AS_EXTERN as_status	aerospike_scan_partitions (aerospike *as, as_error *err, const as_policy_scan *policy, as_scan *scan, as_partition_filter *pf, aerospike_scan_foreach_callback callback, void *udata)
AS_EXTERN as_status	aerospike_scan_partitions_async (aerospike *as, as_error *err, const as_policy_scan *policy, as_scan *scan, as_partition_filter *pf, as_async_scan_listener listener, void *udata, as_event_loop *event_loop)
AS_EXTERN bool	as_scan_apply_each (as_scan *scan, const char *module, const char *function, as_list *arglist)
AS_EXTERN bool	as_scan_compare (as_scan *s1, as_scan *s2)
AS_EXTERN void	as_scan_destroy (as_scan *scan)
AS_EXTERN bool	as_scan_from_bytes (as_scan *scan, const uint8_t *bytes, uint32_t bytes_size)
AS_EXTERN as_scan *	as_scan_from_bytes_new (const uint8_t *bytes, uint32_t bytes_size)
AS_EXTERN as_scan *	as_scan_init (as_scan *scan, const char *ns, const char *set)
static bool	as_scan_is_done (as_scan *scan)
AS_EXTERN as_scan *	as_scan_new (const char *ns, const char *set)
AS_EXTERN bool	as_scan_select (as_scan *scan, const char *bin)
AS_EXTERN bool	as_scan_select_init (as_scan *scan, uint16_t n)
AS_EXTERN bool	as_scan_set_concurrent (as_scan *scan, bool concurrent)
AS_EXTERN bool	as_scan_set_nobins (as_scan *scan, bool nobins)
static void	as_scan_set_paginate (as_scan *scan, bool paginate)
static void	as_scan_set_partitions (as_scan *scan, as_partitions_status *parts_all)
AS_EXTERN bool	as_scan_to_bytes (const as_scan *scan, uint8_t **bytes, uint32_t *bytes_size)

Typedef Documentation

aerospike_scan_foreach_callback

typedef bool(* aerospike_scan_foreach_callback) (const as_val *val, void *udata)

This callback will be called for each value or record returned from a synchronous scan. Multiple threads will likely be calling this callback in parallel. Therefore, your callback implementation should be thread safe.

Parameters

val	The value received from the query.
udata	User-data provided to the calling function.

Returns

true to continue to the next value. Otherwise, the scan will end.

Definition at line 121 of file aerospike_scan.h.

as_async_scan_listener

typedef bool(* as_async_scan_listener) (as_error *err, as_record *record, void *udata, as_event_loop *event_loop)

Asynchronous scan user callback. This function is called for each record returned. This function is also called once when the scan completes or an error has occurred.

Parameters

err	This error structure is only populated when the command fails. NULL on success.
record	Returned record. The record will be NULL on final scan completion or scan error.
udata	User data that is forwarded from asynchronous command function.
event_loop	Event loop that this command was executed on. Use this event loop when running nested asynchronous commands when single threaded behavior is desired for the group of commands.

Returns

true to continue to the next value. Otherwise, the scan will end.

Definition at line 140 of file aerospike_scan.h.

Function Documentation

aerospike_scan_async()

AS_EXTERN as_status aerospike_scan_async	(	aerospike *	as,
		as_error *	err,
		const as_policy_scan *	policy,
		as_scan *	scan,
		uint64_t *	scan_id,
		as_async_scan_listener	listener,
		void *	udata,
		as_event_loop *	event_loop )

Asynchronously scan the records in the specified namespace and set in the cluster.

Call the listener function for each record scanned. When all records have been scanned, then listener will be called with a NULL value for the record.

Scans of each node will be run on the same event loop, so the listener's implementation does not need to be thread safe.

bool my_listener(as_error* err, as_record* record, void* udata, as_event_loop* event_loop)
{
    if (err) {
        printf("Scan failed: %d %s\n", err->code, err->message);
        return false;
    }
 
    if (! record) {
        printf("Scan ended\n");
        return false;
    }
 
    // Process record
    // Do not call as_record_destroy() because the calling function will do that for you.
    return true;
}
 
as_scan scan;
as_scan_init(&scan, "test", "demo");
 
as_status status = aerospike_scan_async(&as, &err, NULL, &scan, NULL, my_listener, NULL, NULL);
as_scan_destroy(&scan);

Parameters

as	The aerospike instance to use for this operation.
err	The as_error to be populated if an error occurs.
policy	Scan policy configuration parameters, pass in NULL for default.
scan	The scan to execute against the cluster.
scan_id	This legacy argument is ignored. scan_ids are now internally generated. Always pass in NULL.
listener	The function to be called for each record scanned.
udata	User-data to be passed to the callback.
event_loop	Event loop assigned to run this command. If NULL, an event loop will be chosen by round-robin.

Returns

AEROSPIKE_OK if async scan succesfully queued. Otherwise an error.

aerospike_scan_background()

AS_EXTERN as_status aerospike_scan_background	(	aerospike *	as,
		as_error *	err,
		const as_policy_scan *	policy,
		const as_scan *	scan,
		uint64_t *	scan_id )

Scan the records in the specified namespace and set in the cluster.

Scan will be run in the background by a thread on client side. No callback will be called in this case.

as_scan scan;
as_scan_init(&scan, "test", "demo");
 
uint64_t scanid = 0;
 
if (aerospike_scan_background(&as, &err, NULL, &scan, &scanid) != AEROSPIKE_OK) {
    printf("error(%d) %s at [%s:%d]", err.code, err.message, err.file, err.line);
}
else {
    printf("Running background scan job: %ll", scanid);
}
as_scan_destroy(&scan);

The scanid can be used to query the status of the scan running in the database via aerospike_scan_info().

Parameters

as	The aerospike instance to use for this operation.
err	The as_error to be populated if an error occurs.
policy	Scan policy configuration parameters, pass in NULL for default.
scan	The scan to execute against the cluster.
scan_id	The id for the scan job, which can be used for obtaining scan status.

Returns

AEROSPIKE_OK on success. Otherwise an error occurred.

aerospike_scan_foreach()

AS_EXTERN as_status aerospike_scan_foreach	(	aerospike *	as,
		as_error *	err,
		const as_policy_scan *	policy,
		as_scan *	scan,
		aerospike_scan_foreach_callback	callback,
		void *	udata )

Scan the records in the specified namespace and set in the cluster.

Call the callback function for each record scanned. When all records have been scanned, then callback will be called with a NULL value for the record.

If "scan.concurrent" is true (default false), the callback code must be thread-safe.

bool callback(const as_val* val, void* udata)
{
    if (!val) {
        return false; // Scan complete.
    }
 
    as_record* rec = as_record_fromval(val);
    // Process record
    // Do not call as_record_destroy() because the calling function will do that for you.
    return true;
}
 
as_scan scan;
as_scan_init(&scan, "test", "demo");
 
if (aerospike_scan_foreach(&as, &err, NULL, &scan, callback, NULL) != AEROSPIKE_OK) {
    printf("error(%d) %s at [%s:%d]", err.code, err.message, err.file, err.line);
}
as_scan_destroy(&scan);

Parameters

as	The aerospike instance to use for this operation.
err	The as_error to be populated if an error occurs.
policy	Scan policy configuration parameters, pass in NULL for default.
scan	The scan to execute against the cluster.
callback	The function to be called for each record scanned.
udata	User-data to be passed to the callback.

Returns

AEROSPIKE_OK on success. Otherwise an error occurred.

aerospike_scan_info()

AS_EXTERN as_status aerospike_scan_info	(	aerospike *	as,
		as_error *	err,
		const as_policy_info *	policy,
		uint64_t	scan_id,
		as_scan_info *	info )

Check the progress of a background scan running on the database. The status of the scan running on the datatabse will be populated into an as_scan_info.

uint64_t scan_id = 1234;
as_scan_info scan_info;
 
if (aerospike_scan_info(&as, &err, NULL, &scan, scan_id, &scan_info) != AEROSPIKE_OK) {
    printf("error(%d) %s at [%s:%d]", err.code, err.message, err.file, err.line);
}
else {
    printf("Scan id=%ll, status=%d percent=%d", scan_id, scan_info.status, scan_info.progress_pct);
}

Parameters

as	The aerospike instance to use for this operation.
err	The as_error to be populated if an error occurs.
policy	Scan policy configuration parameters, pass in NULL for default.
scan_id	The id for the scan job to check the status of.
info	Information about this scan, to be populated by this operation.

Returns

AEROSPIKE_OK on success. Otherwise an error occurred.

aerospike_scan_node()

AS_EXTERN as_status aerospike_scan_node	(	aerospike *	as,
		as_error *	err,
		const as_policy_scan *	policy,
		as_scan *	scan,
		const char *	node_name,
		aerospike_scan_foreach_callback	callback,
		void *	udata )

Scan the records in the specified namespace and set for a single node.

The callback function will be called for each record scanned. When all records have been scanned, then callback will be called with a NULL value for the record.

bool callback(const as_val* val, void* udata)
{
    if (!val) {
        return false; // Scan complete.
    }
 
    as_record* rec = as_record_fromval(val);
    // Process record
    // Do not call as_record_destroy() because the calling function will do that for you.
    return true;
}
 
char* node_names = NULL;
int n_nodes = 0;
as_cluster_get_node_names(as->cluster, &n_nodes, &node_names);
 
if (n_nodes <= 0)
    return <error>;
 
as_scan scan;
as_scan_init(&scan, "test", "demo");
 
if (aerospike_scan_node(&as, &err, NULL, &scan, node_names[0], callback, NULL) != AEROSPIKE_OK ) {
    printf("error(%d) %s at [%s:%d]", err.code, err.message, err.file, err.line);
}
 
free(node_names);
as_scan_destroy(&scan);

Parameters

as	The aerospike instance to use for this operation.
err	The as_error to be populated if an error occurs.
policy	Scan policy configuration parameters, pass in NULL for default.
scan	The scan to execute against the cluster.
node_name	The node name to scan.
callback	The function to be called for each record scanned.
udata	User-data to be passed to the callback.

Returns

AEROSPIKE_OK on success. Otherwise an error occurred.

aerospike_scan_node_async()

AS_EXTERN as_status aerospike_scan_node_async	(	aerospike *	as,
		as_error *	err,
		const as_policy_scan *	policy,
		as_scan *	scan,
		uint64_t *	scan_id,
		const char *	node_name,
		as_async_scan_listener	listener,
		void *	udata,
		as_event_loop *	event_loop )

Asynchronously scan the records in the specified namespace and set for a single node.

The listener function will be called for each record scanned. When all records have been scanned, then callback will be called with a NULL value for the record.

bool my_listener(as_error* err, as_record* record, void* udata, as_event_loop* event_loop)
{
    if (err) {
        printf("Scan failed: %d %s\n", err->code, err->message);
        return false;
    }
 
    if (! record) {
        printf("Scan ended\n");
        return false;
    }
 
    // Process record
    // Do not call as_record_destroy() because the calling function will do that for you.
    return true;
}
 
char* node_names = NULL;
int n_nodes = 0;
as_cluster_get_node_names(as->cluster, &n_nodes, &node_names);
 
if (n_nodes <= 0)
    return <error>;
 
as_scan scan;
as_scan_init(&scan, "test", "demo");
 
as_status status = aerospike_scan_node_async(&as, &err, NULL, &scan, NULL, node_names[0], my_listener, NULL, NULL);
 
free(node_names);
as_scan_destroy(&scan);

Parameters

as	The aerospike instance to use for this operation.
err	The as_error to be populated if an error occurs.
policy	Scan policy configuration parameters, pass in NULL for default.
scan	The scan to execute against the cluster.
scan_id	This legacy argument is ignored. scan_ids are now internally generated. Always pass in NULL.
node_name	The node name to scan.
listener	The function to be called for each record scanned.
udata	User-data to be passed to the callback.
event_loop	Event loop assigned to run this command. If NULL, an event loop will be chosen by round-robin.

Returns

AEROSPIKE_OK if async scan succesfully queued. Otherwise an error.

aerospike_scan_partitions()

AS_EXTERN as_status aerospike_scan_partitions	(	aerospike *	as,
		as_error *	err,
		const as_policy_scan *	policy,
		as_scan *	scan,
		as_partition_filter *	pf,
		aerospike_scan_foreach_callback	callback,
		void *	udata )

Scan records in specified namespace, set and partition filter.

Call the callback function for each record scanned. When all records have been scanned, then callback will be called with a NULL value for the record.

If "scan.concurrent" is true (default false), the callback code must be thread-safe.

bool callback(const as_val* val, void* udata)
{
    if (!val) {
        return false; // Scan complete.
    }
 
    as_record* rec = as_record_fromval(val);
    // Process record
    // Do not call as_record_destroy() because the calling function will do that for you.
    return true;
}
 
as_scan scan;
as_scan_init(&scan, "test", "demo");
 
as_partition_filter pf;
as_partition_filter_set_range(&pf, 0, 1024);
 
if (aerospike_scan_partitions(&as, &err, NULL, &scan, &pf, callback, NULL) != AEROSPIKE_OK) {
    printf("error(%d) %s at [%s:%d]", err.code, err.message, err.file, err.line);
}
as_scan_destroy(&scan);

Parameters

as	The aerospike instance to use for this operation.
err	The as_error to be populated if an error occurs.
policy	Scan policy configuration parameters, pass in NULL for default.
scan	The scan to execute against the cluster.
pf	Partition filter.
callback	The function to be called for each record scanned.
udata	User-data to be passed to the callback.

Returns

AEROSPIKE_OK on success. Otherwise an error occurred.

aerospike_scan_partitions_async()

AS_EXTERN as_status aerospike_scan_partitions_async	(	aerospike *	as,
		as_error *	err,
		const as_policy_scan *	policy,
		as_scan *	scan,
		as_partition_filter *	pf,
		as_async_scan_listener	listener,
		void *	udata,
		as_event_loop *	event_loop )

Asynchronously scan records in specified namespace, set and partition filter.

Call the listener function for each record scanned. When all records have been scanned, then listener will be called with a NULL value for the record.

Scans of each node will be run on the same event loop, so the listener's implementation does not need to be thread safe.

bool my_listener(as_error* err, as_record* record, void* udata, as_event_loop* event_loop)
{
    if (err) {
        printf("Scan failed: %d %s\n", err->code, err->message);
        return false;
    }
 
    if (! record) {
        printf("Scan ended\n");
        return false;
    }
 
    // Process record
    // Do not call as_record_destroy() because the calling function will do that for you.
    return true;
}
 
as_scan scan;
as_scan_init(&scan, "test", "demo");
 
as_partition_filter pf;
as_partition_filter_set_range(&pf, 0, 1024);
 
as_status status = aerospike_scan_partitions_async(&as, &err, NULL, &scan, &pf, my_listener, NULL, NULL);
as_scan_destroy(&scan);

Parameters

as	The aerospike instance to use for this operation.
err	The as_error to be populated if an error occurs.
policy	Scan policy configuration parameters, pass in NULL for default.
scan	The scan to execute against the cluster.
pf	Partition filter.
listener	The function to be called for each record scanned.
udata	User-data to be passed to the callback.
event_loop	Event loop assigned to run this command. If NULL, an event loop will be chosen by round-robin.

Returns

AEROSPIKE_OK if async scan succesfully queued. Otherwise an error.

as_scan_apply_each()

AS_EXTERN bool as_scan_apply_each ( as_scan * scan, const char * module, const char * function, as_list * arglist )

related

Apply a UDF to each record scanned on the server.

as_arraylist arglist;
as_arraylist_init(&arglist, 2, 0);
as_arraylist_append_int64(&arglist, 1);
as_arraylist_append_int64(&arglist, 2);
 
as_scan_apply_each(&q, "module", "func", (as_list *) &arglist);
 
as_arraylist_destroy(&arglist);

Parameters

scan	The scan to apply the UDF to.
module	The module containing the function to execute.
function	The function to execute.
arglist	The arguments for the function.

Returns

On success, true. Otherwise an error occurred.

as_scan_compare()

AS_EXTERN bool as_scan_compare ( as_scan * s1, as_scan * s2 )

related

Compare scan objects.

as_scan_destroy()

AS_EXTERN void as_scan_destroy ( as_scan * scan )

related

Releases all resources allocated to the scan.

as_scan_destroy(scan);

as_scan_from_bytes()

AS_EXTERN bool as_scan_from_bytes ( as_scan * scan, const uint8_t * bytes, uint32_t bytes_size )

related

Deserialize bytes to scan definition. Scan definition is assumed to be on the stack. as_scan_destroy() should be called when done with the scan definition.

Returns

true on success and false on failure.

as_scan_from_bytes_new()

AS_EXTERN as_scan * as_scan_from_bytes_new ( const uint8_t * bytes, uint32_t bytes_size )

related

Create scan definition on the heap and deserialize bytes to that scan definition. as_scan_destroy() should be called when done with the scan definition.

Returns

scan definition on success and NULL on failure.

as_scan_init()

AS_EXTERN as_scan * as_scan_init ( as_scan * scan, const char * ns, const char * set )

related

Initializes a scan.

as_scan scan;
as_scan_init(&scan, "test", "demo");

When you no longer require the scan, you should release the scan and related resources via as_scan_destroy().

Parameters

scan	The scan to initialize.
ns	The namespace to scan.
set	The set to scan.

Returns

On succes, the initialized scan. Otherwise NULL.

as_scan_is_done()

static bool as_scan_is_done ( as_scan * scan )

related

If using scan pagination, did previous paginated scan with this scan instance return all records?

Definition at line 603 of file as_scan.h.

References as_partitions_status::done, and as_scan::parts_all.

as_scan_new()

AS_EXTERN as_scan * as_scan_new ( const char * ns, const char * set )

related

Create and initializes a new scan on the heap.

as_scan* scan = as_scan_new("test","demo");

When you no longer require the scan, you should release the scan and related resources via as_scan_destroy().

Parameters

ns	The namespace to scan.
set	The set to scan.

Returns

On success, a new scan. Otherwise NULL.

as_scan_select()

AS_EXTERN bool as_scan_select ( as_scan * scan, const char * bin )

related

Select bins to be projected from matching records.

You have to ensure as_scan.select has sufficient capacity, prior to adding a bin. If capacity is insufficient then false is returned.

as_scan_select_init(&scan, 2);
as_scan_select(&scan, "bin1");
as_scan_select(&scan, "bin2");

Parameters

scan	The scan to modify.
bin	The name of the bin to select.

Returns

On success, true. Otherwise an error occurred.

as_scan_select_init()

AS_EXTERN bool as_scan_select_init ( as_scan * scan, uint16_t n )

related

Initializes as_scan.select with a capacity of n using malloc().

For stack allocation, use as_scan_select_inita().

as_scan_select_init(&scan, 2);
as_scan_select(&scan, "bin1");
as_scan_select(&scan, "bin2");

Parameters

scan	The scan to initialize.
n	The number of bins to allocate.

Returns

On success, true. Otherwise an error occurred.

as_scan_set_concurrent()

AS_EXTERN bool as_scan_set_concurrent ( as_scan * scan, bool concurrent )

related

Scan all the nodes in prallel

as_scan_set_concurrent(&q, true);

Parameters

scan	The scan to set the concurrency on.
concurrent	If true, scan all the nodes in parallel

Returns

On success, true. Otherwise an error occurred.

as_scan_set_nobins()

AS_EXTERN bool as_scan_set_nobins ( as_scan * scan, bool nobins )

related

Do not return bins. This will only return the metadata for the records.

as_scan_set_nobins(&q, true);

Parameters

scan	The scan to set the priority on.
nobins	If true, then do not return bins.

Returns

On success, true. Otherwise an error occurred.

as_scan_set_paginate()

static void as_scan_set_paginate ( as_scan * scan, bool paginate )

related

Set to true if as_policy_scan.max_records is set and you need to scan data in pages.

Definition at line 577 of file as_scan.h.

References as_scan::paginate.

as_scan_set_partitions()

static void as_scan_set_partitions ( as_scan * scan, as_partitions_status * parts_all )

related

Set completion status of all partitions from a previous scan that ended early. The scan will resume from this point.

Definition at line 590 of file as_scan.h.

References as_partitions_status_reserve(), and as_scan::parts_all.

as_scan_to_bytes()

AS_EXTERN bool as_scan_to_bytes ( const as_scan * scan, uint8_t ** bytes, uint32_t * bytes_size )

related

Serialize scan definition to bytes.

Returns

true on success and false on failure.