Queries, Long and Short
Overview
This section describes Aerospike's support for short and long queries on the primary index (PI queries, formerly known as scans), and secondary indexes (SI queries, formerly known as queries).
Unified query interface
PI queries and SI queries exist in a single query subsystem and share the following features:
- Queries are partitioned. By default, queries target all the partitions, allowing for large data sets to be processed in parallel. You can also target specific partitions, or a single partition with a query.
- Queries can be paginated. A query may return a large number of results (billions of records), which could overwhelm the client. The query can be paginated, with the size of the result, using the max records query policy. The client partition filter object keeps track of the query progress across multiple partitions, and can be used to continue the query each time.
- Queries are rebalancing-tolerant. Compatible clients in Database 6.0 and later ensure that queries can handle automatic partition migration when cluster sizes change.
Differences between short and long queries
The following table shows the differences between short and long queries.
| Short Query | Long Query |
|---|---|
| Use the short query when you expect a small number of records in the result, and when the query execution is highly concurrent | Use the long query when you expect a large number of records in the result |
Short queries are not tracked. Useful when short-lived queries run frequently in order to avoid polluting the active_jobs queue | Long queries are tracked in active_jobs queue when they are active and in finished_jobs queue when they complete |
| Short queries are measured by query latency and record count histograms | Long queries do not provide latency and record count histograms |
| Short queries cannot be aborted | Long queries can be aborted |
| Short query execution cannot be throttled | Long query execution can be throttled by setting an RPS (records per second) cap |
| Short queries have a default 1s transaction timeout (on the socket) | Long queries do not time out on the transaction length |
Optimizing queries by their expected runtime duration
Developers can specify whether the query is long (the default) or short from the Query policy class of compatible clients.
Long query
Long queries are "long" because they retrieve a large number of records from the database. How long it takes depends on factors such as complex filtering conditions, the size of the dataset, indexing strategies, and the IOPS (input/output operations per second) capacity of the cluster.
When you query an SC namespace, the client reserves partitions and arranges them in order to query only full partitions. This is true even when a cluster is rebalancing by migrating partitions after changing size. Setting a long query with adequate sleep between retries is more insurance that migrations do not negatively affect the query.
Long queries against AP namespaces delay dealing with migrating partitions, so consistency is not an issue. Database 7.1 introduces the expected duration policy field that controls how a ‘long’ query - that returns only a few records - should behave if it hits a migrating partition. This new field essentially converts the long query to a short query and uses what it sees, or errors out.
Short query
Short queries against AP namespaces relax read consistency if they must reserve migrating partitions, and return what they see in a migrating partition instead of erroring out when hitting a migrating partition.
The Scan classes are deprecated starting with the Database 6.0-compatible clients, such as Java client 6.0.