Monitor index memory allocation

Each namespace stores its primary index, secondary index, and set index in a separate memory pool, called an arena. Each arena grows in fixed-size chunks, called stages, and reserves a whole stage at a time. Aerospike fills each stage from start to end, and the allocation point is the position where the next new entry goes. Aerospike Database 8.2.0 reports three figures for each arena: allocated bytes for stages reserved, used bytes for live index elements, and tail bytes for unused space in the newest stage. Use them together to explain the asd process footprint and to predict when an arena reserves another stage. The gap between allocated and used memory can be several GiB.

Applies to: Aerospike Database 8.2.0 and later. Allocation metrics for the primary and secondary indexes require Aerospike Database Enterprise Edition, because index-type and sindex-type are configurable only in Enterprise Edition. set_index_alloc_bytes, process_rss_bytes, and the control group (cgroup) metrics are available in both editions.

Audience: Operators and SREs sizing nodes, setting memory alerts, or investigating asd memory use against a configured budget.

The three numbers

For each index arena, the server reports up to three values.

Value	Metric	Meaning
Allocated	*_alloc_bytes	Total memory the arena has reserved: number of stages multiplied by stage size.
Used	*_used_bytes	Memory holding live index elements.
Tail	*_tail_bytes	Unused space in the newest stage: how much can still be allocated before another stage is reserved.

Record deletion frees space behind the allocation point. Calculate that reusable space as:

reusable space = alloc - used - tail

The following diagram shows where each value lives in an arena with three stages.

 stage 0            stage 1            stage 2 (newest)
+-----------------+-----------------+-----------------+
| live | reusable | live | reusable | live |   | tail |
+-----------------+-----------------+-----------------+
                                          ^
                                          allocation point
 <----------- alloc = stage count * stage size ------->

Each new stage is reserved after the previous stage is full. Earlier stages sit entirely behind the allocation point, so only the newest stage has a tail.

Metrics by arena

Arena	Allocated	Used	Tail
Primary index, shmem	index_shmem_alloc_bytes	index_used_bytes	index_shmem_tail_bytes
Primary index, pmem	index_pmem_alloc_bytes	index_used_bytes	index_pmem_tail_bytes
Primary index, flash	index_flash_alloc_bytes	index_used_bytes	Not reported
Secondary index, shmem	sindex_shmem_alloc_bytes	sindex_used_bytes	sindex_shmem_tail_bytes
Secondary index, pmem	sindex_pmem_alloc_bytes	sindex_used_bytes	sindex_pmem_tail_bytes
Secondary index, flash	sindex_flash_alloc_bytes	sindex_used_bytes	sindex_flash_tail_bytes
Set index	set_index_alloc_bytes	set_index_used_bytes	Not reported

A namespace reports the allocation metrics that match its configured backing type. For example, index-type shmem reports index_shmem_alloc_bytes.

Flash primary index and set index allocate as follows:

• Flash primary index. A flash-backed primary index allocates by 4 KiB chunk within a stage. index_flash_alloc_bytes reports the sum of those allocated chunks, so it is a different quantity from the shmem and pmem stage-footprint metrics. Remaining unused space is spread across chunks. The secondary index arena allocates from one allocation point, so a flash-backed secondary index reports a tail.
• Set index. A namespace holds one set-index arena per data-bearing partition tree per indexed set, and each arena has its own allocation point. set_index_alloc_bytes already includes a 4 KiB stage floor per arena, and remaining per-arena unused space is small next to that floor.

How the budget relates to allocation

Eviction and stop-writes act on used bytes. The eviction supervisor (nsup) compares the combined RAM index usage against indexes-memory-budget using the same numerator as indexes_memory_used_pct, and evict-indexes-memory-pct acts on that value.

*_alloc_bytes reports the larger stage footprint. That allocated figure can sit above indexes-memory-budget while indexes_memory_used_pct remains below 100, because the budget is evaluated against used bytes.

Predict arena growth from the tail

Index stages default to 1 GiB (index-stage-size), so index growth happens in 1 GiB steps. The tail shows how much of the current stage remains.

• A tail above 0 means the current stage still has unused space. Allocation at the allocation point consumes the tail, and the arena reserves another stage only after the current stage is full. A namespace with 2 GiB of primary index tail absorbs 2 GiB of new index elements, roughly 33 million records at 64 bytes each, before growing.
• A tail of 0 means the next allocation may reuse deleted space or reserve a new stage. The primary index arena keeps 64 independent free lists. Allocation selects one in round-robin order, and frees return elements to a list determined by element ID, so an allocation can find its list empty and reserve a new stage while the other 63 lists still hold reusable elements.

Read the two values together:

Tail	Calculated reusable space (alloc - used - tail)	Reading	Action
Trending to 0, used still climbing	Any	Another stage is imminent	Provision for alloc plus one index-stage-size
Flat	Growing	Deletes are landing and being recycled	None. The footprint stays at the highest level reached
Near a full stage	Any	A stage was just reserved	Roughly 16.7 million records of capacity added per 1 GiB stage
0	Large	Capacity exists, but an individual insert can miss its free list	Alert. Other measures can still look fine

How reusable the freed space is differs by arena

Freed space is reused differently in each arena.

Arena	Free-list structure	When reusable space prevents a new stage
Secondary index	One free list, one allocation point	Always. Reusable space is interchangeable with tail, so the derived number is usable free space.
Primary index	64 free lists; allocation selects round-robin, frees return by element ID	While the tail is above 0, reusable space absorbs bulk growth. After the tail reaches 0, an individual allocation can miss its free list.
Set index	One free list per arena, up to one arena per partition tree per indexed set	Within one partition tree and set. A delete in partition 7 creates free space for inserts into that same tree.

Accuracy of the calculated reusable space value

For the primary index, reusable space = alloc - used - tail overstates true space freed by deletes slightly:

• The allocation point advances in batches of 512 elements, and the remainder of each batch is pushed onto a free list before it is handed out. Across 64 free lists this accounts for at most 64 × 512 × 64 bytes, or about 2 MiB per namespace, which is negligible against a 1 GiB stage.
• Element 0 is reserved and never allocated, so it counts as one element of reusable space.

For the secondary index arena, use reusable space = alloc - used - tail as an estimate. The arena reserves element zero (about 4 KiB), which the formula counts as reusable space, and alloc, used, and tail are independent unlocked snapshots.

Memory after deletes

After a mass delete against the primary index, index_{shmem,pmem}_alloc_bytes stays at its highest level reached. Freed elements go onto free lists and remain available for reuse up to the previous used value.

The three arenas free space at different times:

Arena	How space is freed	Consequence for the metric
Primary index	Freed elements go onto one of 64 free lists	index_{shmem,pmem}_alloc_bytes only increases and stays at the highest level reached. index_flash_alloc_bytes can decrease when partition trees are destroyed and chunks are freed. index_*_tail_bytes only shrinks between stage additions for shmem and pmem, because the allocation point only moves forward
Secondary index	Every stage is released when the last element is freed	sindex_*_alloc_bytes can drop to 0 in one step, and sindex_*_tail_bytes goes to 0 with it
Set index	Stages return when the tree is destroyed, which happens when the set index is removed or the partition stops holding data	set_index_alloc_bytes follows partition ownership in steps. It drops on rebalance or removal

Behavior across a warm restart

The three arenas behave differently across a warm restart:

• Primary index and secondary index stages are reattached, so index_*_alloc_bytes and sindex_*_alloc_bytes continue from their pre-restart values. The primary index highest level reached, including its accumulated reusable space, survives.
• Set index stages are heap allocations rebuilt during warm restart before the info service starts, so metrics scrapes do not show a drop to 0 or a climb during resume. The post-restart set_index_alloc_bytes and set_index_used_bytes values can differ from pre-restart values.

Expect the set index graphs to be discontinuous across a warm restart where the other two are continuous.

Reconcile against process RSS

process_rss_bytes reports the resident memory of the asd process. For container capacity, compare cgroup_memory_used_bytes with cgroup_memory_limit_bytes; the control group’s total includes page cache and any other processes in the group, so it can exceed asd’s RSS. The heap_*_kbytes metrics report the allocator’s view and omit shared memory segments and memory-mapped data.

Caution

*_alloc_bytes statistics report allocator capacity (whole stages reserved for shmem and pmem, or heap stages for the set index), not resident memory. Mapped shmem and pmem stages contribute to process_rss_bytes only for pages the process has touched. Do not sum per-namespace *_alloc_bytes with process_rss_bytes to estimate total memory use.

A value of 0 for process_rss_bytes means /proc/self/statm could not be read or parsed. Treat 0 as a failed read.

Control group limits

cgroup_memory_used_bytes and cgroup_memory_limit_bytes report the kernel’s raw usage and limit when the server finds a usable control group with a finite limit. cgroup-mem-tracking controls whether system_free_mem_pct and system_free_mem_kbytes use control-group-aware arithmetic for stop-writes and free-memory reporting.

Caution

With the default (cgroup-mem-tracking false), system_free_mem_pct and system_free_mem_kbytes report host-level free memory even inside a container, while cgroup_memory_*_bytes may still be published when control group files are readable.

When cgroup-mem-tracking is enabled and a finite limit is found, system_free_mem_pct subtracts reclaimable page cache from usage and clamps the result by host availability. That value can differ from 100 - (cgroup_memory_used_bytes * 100 / cgroup_memory_limit_bytes) because subtracting inactive_file can raise the free percentage, while clamping to host MemAvailable can lower it on a busy host.

Expressing allocation as a percentage

The allocation and tail metrics are byte counts. index_flash_alloc_pct is the allocation percentage the server reports. To express another allocation figure as a fraction of a budget, divide by indexes-memory-budget or the relevant mount budget. Both are available through get-config.

indexes_memory_used_pct uses the same denominator and used bytes as its numerator.

View allocation with asadm

Tools 13.1.0 (asadm 5.1.0) and later display these metrics without manual arithmetic:

• info namespace usage adds an Alloc column to the Primary Index and Secondary Index groups, next to Used, from the allocation metric that matches each namespace’s backing type.
• info memory -v shows allocated and used bytes for the memory-backed primary, secondary, and set index arenas of each node, next to host memory, cgroup memory, and process heap.

The Alloc% column in info memory is a percentage of the node’s memory capacity, not of indexes-memory-budget.

Next steps

• Key metrics to monitor
• info memory in asadm
• Metrics reference
• Configuring the primary index
• Configuring the secondary index