KV cache tiering: Why GPU memory alone won't scale your LLM app

Large language models (LLMs) are remarkably efficient at generating text, but serving them at a production scale remains an infrastructure challenge for many organizations. And while the industry at large focuses on larger GPUs, faster interconnects, or more efficient inference engines, another bottleneck quietly grows alongside every conversation: the key-value (KV) cache.

Inference performance in production environments is no longer constrained solely by compute capacity; it is increasingly limited by the finite amount of high-bandwidth memory (HBM) available on GPUs. As context windows expand from thousands to hundreds of thousands of tokens, applications shift toward longer multi-turn interactions, and workloads such as retrieval-augmented generation (RAG) and autonomous AI agent workflows become commonplace, KV cache growth will increasingly consume a larger share of GPU memory.

Over time, managing this expanding cache footprint will become one of the defining challenges in scaling LLM inference. Every additional token increases the amount of cached data that must remain available for future inference, and every additional user multiplies this requirement. However, GPU memory does not scale as well, and modern LLM infrastructure requires multiple storage tiers beneath GPU memory to allow data to move intelligently between fast and slower storage, rather than recomputing from scratch.

The cache that outgrows its memory

Tools like LMCache make AI models faster and more cost-efficient by preserving the “memory” generated from previous conversations and document processing. Instead of forcing a model to recompute the same information every time it receives a similar request, LMCache enables it to reuse previously generated intermediate results.

When an AI model processes text, it converts that information into a key-value (KV) cache: a collection of intermediate representations that serve as the model’s working memory, capturing what it has already computed and understood. By retaining these cached values, the model can skip redundant calculations during follow-up requests, reducing inference latency and lowering the computational resources required to generate responses.

Traditional AI inference systems typically discard this cache after generating a response. As a result, when a user asks a related question, the model must reread and reprocess the entire conversation or document from the beginning. LMCache changes this by storing KV cache data in a reusable library. When a new prompt arrives, LMCache searches for matching cached information and retrieves previously computed results. By reusing this memory, the AI model can respond faster, reduce unnecessary computation, and lower inference costs.

Diving deeper into the KV cache

Every token processed by an LLM produces three vectors during attention computation:

• Query (Q)

• Key (K)

• Value (V)

The query vector is temporary and used only to compute attention for the current token. The key and value vectors, however, must persist because future tokens may need to reference information from earlier parts of the sequence. Rather than recomputing these vectors for every new token, inference engines store them in the KV cache, allowing the model to efficiently reuse previously generated attention states. This caching mechanism is essential because transformer models continuously rely on information from prior tokens throughout the text generation process.

KV caches are organized by layer, attention head, and token. Every transformer layer maintains its own cache, meaning a model with dozens of layers accumulates a surprisingly large amount of cached information as a prompt grows. Because every newly generated token immediately references previously generated tokens, this cache must reside as close to the GPU's tensor cores as possible. The KV cache must therefore live in a GPU’s HBM. HBM provides enormous bandwidth, measured in terabytes per second, that allows attention operations to retrieve cached vectors with minimal latency.

Speed comes with a trade-off

HBM is extremely limited in capacity, creating a fundamental bottleneck as KV caches grow with each additional token and compete with the model for the same scarce memory resources. Consider a Llama-3-70B-style model with 80 transformer layers, 8 KV heads, and a head dimension of 128, using FP16 precision. Under these assumptions, each token consumes roughly 0.5 MB of KV cache, which sounds insignificant until context windows become large.

A single conversation with a 10,000-token prompt requires approximately 10,000 tokens × 0.5 MB ≈ 5 GB of KV cache. 

Now consider a production deployment serving multiple users simultaneously:

Most production AI accelerators provide roughly 40-192 GB of HBM capacity. Even systems equipped with multiple accelerators must carefully partition memory between model weights, activation buffers, temporary tensors, and the KV cache.

The implications of KV cache growth are essentially linear: double the context length and the cache size; double the concurrent users, and the cache size doubles again. GPU memory, meanwhile, is fixed. Purchasing larger GPUs raises the ceiling but does not fundamentally change the scaling relationship. As applications increasingly depend on longer prompts, persistent conversations, and reusable context, architects eventually encounter a point where GPU memory simply cannot hold every useful cache entry.

You can estimate your own exposure using the following formula as a rough approximation:

Required KV cache ≈ Per-token cache × Context length × Concurrent sessions

Exact numbers vary depending on the model architecture, context length, and precision format, but the broader implication remains the same: as context windows expand, memory quickly shifts from a supporting resource to the primary bottleneck limiting large-scale inference performance and scalability.

What happens when the cache doesn't fit

Once GPU memory fills, inference systems must make difficult compromises. Most current deployments rely entirely on GPU-resident caching with no automatic overflow tier beneath HBM. When memory becomes scarce, older cache entries are simply evicted, which creates the following operational issues:

Cross-worker cache misses

Prefix caching works extremely well, provided subsequent requests land on the same inference worker. In a typical production deployment, however, dozens of inference servers operate behind a load balancer. A prompt whose prefix has already been processed on Worker 12 offers no benefit if the next request is routed to Worker 37. Each worker maintains its own isolated GPU cache. The result is duplicated computation despite identical prompt prefixes. Organizations often assume that prefix caching automatically accelerates repeated prompts across an entire cluster. In reality, GPU-local caches rarely provide that level of sharing.

Restart penalties

GPU memory is volatile. Whenever an inference server restarts, whether due to maintenance, software upgrades, failures, or autoscaling events, the entire KV cache is immediately lost. Every subsequent request must perform a full prefill operation because none of the previously computed key/value vectors are preserved. For production workloads, rebuilding these warm caches can take tens of minutes before performance returns to steady state, depending on traffic patterns and model behavior. These performance penalties occur every time infrastructure is cycled; they are not limited to rare disaster recovery scenarios.

Hard eviction

GPU memory has hard capacity limits. As active inference requests demand memory, cached prefixes eventually compete with live workloads. Without another storage tier, valuable cache entries are discarded permanently. The next time those prompts appear, the model performs the expensive prefill stage again, consuming GPU cycles that were already spent previously.

When GPU memory becomes the bottleneck

Cross-worker cache misses, restart penalties, and hard evictions are the default behavior of GPU-only caching. And as concurrency increases, they become more commonplace. Architects evaluating production inference stacks must ask a critical question: What happens to cached information when the underlying infrastructure changes? Can cache entries survive worker restarts, can cached prefixes be shared across multiple workers, and how does the system behave when GPU memory reaches capacity?

If the answer to each question is that cached data is simply discarded, then GPU memory has become the limiting factor, not just for performance, but for the scalability and efficiency of the entire inference architecture.

Tiering as the answer beneath the GPU

Rather than treating GPU memory as the only location where cached tensors can reside, modern inference systems increasingly introduce additional storage tiers beneath HBM to avoid recomputing expensive KV tensors whenever possible.

Here’s a typical tiered storage hierarchy:

Each level trades bandwidth for capacity. HBM is extraordinarily fast but extremely scarce. CPU memory offers significantly larger capacity with lower bandwidth. NVMe SSDs sacrifice additional performance while dramatically increasing available storage. And remote storage introduces network latency but enables cache sharing across servers and persistence beyond individual machines.

Designing a cache hierarchy for scalable AI inference

The question is not which storage tier is best, as each tier serves a specific purpose. The real architectural challenge is determining where each cache entry belongs within the hierarchy. Some cache entries may justify occupying scarce GPU memory because they are frequently accessed and latency-sensitive, while others can be moved to CPU memory, persisted on SSD, or stored in remote systems for future reuse.

Defining how cache data moves between these tiers is equally important. For example, an effective cache hierarchy continuously promotes frequently accessed data closer to compute while demoting colder entries to lower-cost storage layers, thereby avoiding the need to repeatedly recompute expensive intermediate states. By thinking in terms of a tiered architecture rather than individual storage devices, architects gain a more realistic framework for planning capacity, optimizing performance, and controlling costs. In this model, GPU memory serves as the fastest cache layer, not the only one.

What a tiering layer needs to get right

Tools like LMCache provide a modular KV cache management layer purpose-built for LLM inference workloads, enabling models to leverage multi-tier cache architectures. Rather than replacing existing inference engines, these systems operate beneath them, managing the movement of KV tensors between ultra-fast GPU memory and lower-cost storage tiers.

At a functional level, LMCache exposes familiar key-value store operations, including:

• put(key, blob)

• get(key)

• delete(key)

• exists(key)

• list(prefix)

While these APIs resemble traditional caching systems, the underlying data model is fundamentally different. Conventional web caches are typically designed around small objects measured in kilobytes, such as images, API responses, or web assets.   

In contrast, KV caches are structured collections of tensor states that can scale into hundreds of megabytes or more per request, far beyond the kilobyte-sized objects conventional caches expect. These cached key-value tensors capture the results of prior attention-layer computations, allowing an LLM to reuse previously processed context rather than recomputing it at each generation step.  The size of these cache objects quickly changes the storage equation. 

For example, in a 70-billion-parameter model with 80 transformer layers, eight KV heads, and a head dimension of 128, a 1,000-token prefix can require approximately 328 MB of KV cache using FP16 precision, 164 MB using INT8, or 82 MB using INT4. These are the sizes of individual cache objects, meaning production inference systems may routinely need to retrieve hundreds of megabytes of cached tensor data to serve requests efficiently.

As a result, KV cache tiering requires a fundamentally different approach to infrastructure design. The challenge is not merely finding additional capacity for storing cache data, but also efficiently moving large computational artifacts across storage tiers while maintaining the low-latency performance that makes caching effective in the first place.

Achieving this requires several critical capabilities:

High-throughput bulk transfers

KV cache retrieval typically involves loading entire tensor blocks before inference can continue. Storage systems optimized for small random reads provide limited value when workloads require moving hundreds of megabytes at a time.

Asynchronous writes

Cache persistence should not add latency to user requests. Most tiering systems write cache entries asynchronously after inference completes, allowing response generation to remain focused on minimizing user-facing latency.

Massively parallel reads

Frequently used prompt prefixes may be requested simultaneously by multiple inference workers. A tiering system must sustain high levels of concurrent read traffic without becoming a bottleneck under contention.

Policy-driven eviction

Not every cached tensor should be stored indefinitely. Effective tiering layers require intelligent metadata management to support eviction policies, such as least recently used (LRU), least frequently used (LFU), and time-to-live (TTL), while minimizing unnecessary data movement.

These requirements underscore why KV cache tiering differs fundamentally from traditional caching. Conventional cache systems are designed to manage millions of small objects with relatively predictable access patterns. KV cache systems, by contrast, handle a much smaller number of extremely large tensor objects with highly asymmetric read and write behavior. Supporting these workloads requires a new class of caching infrastructure purpose-built for AI inference, with specialized data-layer mechanisms to efficiently store, retrieve, and move massive tensor data while maintaining predictable, low-latency performance.