Getting the most from your LLM inference system

Training gets the headlines, but inference is where a large language model (LLM) earns its keep. Every chatbot reply, every code completion, and every summarized document is one pass of LLM inference. 

It is also where most teams first hit a wall: the model runs, the text appears, and it appears slowly. Why does that happen? Here’s what to do about it, starting from the mechanics and building toward the memory problem that now affects how inference infrastructure gets designed.

What is LLM inference?  

LLM inference uses a trained artificial intelligence (AI) model to answer a new prompt. Unlike LLM training, it doesn't change the model or teach it anything new. Instead, the model reads your prompt, delivers a prediction of the next word or portion of a word (called a token), adds it to the response, and repeats that process until the answer is finished. Because inference only uses the model's existing knowledge instead of updating it, each request is much cheaper than training. But because inference happens every time someone uses the model, it creates an ongoing operating cost rather than a one-time expense. 

However, inference is not just one thing. It runs in two phases with different hardware demands.

• The prefill phase¹ processes the entire input prompt at once. Because every input token is known upfront, the model computes their representations in parallel through large matrix-matrix operations, which keeps the GPU busy and makes prefill processor-bound.

• The decode phase then generates the response one token at a time, with each new token depending on all the tokens before it. That step is typically a matrix-vector operation that leaves much of the GPU's processing idle while it waits on memory. 

For most single-stream deployments, decode is primarily bound by how fast data moves rather than how fast the chip multiplies. However, that balance can change depending on how the model is run. Processing many requests at once, using lower-precision numbers (quantization), or using different model designs may make processing power, rather than memory, the performance bottleneck. 

This is why one tokens-per-second figure doesn’t mean much. Prefill and decode run at different speeds. Serving requests in large batches increases overall throughput, but one user's response may be much slower than the system's aggregate performance suggests. It is also why the automatic "buy more processing" often doesn’t help. When the decode phase dominates a workload, the limiting resource is frequently memory bandwidth. 

Most optimization techniques either target one phase or the other, or the memory that connects them.`

Why is my inference so slow?

The most common inference problem is a powerful GPU that produces only a few tokens per second, far below its potential.  Because it may be caused by several problems that look similar, the first task is diagnosis. 

1. Start with GPU activity. If utilization sits near zero with slow generation, the model is running on CPU rather than GPU. This happens with a CPU-only build or when no layers were offloaded to the GPU, and it produces the 4-to-7-tokens-per-second-with-an-idle-GPU symptom.  

2. If the GPU is busy but the model is still slow, the model may be too large to fit in the GPU's memory. When that happens, the GPU has to repeatedly retrieve data from the computer's main memory, which is slower. The exact slowdown depends on the hardware and the software used to manage memory. 

3. If one request is fast but performance drops as soon as several users arrive, the problem is the serving setup rather than the hardware.

4. If utilization swings erratically, a wrapper layer may be adding overhead the raw engine does not have.

“Slow inference" is a family of problems, not one. CPU fallback and VRAM spillover don’t create an error; the model loads and the text streams, just slower than the expected rate. Figuring out which failure you have is most of the work, because each points to a different specific fix. 

Diagnosing speed almost always leads back to whether the model, plus everything it needs at runtime, fits in GPU memory.

How much VRAM do you need?

Two aspects use GPU memory during inference.

• Model weights are the obvious cost and a fixed one: a 7-billion-parameter model in 16-bit precision needs roughly 14 GB just to load.

• The second major inference cost is the key-value (KV) cache, which stores information about previous tokens so the model does not have to recalculate them every time it generates a new token. While this improves performance, cache memory grows with context length and concurrent users. As a result, a model that fits based on weight size alone may not have enough GPU memory for production workloads once the KV cache and runtime overhead are included. 

The size of the KV cache depends on the model's design, the precision used to store it, and the software running the model, so it must be calculated for each model rather than estimated from parameter count alone. Memory usage, not just the number of parameters, determines how many users a system supports and whether it will run out of memory under production workloads.

Attention architecture is another factor. ”Attention” helps an LLM decide what information matters. It does this with “attention heads,” which are the parallel "viewpoints" that perform this analysis. The number and design of those heads affect memory usage during inference. 

• Multi-head attention keeps a separate set of key and value tensors for every attention head, which makes its cache large. 

• Grouped-query and multi-query attention¹ share key and value heads across groups of query heads, cutting cache size substantially while keeping most of the model quality. 

Two models with the same number of parameters may have different memory requirements when running the same workload. 

The rule of thumb is to choose a GPU based on enough VRAM capacity first to hold the model, memory speed second, and raw processing power last. That ordering runs against the instinct to compare processing numbers. 

Bandwidth first, compute last

Single-user LLM generation is usually limited by memory bandwidth, not processing power. GPU specifications such as TFLOPs and CUDA core counts measure how fast a GPU performs calculations, but token generation often slows down because the GPU is waiting for model data to be loaded from memory. 

Let’s look at some common GPUs.

• The RTX 3090 delivers 936 GB/s of memory bandwidth across its 24 GB of GDDR6X on a 384-bit bus, and that bandwidth already keeps single-stream decode reasonably fed, which is why the used 3090 remains a strong value pick for local serving.

• The RTX 4090 is faster: a larger L2 cache, newer tensor cores with FP8 support, and better kernels all help. But for memory-bound single-stream decode, its lead is narrower than a raw TFLOPs comparison implies, because much of that extra capability aids training, fine-tuning, and large-batch or low-precision workloads more than it aids one user generating tokens.

• Similarly, NVLink does little for a single-stream workload, but once inference spans multiple GPUs through tensor, pipeline, or expert parallelism, the interconnect between cards matters more. 

The buying decision should be based more on the workload shape than on the performance benchmark.

• Maximum single-stream speed favors the 4090

• Larger single-card models favor a 32 GB card

• A 70-billion-parameter model favors pooling two 3090s to hold the weights and cache together

The overall idea is that memory is frequently the limiting resource for inference, and treating the problem by throwing processing at it doesn’t help. 

Quantization without the anxiety

Precision is the other factor of memory use. 

Quantization reduces the numerical precision of a model to use less memory and move more of it per unit of bandwidth. The questions then become how aggressively to quantize, and which format to use.

How aggressively is a quality trade-off that depends on the task. Quantization² to 8-bit or 4-bit often preserves quality surprisingly well on general language tasks, which is where the "basically free" reputation comes from. 

However,  it doesn’t work as well on some tasks. On tasks involving large documents and generating code,  degradation becomes more noticeable. Perplexity, a measure of how well a language model predicts text, can't by itself determine how good a quantized model is because performance varies by task. 

Which format is about compatibility as much as quality. Format has to match the inference engine: a GGUF file will not load in an engine expecting a different format. The field now spans several viable options, including GGUF, AWQ, GPTQ, EXL2, and low-precision floating-point formats, such as FP8 and MXFP4, and the right pick depends on the workload. 

As a rough guide:

• GGUF in a Q4_K_M variant suits CPU, Apple silicon, and consumer GPUs served through llama.cpp or Ollama

• AWQ remains a popular GPU quantization format with a strong quality-performance tradeoff

• GPTQ works as a fallback, ideally paired with faster kernels

• FP8 is increasingly common on newer hardware that supports it natively

Keep in mind that a smaller quantization is not automatically faster. Dequantization adds overhead, and once a model already fits in memory, quantizing more aggressively can cost speed rather than gain it, because fit and bandwidth were the constraints, not precision. Choose precision by task and format by deployment target, and skip the impulse to shrink for its own sake.

Choosing a serving engine

Once precision and memory are settled, the next decision is which software runs the model.

An inference framework determines how a model handles real traffic. Generally, developers prototype locally on a lightweight tool, then move to a production engine before serving concurrent users. That’s because lightweight local tools serve a handful of users well but don’t do well under load, while a production inference engine handles multiple users better.  

One reproducible comparison³ on one A100-40GB running Llama 3.1-8B measured vLLM at 793 tokens per second against 41 for a local-first tool. Of course, that specific figure is configuration-dependent, but in general it shows the difference. 

The reason? Continuous batching. Rather than waiting for a fixed group of requests to finish before starting the next group, a continuous-batching scheduler evaluates the batch at every generation step, evicting a request when it completes and admitting a waiting one into the freed slot. This keeps the GPU busy across requests of different lengths, which is the normal condition in production, and it is why production engines are better than single-user tools under load. 

Speculative decoding is a second option some engines offer: A smaller draft model proposes several tokens ahead and the main model verifies them in parallel, which reduces latency when the draft is often right, though it wastes processing on the tokens it rejects.

What matters most is the workload:

• Local, development, or air-gapped use favors Ollama, llama.cpp, or LM Studio. 

• Production concurrency favors vLLM. 

• Prefix-heavy chat, retrieval, and agent workloads favor SGLang, whose RadixAttention indexes cached tokens in a radix tree and reuses shared prefixes across requests, reporting roughly 29% higher throughput than vLLM on 7-to-8-billion-parameter models and narrowing to a few percent at 70-billion scale. 

• One long-lived model where maximum throughput on NVIDIA hardware justifies heavier setup favors TensorRT-LLM.

What matters is throughput under realistic concurrency on identical hardware, not a feature comparison. Which engine is best depends on the workload, and the best way to choose is to match the engine to the traffic rather than to a benchmark.

The self-host vs. API decision

The biggest business question is whether to run the hardware yourself, or call a hosted API. A GPU rented or bought by the hour looks cheaper than per-token API pricing, but self-hosting doesn’t always save money. 

Here are some other costs to consider:

• A self-hosted GPU is paid for whether or not it is busy, so low utilization inflates the per-token cost. 

• DevOps labor to build, secure, monitor, and update the serving stack is a continuing expense. 

• Handling traffic spikes means overprovisioning capacity that doesn’t get used most of the time. 

• Model updates have to be taken care of. 

Add these in, and hosted APIs are often cheaper. Estimates of where the break-even falls range widely, from tens of millions to hundreds of millions of tokens per month, which shows how much the answer depends on utilization, labor cost, and traffic. 

Obviously, sometimes you have to self-host. Regulated data under HIPAA or SOC 2, air-gapped environments, custom fine-tunes, and hard low-latency requirements may all require self-hosting regardless of the break-even. 

To answer this question, compute the break-even with realistic utilization and labor assumptions, then check whether a compliance or latency constraint requires it. 

The KV cache is not a speed trick. It's system state.

The KV cache is usually introduced as a local optimization: Store the key and value tensors so the decode phase does not recompute attention over the whole sequence at every step. However, the cache also grows without bound, crashes servers, and, at scale, is a piece of state that no longer is associated with any one GPU.

Here’s why:

• The cache grows with context length and concurrency, so a serving engine that reserves cache space for the maximum context can run out of memory on the first request if that reservation is set too high. 

• Requests that return token-level probabilities can push memory past its limit. 

• Very long inputs can cause KV cache performance problems, and many users want the option to keep the model on the GPU while storing the KV cache in CPU memory.

These are not edge cases at the margin; they are common problems with running inference at scale, and they are largely memory problems. 

Here’s a series of responses that can fix those problems:

• FlashAttention¹ reorders the attention computation to reduce how often data moves through the GPU's memory hierarchy, cutting the memory-movement cost of attention. 

• Paged attention stores the cache in non-contiguous fixed-size blocks so memory is not wasted on over-reservation. 

• Prefix caching reuses the cache for shared prompt segments across requests. 

• Cache offloading moves less-active entries to CPU or disk. 

• Prefill-decode disaggregation separates the processing-bound prefill phase and the bandwidth-bound decode phase onto different hardware pools, which requires transferring the KV cache between them; one production deployment reported roughly a 3x gain in output tokens per second and a 2x reduction in time-to-first-token after adopting prefix-cache-aware routing. 

• Move the cache out of GPU memory into a shared tier. The LMCache⁴ system, paired with vLLM, reports up to a 15x throughput improvement on multi-round question-answering and document-analysis workloads by doing that. 

LMCache is not a final solution, but it shows what becomes possible once the cache is treated as a shared, externalized tier rather than per-GPU scratch space.

These techniques show the KV cache migrating from local GPU memory toward shared, disaggregated storage. 

Cache quantization makes this more complicated, because architectures handle it differently. A model using latent attention showed a measurable perplexity increase with a 4-bit cache, where another model's quality barely moved, so the safe cache precision is architecture-dependent rather than universal. 

The KV cache is one of the primary scaling bottlenecks of inference, and treating it as shared system state rather than local scratch space is where much of the current work in inference optimization is now concentrated.

Measuring what matters

Managing that state well depends on measuring the right things.

Inference performance needs a few specific metrics:

• Time-to-first-token measures prefill latency is the wait before any output appears. 

• Time-per-output-token, or inter-token latency, measures the decode phase, or how quickly text streams once it starts. 

• Throughput measures aggregate capacity in tokens or requests per second. 

The most important thing to remember is separating single-user latency from batched throughput, because optimizing one may come at the expense of the other. 

Another important metric is tail latency. Median latency describes a typical request, but a better gauge for interactive services live is p99, the slowest 1%, because that is what users notice and what service-level agreements are written against. 

Tail latency has specific causes worth watching: 

• A continuous-batching scheduler that interrupts in-flight decode to admit a new prompt introduces preemption spikes. 

• Cold starts are their own category: bringing a fresh GPU replica online can take several minutes depending on orchestration, container image size, model size, and caching, and even a warm engine may need tens of seconds to initialize a model. 

Averages don’t show these, which is why they are the wrong thing to optimize for interactive workloads.

Metric choice also determines whether autoscaling works well. GPU utilization alone often isn’t a good indicator for scaling, because if memory fills up before processing is full, a utilization-triggered scale-out may happen too late, so the server runs out of memory before new capacity arrives. Scaling on queue depth or time-to-first-token tracks the resource that tends to be the bottleneck. 

Production platforms handling large, spiky traffic address this with load-aware routing and capacity abstractions that account for the non-uniform cost⁵ of requests, where a few long-context requests load a server more than many short ones. Measuring the resource that runs out first keeps a serving system stable under load.

Where this leaves you

LLM inference is a resource-management problem, and the resource that most often scales worst is memory.

• Speed diagnostics are determined by whether the model fits. 

• Sizing resolves to weights plus a KV cache that grows with context and concurrency. 

• Hardware choice leans on bandwidth over processing for single-stream decode. 

• Engine and cost decisions matter by how efficiently a fixed pool of memory gets shared across requests. 

Memory is not the whole story, because networking, scheduling, batching strategy, and parallelism affect performance too, but it affects most decisions.

The KV cache is moving off individual GPUs and toward shared, disaggregated storage, because managing it as local scratch space stops working once context windows and concurrency grow. Prefill-decode disaggregation, fleet-wide cache tiers, and cache-aware routing are early forms of treating inference memory as distributed system state rather than a per-GPU concern.

That changes what the storage layer has to do. Systems that externalize inference state need storage designed for predictable latency under high concurrency, because a cache lookup that stalls is a token that does not stream. 

That is where high-performance distributed data infrastructure becomes important. When the KV cache becomes a shared state that has to be stored, retrieved, and served with tight latency, Aerospike is built for the predictability-under-load that shared inference state demands.