Context engineering for AI agents: Where should memory and state be stored?

An AI agent calls a model dozens of times per task. Between those calls, its progress, memory, and working files have to be stored somewhere. Where that information is stored determines what each step costs, how fast it runs, and whether a task survives a failure. The answer starts with context engineering, the practice of deciding what reaches the model at each step.

Quick answer

Store an agent's state in reliable storage that keeps replicated copies of the data. Keep this storage separate from the model's context window, or the information the model considers at one time, and organize the data by how long it needs to be retained.

Per-task progress needs a store that saves a checkpoint at each step and gets rid of it when the task ends. But facts that persist across sessions need a store of record that replaces outdated facts, because similarity search alone cannot tell which of two stored facts is current. Large files and tool results can sit in cheaper storage and be referenced by key. For interactive or high-concurrency agents, that storage also needs predictable latency at the 99th percentile (p99).

What is context engineering?

Context engineering is the design of the system that assembles an AI model's input for each step of a task. That input includes the system prompt, the user's message, conversation history, retrieved documents, tool definitions, tool outputs, and stored memory. Context engineering matters most for AI agents, which call a model many times per task and rebuild the input on every call.

Context engineering and state management are related but separate concerns. Context engineering decides what the model receives at a given step, while state management decides what the application keeps between steps. When information is moved out of the context window to save space, external storage keeps it available for later steps.

Teams are starting to store context as maintained files, but the practice is early. A study of 10,000 actively maintained open-source repositories, Context Engineering for AI Agents in Open-Source Software^[1], found that only 466, under 5%, had adopted AI context files such as AGENTS.md or CLAUDE.md, and that those files varied widely in content and phrasing. For a team starting now, there is no settled convention to follow, so the structure of context files is still a design decision.

How is context engineering different from prompt engineering?

Prompt engineering covers the wording and structure of instructions. Context engineering covers everything that reaches the model alongside those instructions, most of which code assembles at runtime. Prompt engineering is one part of context engineering.

A customer support agent handling a refund shows the difference. A person writes the agent's instructions once. Code assembles the customer's account record, earlier tickets, the refund policy, the refund tool's definition, and the last tool result, and that assembly runs again on every turn.

Clear instructions and well-organized project notes are part of context engineering. A specification is written once and read when needed, while an agent's context is rebuilt from changing sources every time the model runs, so most of the work lies in the system that rebuilds it. Prompt engineering alone is enough for single-turn features with no stored state, but once an application keeps state across turns, calls external tools, or retrieves documents, output quality depends on context engineering.

What counts as context in an AI agent?

Context is everything an AI model receives in one call, whether a person wrote it or code inserted it. In AI agents, context usually comes from eight components that differ in who produces them and how often they change. Tool definitions often come from model context protocol (MCP) servers, which give an agent access to an external system's tools through a standard interface.

Anthropic's guidance on effective context engineering recommends keeping each component lean^[2]: system prompts specific enough to guide behavior without hard-coding a rule for every case, tools with little overlap in what they do, and a few representative examples instead of a long list of edge cases.

System prompts, output formats, and tool definitions change only when someone updates the agent or connects a different tool server. Conversation history, retrieved documents, tool outputs, and long-term memory accumulate while the agent runs, so these four grow with every turn and are the first place to look when results degrade. Not all of them need to be kept, but anything a later step may need has to be stored outside the window.

Why do AI agents get worse during long sessions?

AI agents get worse during long sessions because model accuracy declines as input length grows, a pattern that Chroma, a company that builds an open-source vector database, named “context rot^[3]” in a 2025 study. Context rot starts well before the context window is full, and its severity varies by model, task, and the kind of material filling the window.

Anthropic offers a partial explanation: a model has a limited amount of attention^[2] to spread across its input, so each added token leaves less for the parts that matter. The size of the effect still varies by model and task, which one mechanism does not fully explain.

Chroma's context rot study tested 18 models, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, across 194,480 calls, and performance declined as input length grew in every experiment. In a conversational memory test, models answered more accurately from a focused prompt of about 300 tokens than from the full history of about 113,000 tokens. Distractors that resembled the right answer did more damage as inputs grew.

Context overflow occurs when input exceeds the context window's hard limit, so the request fails or gets truncated. Context rot is less obvious: the call succeeds, but the answer is slightly worse.

How to diagnose context rot in an agent

1. Rerun the failing step with only the instructions, the current task, and the facts that step needs.

2. Compare the result with the original run.

3. If it’s more accurate, then length and noise likely contributed to the failure, and trimming, compression, or retrieval should help.

4. If it’s still inaccurate, look for a missing fact or a flawed instruction instead.

What are the main types of context failure?

Five situations account for most context failures^[4]. 

In Google DeepMind's Pokémon-playing Gemini agent, hallucinated game state contaminated the agent's goals^[5] and took a long time to undo. For context confusion, how distinct the tools are matters more than how many there are: Anthropic reports that the most common tool-use failures are wrong tool selection and incorrect parameters^[6], especially when tools have similar names, such as notification-send-user and notification-send-channel.

Context clash is the most well-known. Microsoft and Salesforce researchers found that splitting one instruction across several turns cut performance by 39% on average^[7] across 15 models, because early, incomplete answers stayed in the context and the models built on them.

Researchers studying multi-turn agents describe conversational inertia^[8] as strong attention to a model's own previous responses, with a stronger effect in longer contexts. Manus, a company that builds a general-purpose AI agent, counters a related effect^[9] by adding small, structured variation to repeated actions so the agent does not fall into a fixed rhythm.

How much context do tool definitions use?

Tool definitions use tens of thousands of tokens before an agent does any work. Every tool has a definition made up of a name, a description, and an input schema, and those definitions are sent with every request, usually near the start of the context. Connecting an MCP server adds the server's full tool catalog to that fixed overhead.

In Anthropic's Tool Search Tool comparison^[6], loading definitions for more than 50 MCP tools upfront used about 77,000 tokens before any work began. Loading only the search tool and the three to five tools a task needed used about 8,700 tokens, an 85% reduction. Anthropic recommends considering tool search when tool definitions exceed about 10,000 tokens, when more than 10 tools are available, or when tool selection accuracy suffers. Token counts depend on schema size, description length, and the model's tokenizer.

The two common approaches treat the start of the prompt differently. Model providers cache the processed start of a prompt and reuse it on later calls, but only while that start stays identical. Anthropic's Tool Search Tool leaves deferred tools out of the initial prompt and adds them only after a search, so the system prompt and core tool definitions stay cacheable^[6]. Manus keeps the full tool catalog in context^[9] and restricts which tools the model may choose at each step, which pays the full definition cost on every call but never changes the tool list.

Tool outputs add a second cost. Web pages, PDFs, and log files returned by external tools can exceed everything else in the window combined. Manus drops a web page's content from the context while keeping its URL, so the agent can fetch the page again if a later step needs it.

Choosing a tool-loading approach

To audit an agent, count the input tokens of a request that contains only the system prompt and tool definitions. If that number is a big part of the context window, tool overhead competes with the task before work even starts.

What techniques control what enters the context window?

Five groups of techniques control what enters an agent's context window: offloading, on-demand retrieval, compression, isolation in subagents, and staged loading of instructions. The first three depend on storage outside the window; the latter two change what each model call sees.

Offloading writes material to a file or database and leaves a reference in the context. LangChain's Deep Agents harness writes tool results above about 20,000 tokens to a file^[10] and shows the model only the first and last few lines plus the path. Manus has its agent maintain a todo.md file^[9], which places current goals at the end of the context, where recent attention is strongest.

On-demand retrieval keeps lightweight identifiers in context, such as file paths, stored queries, and links. Claude Code combines upfront and runtime loading^[2]: CLAUDE.md files enter the context at the start of a session, while glob and grep tools find other files as the task requires. LangChain notes that semantic search doesn’t work well for code and technical API references, where the text doesn’t have a lot of meaning^[11] and exact-match search finds the right file more directly.

Compression includes compaction, which summarizes older history, along with trimming and clearing old tool results. Anthropic describes compaction as the first step^[2] for coherence in long-running agents. Deep Agents triggers summarization at 85% of the window by default and writes the summarized messages to a file^[10] to read them back later. Manus designs its compression to be restorable^[9]: a web page's content can leave the context as long as its URL stays. Compaction cannot repair a poisoned history, because a summary repeats the error in fewer tokens and gives it a larger share of what the agent sees.

Isolation hands focused work to a subagent with its own window. According to Anthropic, subagents may use tens of thousands of tokens^[2] while returning condensed summaries of 1,000 to 2,000 tokens to the lead agent.

Staged instruction loading, also called progressive disclosure, keeps procedures out of the window until a task calls for them. In the Deep Agents skills implementation, full skill instructions load only when the model decides a skill applies^[10]. Thresholds such as these come from each framework's documentation and can change between releases.

When should an agent compact, clear, or start over?

An agent should compact when its goals haven't changed and older history has served its purpose, clear old tool results when they can be fetched again, and start a new thread when it keeps returning to an error. The deciding factor is reversibility: whether the original information can be recovered afterward.

Anthropic's guidance matches techniques to types of^[2] tasks: compaction suits tasks with extensive back-and-forth, structured note-taking suits iterative work with clear milestones, and multi-agent designs suit research where parallel exploration pays off.

How to choose

• Compact when the goals are unchanged, and most of the history has done its job.

• Clear tool results when old outputs make up most of the window and can be fetched again.

• Start a new thread when the agent keeps returning to an error or an abandoned approach, because models tend to build on their own early answers. Write down the verified decisions and open questions first, leave the incorrect information out of that brief, and correct any memory or stored state that supplied it.

Each of these options is safer when the original material is stored. A team that keeps originals can compact earlier and more often.

When should you use RAG instead of long context?

Retrieval-augmented generation (RAG) tends to suit sources that are bigger than the context window, fragmented or conversational content, and high query volumes where per-query cost matters. Long-context prompting tends to suit one coherent document that fits comfortably in the window. Retrieval latency, how often documents change, cache reuse, grounding requirements, and retrieval quality also affect the choice.

RAG inserts only the passages a retriever selects, while long-context prompting loads the whole source into the window. A 2025 evaluation, Long Context vs. RAG for LLMs: An Evaluation and Revisits^[12], found that long-context models were generally more accurate on self-contained material such as stories and encyclopedia-style questions, while RAG did better on dialogue and fragmented information. The LaRA benchmark, which tested seven open-source and four proprietary models, concluded that the better choice depends on model size, long-text ability, context length, task type, and the retrieved chunks.^[13]

Cost is another factor to consider. One 2026 study of machine-safety questions found long-context prompting more accurate than semantic RAG but 26 times as expensive in tokens per query^[14]. Because the study tested only two small models, its results suggest a tradeoff between accuracy and cost, but more research is needed to determine the size of the tradeoff across different models and tasks.

How to choose

• Choose long-context prompting for low-volume analysis of one coherent document that fits in the window.

• Choose RAG for agents that answer many requests a day over large or fragmented sources, after confirming that retrieval quality and retrieval latency meet the task's needs.

• Choose a hybrid when both apply: retrieve to narrow the source, then load the narrowed set whole.

What is the context state layer?

The context state layer is the set of stores that hold an AI agent's information outside the model's context window. It has four parts, each with different storage requirements:

• Working context: the tokens in the current model call, rebuilt on every call

• Session state: the record of one task or conversation, including history and checkpoints of the agent's progress

• Long-term memory: facts, preferences, and learned procedures that carry across sessions

• Artifacts: large items an agent produces or collects, such as offloaded tool results, documents, and plans

For example, LangGraph splits persistence into checkpointers and stores^[15]: Checkpointers save one thread's state for conversation continuity, human review, and fault tolerance, while stores hold cross-thread data such as user preferences and shared knowledge. The same documentation warns that in-memory checkpointers lose every checkpoint when the process restarts.

Other systems store long-term memory and artifacts as files. Anthropic's memory tool^[2] uses a file-based system for long-term knowledge, and Manus treats the file system as persistent, external context^[9]. A 2026 study, Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads^[16], found that the kind of storage system underneath determines how much memory the agent uses, what updates cost, whether indexes must be rebuilt, and which retrieval operations are available.

Context state requirements matrix

Session state and long-term memory both get called "memory," but they have different methods of access, lifetimes, and problems. Lost session state is visible because a task has to restart. Stale long-term memory isn’t obvious; the agent keeps running but answers from facts that are no longer true. Filling in the matrix with steps per task, concurrent sessions, required session lifetime, and the components allowed to write memory helps you figure out the storage requirements.

Is a vector database enough for agent memory?

No. A vector database handles the retrieval step of agent memory, but long-term memory also needs revision, expiry, and conflict handling. Similarity search alone cannot tell which of two similar facts is current.

Here’s an example from a support agent. A customer says in March that they are on a team plan, then upgrades to an enterprise plan in June and says so. Both statements embed close together, so a later question about the customer's plan retrieves both, and the similarity score doesn’t say which one replaced the other. If the agent answers from the March statement, it applies team-plan limits, such as a lower seat count, to a customer who is paying for the enterprise plan, and that causes problems. 

Research on agent memory shows the same problem. The 2026 paper Is Agent Memory a Database?^[17] identifies four recurring failures in current memory systems: unregulated growth, missing semantic revision, forgetting caused by capacity limits instead of relevance, and retrieval that doesn’t update what it reads. The authors propose ingestion, revision, forgetting, and retrieval as the four operations long-term memory needs.

Some memory designs handle revision by keeping history. Bitemporal knowledge graphs such as Zep's Graphiti^[18] retain superseded facts and mark them as replaced. LangGraph's long-term store is organized around namespaces and keys, with semantic search as an option^[19] enabled on top.

Four questions to test a memory design

1. How do new facts enter memory?

2. How are outdated facts replaced?

3. How do items expire?

4. How are items found?

A memory design that considers only the will drift as the facts behind it change. A common method pairs a store of record, with explicit revision and expiry, and a similarity index over that store for fuzzy recall. Many vector databases offer metadata filtering, versioning, or expiry features that cover part of this list; the distinction is between retrieval capability and the full set of memory-management operations, whichever product provides them.

Revision also needs a rule for concurrent updates. When two agents update the same fact, then conditional writes, version numbers, transactions, or explicit conflict resolution decide which value stands, and earlier values belong in a history kept separate from the current value.

Which database should store agent memory and state?

The right database for agent memory and state depends on which state layers the agent stores, the per-step latency target, expiry requirements, failure behavior, and the number of concurrent writers. In-memory storage or local files are enough for development and single-user agents. Production agents need a durable, replicated store with expiry and concurrency control, plus a similarity index when they need semantic recall.

How to evaluate a database for agent state

• Durability: Does an acknowledged write survive a process crash or restart?

• Availability and replication: What replication factor and failover behavior does the store offer, and what happens to a checkpoint write during a network partition?

• Latency: What p99 read and write latency per agent step does the product need, at expected concurrency and at the record sizes the agent writes?

• Expiry: Can sessions and stale memory expire without separate cleanup jobs, and do retention rules cover resumable tasks and audits?

• Concurrency and consistency: How many agents or workers update the same records, and do those updates need conditional writes or transactions?

• Recovery: How is state restored after a failure, and can a task resume from its last checkpoint?

• Query model: Does the agent need key lookups, filters, similarity search, or a mix?

• Cost: What do storage and operations cost at expected concurrency, including the engineering time to run the system?

The agent framework ecosystem supports a range of backends. LangGraph's checkpointer integrations^[20] include Aerospike, CockroachDB, in-memory, and Redis options, and AWS publishes DynamoDB and Valkey checkpointers^[21] in its langgraph-checkpoint-aws package. LangGraph's documentation also shows PostgreSQL-backed checkpointers and stores^[19], and MongoDB Atlas has integrations for short-term and long-term memory^[22]. Memory frameworks such as Letta, Mem0, and Zep add logic for extracting and updating memories to an underlying storage system.

Aerospike for agent state

Aerospike publishes a checkpointer, langgraph-checkpoint-aerospike, and a long-term store, langgraph-store-aerospike, both documented on LangChain's Aerospike integration page^[23]. Aerospike's agent-memory tutorial includes forking a task from a saved checkpoint after an agent takes a wrong path.

Both the checkpointer and the long-term store can run on one Aerospike cluster, but they usually belong in separate namespaces, because Aerospike configures storage and retention per namespace. Checkpoints are large, written often, and short-lived, while memory records are small and kept for months.

Checkpoints written through the Aerospike checkpointer can carry a per-write time-to-live. Expiration makes a record invisible to clients once its TTL passes, while eviction removes the records nearest to expiry early when storage crosses a configured threshold; defragmentation later reclaims the space. TTL support is off by default and is enabled per namespace.

By default, Aerospike writes to every replica before acknowledging a write and flushes buffered writes to the device at an interval of one second unless configured otherwise. Acknowledged writes can be lost only if the number of nodes that shut down unexpectedly before the data is flushed equals or exceeds the replication factor. The commit-to-device option removes that risk by flushing every write before acknowledging it, at a performance cost for write-heavy workloads.

Aerospike also supports revision mechanisms for long-term memory needs. A write can be conditioned on a record's generation, so it fails if another writer changed the record after it was read, and Aerospike Database 8.0 added distributed ACID transactions across records in the same namespace. Those transactions build on Aerospike's strong consistency mode, which has been validated externally with Jepsen testing and is available in Aerospike Database Enterprise Edition and Aerospike Cloud. In its availability mode, Aerospike provides eventual consistency instead.

Aerospike does not offer vector search, so a design that needs semantic recall over memory pairs Aerospike with a separate similarity index. Aerospike's data model centers on primary-key and indexed access, so large exploratory analytical scans are usually better served by a columnar analytics system.

How to choose

• Keep PostgreSQL when the team already runs it, and the state-layer matrix shows modest step counts and concurrency.

• Choose a low-latency store with native expiry when many concurrent agents read memory and write checkpoints on every step.

• Add a similarity index when the agent needs semantic recall, and keep revision and expiry in the store of record.

Does database latency matter for AI agents?

For one user running one long task, database latency rarely matters, because model calls take longer than storage operations. Database latency matters for interactive and voice agents, high-concurrency services, multi-agent fan-out, and any step that waits on a memory read before calling the model. Storage failures matter for every agent.

Estimate storage time per task as the number of steps multiplied by the storage time per step, where storage time per step is reads per step times read latency plus writes per step times write latency. The estimate assumes operations run one after another; overlapping or batched operations shorten the duration, while retries and queueing lengthen it. Manus reports that a typical Manus task takes around 50 tool calls^[9], and LangGraph saves checkpoints^[15] as a graph runs, so at least one write per step is common.

Average latency understates what a task experiences. Tail delays compound when one interaction makes many requests. If each operation has a 1% chance of exceeding the store's p99 latency, the chance that a task includes at least one such operation is 1 − 0.99ⁿ for n operations, whether those operations run in sequence or in parallel.

How much a slow operation costs depends on how long each model step takes. Three storage operations at a 50-millisecond p99 add 150 milliseconds to a step. Besides a reasoning step that runs for 20 seconds, which is under 1%; in a voice agent with a turn budget of a few hundred milliseconds, it can use half the budget. Four situations make storage latency matter:

• Interactive and voice agents: Reads that must finish before the model call add to response time

• Sustained concurrency: At a few thousand storage operations per second, throughput is rarely the first constraint; tail latency under sustained load usually is, so test at the agent's own record sizes

• Multi-agent fan-out: A lead agent cannot continue until its slowest subagent returns, so tail latency on shared-memory reads sets the pace

• Dependent reads: When a memory read decides what the next model call contains, the read cannot overlap with anything else

How to measure storage latency for an agent

1. Count steps per task and storage reads and writes per step.

2. Run the storage layer at expected peak concurrency.

3. Record p99 latency for reads and writes at the record sizes the agent writes; large checkpoint blobs and small memory records behave differently on any storage engine.

4. Measure end-to-end task latency at p95 and p99 as well, then compare the results with the per-turn response budget.

Storage chosen for convenience and durability is enough when a handful of users run long tasks. When the workload involves many concurrent sessions, interactive response targets, or fan-out, per-step p99 latency belongs in the requirements.

What happens to an agent when the state store fails?

When the state store fails mid-task, the agent loses whatever progress was not durably checkpointed. An agent with a durable checkpoint at step 39 loses one step if it fails at step 40. An agent whose checkpoints were acknowledged but never persisted loses every step since the last one that was persisted, which can be most of the task.

Three properties determine what a failure costs:

• Durability: Whether a write the store has acknowledged survives a process crash or restart. Under Redis's default snapshot persistence^[26], a crash loses writes made since the last snapshot.

• Availability and replication: Whether the store keeps serving reads and writes when a node fails. Aerospike namespaces are often configured with a replication factor of 2, a master copy and one replica, so the data survives the loss of one node.

• Partition behavior: What happens to a checkpoint write when nodes cannot reach each other. A store can reject the write, delay it, or accept it and lose it later. In Aerospike's strong consistency mode, acknowledged writes are not lost to partitioning events, and consensus-based stores such as CockroachDB require a majority of replicas to communicate before committing.

Resuming has its own cost. LangGraph checkpointers support fault tolerance by resuming from saved checkpoints, but a step re-run after a resume may repeat its tool calls, so tools with side effects need idempotency keys or a check before acting. For long or expensive tasks, test failure as well as latency: stop a storage node mid-task and confirm that the agent resumes from its last checkpoint without repeating side effects.

How do you stop agent checkpoints from growing without limit?

Agent checkpoints stop growing without limit when a retention policy controls checkpoint granularity, artifact size, per-thread expiry, and checkpoint history. LangGraph's persistence documentation lists unbounded checkpoint growth as a known issue and recommends pruning old checkpoints periodically or setting a retention policy.

Checkpoint storage grows with the number of steps, the number of sessions, and the size of each saved state. Without delta storage, a conversation's full message list is re-serialized into every checkpoint^[27]. ADeltaChannel option stores only the new messages at each step, at the cost of extra work when a checkpoint is read back. When the full list is saved every time, cumulative bytes written over a session grow roughly with the square of the number of turns, so a session twice as long writes about four times as much. Retention, deduplication, and compaction reduce the live storage it uses.

Four controls for checkpoint growth

1. Choose checkpoint granularity, and check whether the checkpointer stores deltas or full state. LangGraph lets subgraphs write to the parent checkpoint^[15], which reduces write volume for short-lived subtasks.

2. Store large artifacts by reference. A checkpoint that holds a file path or record key instead of a 20,000-token tool result stays small.

3. Set expiry per thread. A time-to-live set when the checkpoint is written removes expired sessions without a separate cleanup job. Lifetimes should cover resumable tasks, human approval pauses, and audit requirements, and artifacts referenced by a live checkpoint should not expire before that checkpoint does.

4. Keep only the history the product uses. If the product never rewinds beyond the last few steps, older checkpoints in a thread add cost without doing anything. 

Long-term memory needs a different kind of forgetting. Session checkpoints can expire on a timer because sessions end, while memories outlive sessions and should leave when they are superseded or no longer relevant. The authors of Is Agent Memory a Database?^[17] list capacity-driven forgetting as a recurring failure: items disappear when space runs out, whether or not they are still true.

How does prompt caching change context design?

Prompt caching rewards contexts with a stable prefix and a mostly append-only history, because providers that cache by prefix reuse cached work only for an identical prefix. Any change near the start of the context invalidates the cache for everything after it.

An AI agent may have three caches:

• Prompt cache (KV cache): The model provider reuses work already done on an identical prefix of the input

• Response cache: The application returns a stored answer when a new query matches an earlier one, exactly or semantically

• Memory cache: The application keeps frequently read state close to the agent to avoid a store round trip on every step

The prompt cache has the largest effect on agents because agent inputs are dominated by context. Manus reports an average input-to-output token ratio of about 100:1 and calls KV-cache hit rate the most important metric for a production agent^[9], citing cache d input at a tenth of the uncached price for Claude Sonnet; that ratio varies by provider, model, and pricing schedule.

Anthropic's prompt caching^[28] matches the full prefix in a fixed order: tools first, then the system prompt, then messages. As of October 2026, the default cache lifetime is five minutes, refreshed each time the cache is used, with a one-hour option at additional cost. These rules are specific to Anthropic's API, and other providers use different cache keys, lifetimes, and pricing. Manus lists three common ways teams break their own cache: a timestamp at the top of the system prompt, edits to earlier turns, and JSON serialization that doesn't keep keys in a stable order.

Those rules have three consequences for context design:

• Compaction has a cache cost: Rewriting the history invalidates cache reuse from the first changed token onward, while an unchanged earlier prefix, such as the tools and system prompt, stays reusable. Frequent small compactions cost more than occasional large ones.

• Human approval steps need a cache plan: An agent that pauses longer than the cache lifetime returns to a cold cache, which makes the one-hour option worth evaluating.

• Dynamic tool lists change the prefix: Adding or removing tools mid-session changes the start of the context.

Prompt caching checklist

1. Put stable material, such as tools and the system prompt, at the front of the context.

2. Prefer appending new turns to editing earlier ones, and measure cache-hit rate and cost.

3. Serialize structured data deterministically.

4. Compact in larger batches.

5. Size the cache lifetime to the longest expected pause.

A memory cache raises a separate question. A cache in front of the memory store adds a second copy of every record, and that copy has to be invalidated whenever memory is revised. When the store of record already serves reads within the per-step latency budget, which low-latency operational databases such as Aerospike are designed to do, you don’t need a separate cache. 

How should multiple agents share context?

Multiple agents should share the minimum sufficient state: verified findings, their provenance, assumptions, decisions, open questions, and references to full traces. Full traces stay available for audit and retrieval rather than being included in every agent's context, and write actions should stay with agents that see each other's decisions. Parallel subagents work best for read-heavy research, where each subagent returns a condensed result to one decision-maker.

Cognition, which builds the Devin coding agent, made the case for caution in June 2025. Don't Build Multi-Agents^[29] shows how subagents that lack each other's context produce parts that don't fit, and recommends a single-threaded linear agent as the simplest design.

In Multi-Agents: What's Actually Working^[30], published in April 2026, Cognition reports deploying multi-agent systems that work in practice, while repeating that agents should share as much context as possible and noting that decision-making fragments when several agents take write actions.

In a June 2025 post on its multi-agent research system^[31], Anthropic reported that a system with Claude Opus 4 as the lead agent and Claude Sonnet 4 subagents outperformed single-agent Claude Opus 4 by about 90% on an internal research evaluation that Anthropic has not published. The same post reports that multi-agent systems used about 15 times as many tokens as chat interactions, and that they excel at tasks with heavy parallelization, information that’s bigger than a context window, and many complex tools.

A shared knowledge store for multiple agents has four storage requirements:

• Consistency: Every agent reads the latest version of a shared decision.

• Concurrent writes: Conditional updates prevent one agent from overwriting another's change.

• Trace storage: Each agent's actions and reasoning are kept where authorized agents and reviewers can retrieve them, with access scoped by task and permission.

• Expiry: Shared task state is removed when the task ends.

Parallel subagents suit subtasks that are independent and mostly read-only. Write actions belong in one agent, or in agents that share decisions and references to full traces through a common store.

How do you debug and evaluate agent context and memory?

Debug agent context by logging the full assembled input for each model call and comparing good runs with failed ones. Evaluate agent memory by testing whether the agent retrieves the current, correct fact, in addition to how fast retrieval runs.

Debugging an agent's context

1. Log the complete input for each model call, with a token count for each context component.

2. Compare the logged context of a failed run with a successful run of the same task.

3. Replay the failing step from a saved checkpoint. LangGraph checkpointers support time travel and fault tolerance^[15], which makes a failing step reproducible.

4. Run the agent in a controlled simulation. Anthropic built simulations in its Console with the exact prompts and tools from its research system^[31] and observed each step as the agents worked.

Memory evaluation needs tests that match how memory fails. The LongMemEval benchmark, used in Chroma's context rot study^[3], includes question types for knowledge updates, temporal reasoning, and multi-session recall.

Memory writes need their own tests, because errors introduced at write time persist across sessions:

• Incorrect writes: Store a wrong fact and confirm it is rejected, flagged, or later corrected.

• Conflicting updates: Have two agents update the same fact concurrently, and confirm the documented conflict rule decides the result.

• Deletion and expiry: Confirm deleted or expired items stop appearing in retrieval.

• Provenance: Confirm that each memory records where it came from.

• Isolation: Confirm one user's or tenant's memory doesn’t appear in another's context.

A memory system can pass a latency test while still returning superseded facts, so test accuracy on updated in-memory evaluation.

How does context engineering relate to harness engineering?

Harness engineering is the broader discipline of designing everything around a model that lets an agent work reliably, and context engineering is one of its core parts. An agent's harness covers instructions, tools, the execution loop, feedback mechanisms, and state handling.

OpenAI described this in its February 11, 2026 post on harness engineering^[32]. When Codex agents wrote the code, the OpenAI team's main job became designing environments, specifying intent, and building feedback loops for those agents. The post names context management as one of the biggest challenges in making agents effective on large tasks, and describes moving from one large AGENTS.md file to an AGENTS.md that works as a table of contents.

OpenAI developer experience engineer Gabriel Chua describes Codex as a model plus a harness plus surfaces^[33], with the harness defined as the collection of instructions and tools. LangChain's Deep Agents documentation describes an agent the same way^[10]: a model combined with a harness.

Final recommendation

• Keep the working context small and stable

• Store everything moved out of the window somewhere durable and retrievable

• Treat per-step storage latency as a requirement only where concurrency, interactivity, or fan-out need it 

• Test what happens to a task when the state store fails

The state-layer matrix, the database criteria, the tail-latency arithmetic, and a failure test give a team what it needs to make these choices with numbers from its own workload.