Context Management for AI Agents: Retrieval, Memory, Compaction, Tool Results, and Long-Running Sessions

Context Management for AI Agents: Retrieval, Memory, Compaction, Tool Results, and Long-Running Sessions

Context management is the discipline of deciding what information an AI agent should see now, what should stay outside the prompt, what should be retrieved later, and what must survive when a long-running session is compacted.

This is different from simply giving a model the largest possible context window.

A useful context is:

• relevant
• fresh
• trustworthy
• compact enough to preserve attention
• complete enough to avoid dangerous assumptions

Anthropic's context-engineering guidance describes context as a finite resource whose utility should be optimized, while modern long-running agent platforms increasingly use native compaction to preserve important state across context windows.

Think in layers of context

A production agent can draw from several layers:

system/developer instructions
current user request
recent conversation
retrieved files/docs
memory/preferences
live tool results
workspace/runtime state
compacted prior session state

These sources have different freshness and authority.

A live database result should normally override an old conversation summary about the same record.

Keep hard instructions separate from data

System/developer instructions define behavior and constraints.

Retrieved documents/tool responses are data, and may contain untrusted text.

Do not allow a file, webpage, email, or tool result to silently replace higher-level rules.

For agents processing arbitrary external content, treat that content as potentially hostile input rather than executable instruction.

Retrieve only what the task needs

A user asking about one section of a contract does not need the model to receive every file in the workspace.

Use:

• semantic search to find likely evidence
• exact search for identifiers/headings
• metadata filters
• targeted range reads

Then expand around the relevant hit only when more context is needed.

This improves information density and reduces distraction.

Long context can create context rot

More tokens are not always more useful.

A long session can accumulate:

• stale plans
• repeated tool outputs
• superseded values
• irrelevant side discussions
• failed approaches

If all of this remains in the prompt forever, the model must continually distinguish current state from obsolete state.

Good context management removes or compresses information that no longer affects the task.

Preserve invariants, not narration

When summarizing/compacting a long task, preserve things that change future decisions:

user goal
hard constraints
exact IDs
current versions/timestamps
completed milestones
chosen architecture decisions
unresolved failures
next action

You can usually discard:

• repetitive status narration
• old search attempts
• intermediate wording drafts
• reasoning that no longer affects state

The goal is a continuation state, not a transcript archive.

Compaction extends long-running sessions

OpenAI's current Agents/Responses platform supports native context compaction for long-running workflows.

Recent OpenAI engineering material describes compaction as preserving key prior task state while removing extraneous context as sessions approach context limits.

The general design principle applies even if you implement compaction yourself:

observe context pressure
select important durable state
compress old interaction history
continue with compact state + recent high-value context

Do not wait until the context window is already exhausted.

Compact at meaningful milestones

Compacting every turn wastes effort and can repeatedly distort summaries.

Good times include:

• after completing one project phase
• after a large tool/research phase
• before switching to a new subtask
• when context usage approaches a configured threshold

A compaction item should summarize stable state after the milestone, not an uncertain mid-operation state if avoidable.

Do not lose concurrency/version guards during compaction

Some details look small but are safety-critical:

edit only using latest ETag
expected_updated_at = exact timestamp
next cursor = opaque value
current branch = release/xyz
protected IDs = ...

If compaction removes these, the agent can perform an unsafe write later.

Treat mutation preconditions as durable task state.

Memory and context are different

Context is what the model can use in the current inference.

Memory is a mechanism for storing selected information across future sessions and retrieving/synthesizing it later.

Not every chat detail belongs in persistent memory.

Good persistent memory tends to include:

• stable preferences
• long-running project facts
• recurring constraints

Temporary operation state such as a one-time cursor normally belongs in the current workflow/session, not a long-term user profile.

Freshness matters for memory

Stored information can become stale:

old job title
old software version
old project status
old user preference

OpenAI's 2026 memory work explicitly highlights freshness, continuity, and relevance as core problems for long-horizon memory systems.

A memory system needs mechanisms to:

• update/replace stale facts
• retain provenance when useful
• avoid presenting conflicting old facts simultaneously

“Remember everything forever” is not a good memory architecture.

Keep memory scoped to the right identity/domain

An enterprise agent may serve:

• several users
• tenants
• projects
• workspaces

Memory keys and retrieval must enforce those boundaries.

Never retrieve one tenant's private project context into another tenant's session because embedding similarity happened to match.

Authorization must be applied before semantic relevance.

Tool search reduces tool-definition context

Large agents can have hundreds of tools.

Loading every tool schema into every prompt consumes tokens and may make selection harder.

OpenAI's current Agents API describes tool search that loads relevant tool definitions as needed.

A general implementation can:

1. maintain a tool catalog
2. retrieve/rank tools from task intent
3. load only likely tools plus core always-available tools
4. allow fallback search if none fits

This keeps the context budget focused on the actual work.

Tool outputs should be filtered before reaching the model

A database/export may return thousands of rows.

Do not automatically paste them all into context.

Use deterministic code/tools to:

• filter
• aggregate
• sort
• paginate
• calculate summaries

then bring only the evidence requiring model judgment into context.

OpenAI's current agent guidance similarly emphasizes programmatic tool calling for filtering/combining results outside the model context.

Preserve raw artifacts outside the prompt

Large files, datasets, images, or generated code should often live in a workspace/filesystem rather than being repeatedly embedded in messages.

Context can contain:

artifact ID/path
summary
relevant excerpt

The agent reads more only when needed.

This turns external storage into an extension of working memory without paying the prompt-token cost every turn.

Summaries should be structured enough to resume

A useful project-state summary might include:

Goal
Constraints
Completed
Current authoritative state
Open issues
Next step
Source/artifact references

Free-form summaries can accidentally omit IDs or mix old/new decisions.

Structure helps both the model and humans resume safely.

Recent conversation still matters

Compaction/memory should not replace keeping the immediately relevant recent turns.

Users frequently refer to:

that last example
the second option
make it shorter
use the same title

Recent local context resolves these references better than a long-term project summary.

A strong context manager combines compact durable state with a recent-window buffer.

Separate working notes from final output

During long tasks the agent may maintain internal/project state such as:

• checklist
• IDs completed
• test results
• research notes

Keep these in a structured workspace/state store when possible rather than repeatedly narrating them into conversation.

This improves context efficiency and creates resumability after failures.

Retrieval needs provenance

When context comes from search or memory, keep metadata about where it came from:

source file/page
URL/document ID
timestamp/version

This lets the agent:

• cite evidence
• reopen the right source
• distinguish current vs historical data

A summary with no provenance can become difficult to verify later.

Re-fetch changing state before mutation

Do not trust context memory for state that can change concurrently.

Before editing an external object:

fetch latest record/version
apply mutation with expected version
verify result

The session may “remember” an older timestamp, but correctness requires the current authoritative value.

Context optimization must never weaken concurrency safety.

Use priority tiers when context is tight

When the context budget is limited, keep information in roughly this order:

Tier 1 — must preserve

• instructions/safety
• current user request
• mutation guards
• authoritative current state

Tier 2 — strongly useful

• relevant evidence
• recent tool results
• project decisions

Tier 3 — optional

• historical discussion
• superseded alternatives
• verbose tool logs

This prevents token pressure from removing the exact details the agent needs for safe actions.

Evaluate context management separately

Build evals for failures such as:

• important constraint lost after compaction
• stale memory overrides live tool data
• irrelevant retrieval distracts answer
• protected tenant data retrieved incorrectly
• tool result truncated without agent noticing
• agent repeats completed work after resume

The final-answer quality alone may not reveal why a long-running task regressed.

Context-management metrics

Useful metrics include:

• input tokens per turn
• retrieved chunks/tool-result tokens
• compaction frequency
• context-window utilization
• retrieval precision/coverage
• repeated tool calls
• resume success rate
• stale-memory corrections
• lost-constraint failures

Optimize for successful tasks per token—not the smallest prompt at any cost.

A practical long-running agent design

Durable session state
  goal + constraints + progress

Workspace
  files/artifacts/structured data

Recent conversation window
  local references + current interaction

Retrieval layer
  files/memory/tools on demand

Compaction layer
  compress old high-volume history at milestones

Live authoritative tools
  re-fetch current mutable state before actions

This architecture lets the model see what it needs without treating the prompt as the only database.

Production checklist

Before calling context management reliable, verify:

• external content is treated as data, not trusted instruction
• retrieval is scoped by authorization before relevance
• only task-relevant sources are loaded
• tool outputs expose pagination/truncation
• large artifacts live outside the prompt
• compaction preserves goals, constraints, IDs, versions, and progress
• recent turns remain available for local references
• persistent memory has freshness/update behavior
• mutable external state is re-fetched before mutation
• tool definitions are loaded/searchable efficiently
• context regressions have dedicated evals

Context management is the difference between an agent that can answer one prompt and an agent that can work coherently for hours or across sessions. The goal is not maximum context—it is maximum useful, current, authorized information per token.

References

• Anthropic — Effective Context Engineering for AI Agents
• OpenAI — Introducing the Agents API
• OpenAI — Equipping the Responses API with a Computer Environment
• OpenAI — Dreaming: Better Memory