deskthebuildCONTEXT ENGINEERING
← All notes · 6 min read

Our context window keeps growing until requests fail. How would you control it?

Imagine a hypothetical assistant that begins each conversation quickly, then becomes slower and less reliable after twenty turns. The application sends the full chat history, retrieved documents, tool output, and policy text on every request. Eventually the prompt reaches the model limit, costs increase, and important instructions compete with a large amount of irrelevant context.

This is not only a token-limit problem. It is a state-management problem: the system has no explicit policy for deciding which information remains useful as a task evolves.

Why the common answer is incomplete

Most candidates answer: “Summarize the conversation when it gets too long.”

Summarization can help, but a summary is a lossy derived state. It may remove a constraint that becomes important later, preserve an incorrect assumption, or mix information from different users and tasks. Summarizing everything also does not address oversized retrieval results, repeated tool schemas, verbose logs, or duplicated policy text.

A production design needs a context budget, source-specific retention rules, and a way to recover original evidence when compressed state is insufficient.

A production approach

1. Measure context by source and outcome

Record input tokens separately for system instructions, conversation turns, retrieved passages, tool definitions, tool results, memory, and the current user request. Connect those measurements to latency, cost, truncation, task success, and user correction rates. A single total-token metric cannot reveal whether growth comes from chat history, retrieval, or a tool returning an entire database row when two fields were needed.

2. Establish a budget before assembling the prompt

Reserve space for the expected output and a safety margin before allocating input tokens. Divide the remaining capacity among controlled instructions, recent interaction state, retrieved evidence, and tool context. Treat these as maximums rather than targets. A request should not fill the model window merely because space exists; larger contexts can increase cost and make relevant details harder to identify.

3. Classify state by retention requirement

Keep durable facts, temporary working state, and raw evidence separate. User preferences may persist across sessions. A plan for the current task should expire when the task ends. Raw messages and tool results should remain retrievable from storage even if they are not included in every prompt. This separation prevents a conversation transcript from becoming an accidental database with undefined retention and deletion behavior.

4. Use layered conversation compaction

Keep the most recent turns verbatim, compress older task state into a structured summary, and retain pointers to the original messages. The summary should preserve decisions, unresolved questions, user constraints, and referenced resource identifiers rather than produce a generic narrative. Version the summarization prompt and record which messages it covered so the system can rebuild the summary after a bug or policy change.

5. Bound retrieval before it enters the prompt

Retrieve candidates broadly enough to preserve recall, then rerank, deduplicate, and pack only passages that fit the evidence budget. Prefer diverse passages over repeated chunks from the same section. Preserve source identifiers and access-control context outside the model-visible prose. If high-scoring evidence does not fit, the system should narrow the question or perform a second retrieval step rather than silently truncate an arbitrary suffix.

6. Compact tool definitions and results

Expose only tools relevant to the current workflow instead of sending every schema on every turn. Return the smallest result required for the next decision: status, identifiers, and selected fields rather than raw payloads or logs. Store full results behind an opaque reference that a later step can request explicitly. Tool compaction reduces tokens while also limiting accidental disclosure of internal data.

7. Make overflow behavior explicit

When the assembled context exceeds its budget, apply a deterministic priority order. Controlled security instructions and the current request must not be dropped. Remove duplicate evidence, stale working state, and optional examples first. If required evidence still does not fit, stop and ask the user to narrow the scope, split the document, or approve a multi-step analysis. Silent truncation creates answers that appear complete while missing constraints.

8. Evaluate compression as a retrieval system

Create tests that require facts from recent turns, old turns, summaries, retrieved documents, and tool output. Measure whether the final answer uses the correct evidence, not merely whether the compressed prompt is shorter. Include long conversations, corrections to earlier facts, conflicting documents, multilingual inputs, and deleted memory. Run these evaluations whenever the model, summarizer, retrieval policy, or context budget changes.

Engineering trade-offs

Keeping more raw history preserves detail but increases cost, latency, and distraction. Aggressive summarization saves tokens but can turn an early mistake into durable state. Structured summaries are easier to validate than free-form prose, while requiring a schema that may evolve. On-demand retrieval of old messages reduces the default prompt size but adds another model or search step when the missing detail becomes relevant.

The right budget also depends on the workload. A short customer-support exchange can use a small recent window. Contract analysis may need exact source passages and stronger evidence retention. An agent executing a multi-hour workflow needs durable checkpoints rather than an ever-growing transcript.

Common mistakes

  • Treating the advertised model context window as the application’s target prompt size.
  • Summarizing old turns without retaining pointers to the original evidence.
  • Mixing long-term user memory with temporary task state.
  • Passing every available tool schema to every request.
  • Truncating from the beginning or end without considering information priority.
  • Counting retrieved tokens without measuring duplicate or irrelevant passages.
  • Testing compression on pleasant conversations that never revise earlier facts.

Failure handling and observability

Emit the chosen context budget, token contribution by source, compaction actions, retrieval packing decisions, and any discarded resource identifiers into the request trace. Track prompt-size percentiles, summary age, overflow frequency, retrieval deduplication rate, tool-result size, truncation attempts, and task success by context-size bucket. A sudden increase in tool tokens should be distinguishable from longer user conversations.

If summarization fails, fall back to a smaller verbatim recent window and clearly limit the task rather than sending an unbounded transcript. If evidence packing cannot include required sources, return a partial result that names the missing scope. If a bad summary version is discovered, invalidate affected summaries and rebuild them from retained source events. Never keep derived memory that cannot be traced to its origin.

Interview insight

The interviewer is not testing whether you know how to count tokens. They are testing whether you can design bounded state, preserve evidence, and degrade safely under a hard resource limit. A staff-level answer treats context as a deliberately assembled working set, not as a transcript that grows until the model refuses it.