09. Context Engineering and Compaction

No matter how large the context window is, an agent will fill it. A coding agent reads files, runs commands, emits diffs, hits errors, and receives user steering. Context Engineering asks not "how do we cram in more tokens" but "how do we preserve the state necessary to keep working." Compaction means projecting old context into a summary the agent can continue executing from — not merely making the chat transcript shorter.

When to compact

The compaction trigger should be based on the model window and a reserved budget:

if context_tokens > context_window - reserve_tokens:
  compact()

reserve_tokens has to leave enough room for the system prompt and the next response. As a reference, one real system's defaults are: reserve 16k tokens, and keep roughly the most recent 20k tokens of messages after compaction. On a model with a 200k window, that means compaction triggers once the context reaches around 184k. If the reserve is too small, the next request gets rejected by the provider before compaction has a chance to run.

The full trigger path actually has three branches, all of which must be handled:

Threshold trigger: check the token count after every model response, and compact if it is over. This is the main path.
Manual trigger: the user explicitly requests compaction (for example, a /compact command). The user sometimes knows earlier than the threshold that "something big is coming next."
Overflow recovery: the request has already been rejected by the provider as "context too long." Here you compact and then automatically retry the request that just failed, so the task continues seamlessly instead of the error being thrown at the user.

Token counting itself takes care. Historical messages do not all carry precise usage (user messages and tool results do not), so you have to estimate. "Character count divided by 4" is a common conservative heuristic — it tends to overestimate, and it is better to compact early than to hit the wall. The real usage returned with each assistant response is then used to calibrate the current total.

The cut point

Do not cut in the middle of an arbitrary message. A safe cut point is usually after a complete turn — that is, once the assistant message and all the tool results it requested have been written to the log. Cutting through half a tool batch makes the model see "I called a tool, but the result vanished," and after recovery it easily re-executes or misjudges. In practice you can walk backward from the end of the log, accumulating estimated tokens until you reach the "keep the most recent N tokens" budget, then align the cut point to the nearest safe boundary — a tool result can never be a cut point; it must travel with its own tool call.

A compaction entry should record:

The summary text.
The token count before compaction.
The id of the first entry that is still kept in full.
The list of key files that were read.
The list of files that were modified.

The file lists deserve special emphasis: they can be extracted mechanically from the summarized messages at compaction time (every read, edit, and write path is right there in the tool call arguments), without relying on the summarizer model to get it right. These details do not all need to reach the model, but they should go into the log, where the UI and extensions can use them.

The shape of a good summary

An agent-facing summary is not meeting minutes. It should help the model keep working. The summary itself is produced by a model call, with a dedicated prompt that asks for a fixed structure:

Goal:
- User wants ...

Current status:
- Done ...
- Still failing ...

Important constraints:
- Do not edit tests.
- Keep public API unchanged.

Files observed:
- src/parser.ts: contains ...

Files modified:
- src/parser.ts: changed ...

Open tool results:
- Last test run failed with ...

Next step:
- Inspect ...

The most important parts of the summary are the constraints, the file facts, and the next step. Do not force the model to rediscover everything after compaction.

When projecting, the summary needs explicit wrapper language, such as "the earlier conversation history has been compacted into the following summary," and clear delimiters marking the summary's extent. Without this framing, the model might mistake the summary for something the user just said, or the other way around, hallucinating nonexistent details from the file descriptions inside the summary.

The post-compaction amnesia test

Every compaction implementation should run an amnesia test: build a long session in which the agent reads a file, discovers a constraint, and modifies another file, then trigger compaction. After compaction, ask the model "what is the next step." If the model has forgotten the user's constraints or the file it just modified, the summary is unusable.

This test does not need a real model. You can have the faux provider check whether the post-compaction context contains these keywords: the goal, the constraints, the modified files, the most recent failure, and the next step. A smoke test with a real model is only used to check that the summary reads naturally.

More context is not better

Many beginners lean toward keeping as many old messages as possible. It feels safe, but in practice it dilutes the model's attention with historical noise. Tool output especially: one failing test run can produce thousands of lines of logs, and what is actually useful is the failure name, the error line, the tail of the stack trace, and the command's exit code.

The goal of context engineering is a high signal-to-noise ratio. For an agent, good compaction is not "lossless"; it is "preserve the state needed to finish the task, and be explicit about what information was discarded." When the summary cannot cover some old fact, it should tell the model to re-read the file, rather than pretend to remember.

Production tradeoffs

Compaction itself calls the model, so it can fail, cost money, and get rate-limited. The runtime has to decide what to do when compaction fails. Common strategies:

If there is still room, continue for one more turn and compact again later.
If you are already near the limit, pause the task and ask the user to confirm.
If the compaction model fails, fall back to a shorter local summary template (goal plus file list), but flag the quality as lower.

User input during compaction also has to be handled: if the user sends another message while compaction is running, it should be queued until compaction finishes rather than spliced into a half-built context. The UI should show a "compacting" state, and machine-readable mode needs a corresponding event.

The compaction summary should be written to the log. Do not keep it only in memory. When a session is recovered, the context builder must be able to see the compaction entry and skip earlier messages based on it. Finally, it is worth making the compaction entry point a replaceable hook: some scenarios (such as an extension maintaining its own knowledge base) need a custom summarization strategy, and the compaction entry the hook produces travels the same projection path as the built-in compaction.

Exercise

Add a compaction entry and a context builder to the session log.

Acceptance criteria:

Compaction only happens at complete turn boundaries; a tool result is never separated from its tool call.
The compaction entry contains a summary, tokensBefore, firstKeptEntryId, and a mechanically extracted file list.
When building context, old messages are replaced by a delimited summary, not deleted from the log.
After compaction, the model can still see the goal, the constraints, the files read, the files modified, and the next step (the amnesia test passes).
When the provider reports that the context is too long, the agent compacts and automatically retries the original request instead of exiting.
A manual compaction entry point is available.