Learn Agents from Pi
中文
On this page

16. Evaluation, Debugging, and the Final Project

The biggest illusion in agent development is "that last run looked fine, so it must be correct." Model output is unstable, real repositories are complex, and tools and context carry a lot of state. Without an evaluation and debugging system, you'll re-test by hand after every prompt or tool-description change — and you won't be able to tell where a regression came from.

Three layers of testing

A teaching project needs at least three layers of tests:

  • Unit tests: tool parameter validation, path resolution, output truncation, the write queue, newline and BOM handling.
  • Loop tests: the faux provider drives tool use, error feedback, steering, compaction, and abort.
  • Replay tests: read a session log, rebuild the context, and verify the projection result.

End-to-end tests against a real model are smoke tests only. They can tell you the system is wired up, but they are unsuitable as your primary regression suite. Primary regressions must be deterministic, cheap, and repeatable. This is also why every earlier chapter treats the faux provider and the event stream as first-class citizens: testability is not bolted on at the end but designed in from the protocol layer. The machine-readable modes (print / JSON) are themselves natural evaluation entry points — feed in an input and assert on the sequence of events and the exit code that come out.

Harness race-condition tests

Ordinary loop tests verify that "input A produces event B." The failures Harness Engineering is most exposed to are timing relationships. Use deferred promises to pause the provider, a tool, or a listener at an exact boundary, then trigger a change from another asynchronous chain. Cover at least these invariants:

  • The active provider request uses one fixed turn snapshot. Changing the model, tools, resources, or system prompt mid-run affects only the request after the next safe point.
  • A second structural operation rejects consistently while busy. Steering, follow-up, and next-turn messages use separate queues rather than one ambiguous pending-input array.
  • Agent messages persist first, followed by session entries requested by message_end listeners. Promise completion order cannot rewrite log order.
  • waitForIdle() waits for the final asynchronous listener and pending write instead of returning when the provider stream ends.
  • Abort eventually restores idle and clears or preserves each queue according to its contract. Failure cleanup still cannot swallow facts waiting to be written.

Do not implement these tests with a fixed sleep(100). Hold an explicit barrier, assert that the harness is not idle yet, release the listener, and then await the completion promise. A failure now means the contract is broken, not merely that the CI machine ran slowly.

Session replay

The session log is natural debugging material. A failed task can be exported as JSONL; the test harness reads it, rebuilds the active branch, and asserts:

  • Whether the context contains the system rules it should.
  • Whether the compaction summary preserves the key facts.
  • Whether the last tool result correctly enters the next turn.
  • Whether a model switch affects subsequent requests.
  • Whether the branch leaf points at the expected entry.

This is far more reliable than screenshots or a hand-written bug description. Replay tests carry an additional value: they are a safety net for format migration. Chapter 8 noted that the log format will evolve; keep real files in old formats as test samples, replay them after every migration, and you guarantee that existing users' history won't be corrupted by the upgrade.

Cost and performance

The agent should record usage, latency, tool time, and retry counts for every model request. Without this data, you can't answer "why did this task take so long" or "which model is the most expensive." Cost information can be recorded as events and session entries; it doesn't have to enter the model context every time.

Common metrics:

  • input tokens.
  • output tokens.
  • cache read/write tokens.
  • reasoning tokens.
  • provider latency.
  • tool latency.
  • retry count.
  • compaction count.
  • estimated cost.

Cache and reasoning tokens deserve their own line items. A cache read is usually far cheaper than ordinary input, and an agent with a stable system-prompt prefix and a high cache hit rate can cost a fraction of an otherwise similar system with a low hit rate — a direct echo of Chapter 11's "put stable content first" assembly order. Reasoning tokens are part of the output but are priced separately, so failing to count them will underestimate cost.

It is clearer to land the cost as a concrete formula. Every assistant response brings back a usage record; multiply it by the unit prices in the model catalog:

turn_cost =
    input_tokens       * price_input
  + output_tokens      * price_output
  + cache_read_tokens  * price_cache_read     // about 0.1x of price_input
  + cache_write_tokens * price_cache_write
  + reasoning_tokens   * price_output          // reasoning tokens priced as output

For example: one turn hits 40k cached tokens, 2k of new input, 500 of output, and 800 of reasoning. If you bill those 40k cached tokens at the ordinary input price, the cost nearly doubles; at 0.1x, the cached portion is worth only 4k of ordinary input. That is why "a stable prefix plus a high cache hit rate" can compress a long session's bill to a fraction. It is also why usage must be recorded in five separate fields rather than a single total — once you record only the total, you can no longer break out how much of it was cache versus reasoning.

The final project: tiny-agent

The final project is not a clone of some existing product; it is proof that you understand the engineering boundaries of an agent. Pick a small TypeScript repository and have tiny-agent complete one real change:

  1. Read the user's goal.
  2. Search for relevant files.
  3. Read the target files.
  4. Implement the change with precise edits.
  5. Run the specified check command.
  6. Keep fixing based on failures.
  7. Produce a final summary.
  8. Write a complete session log.
  9. After exit, resume the session and explain what was just done.

When grading, don't just check whether the final code is correct — check whether the process is auditable.

Capstone checklist

Your tiny-agent must satisfy:

  • Providers are swappable, with at least a real provider and a faux provider, and errors can be classified as retryable or not retryable.
  • Tool calls are driven by message content, not by hard-coded steps, and tools are not dropped because of the stop reason.
  • Failed tool parameter validation is fed back to the model, and extra fields are normalized rather than hard-rejected.
  • read/edit/write/bash all have path boundaries and truncation policies, and edit has normalized matching and newline preservation.
  • Write operations separate the diff details from the model-facing summary, and writes to the same file are queued by real path.
  • The session log is append-only JSONL, with a parentId tree structure and a version number.
  • Resuming does not re-execute historical tools.
  • Compaction never deletes history; it only changes the context projection, and it can retry automatically when the context overflows.
  • Steering and follow-up have independent queues and well-defined consumption timing.
  • The harness creates a stable turn snapshot for every provider request, and mid-run configuration changes take effect only at the next safe point.
  • Structural operations reject while busy, pending session writes commit in deterministic order after agent messages, and waitForIdle() never returns early.
  • In JSON mode, stdout emits machine-readable events only, and exit codes follow a convention.
  • The permission gate and project trust apply consistently across all shells.
  • The faux provider tests cover at least one tool-error self-correction scenario.

Debugging workflow

When the agent misbehaves, investigate in this order:

  1. Check the session log to confirm the facts were written.
  2. Check the harness timeline to confirm the ordering of phase, turn snapshot, save point, pending writes, and settled.
  3. Check the context projection to confirm what was sent to the model is correct (Chapter 11's assembly debugging view is most useful here).
  4. Check the provider adapter to confirm the stop reason and tool calls were converted correctly, and that ids and thinking are handled properly across models.
  5. Check the tool result to confirm the error is fixable and actionable.
  6. Check the system prompt to confirm rule ordering and conflicts.
  7. Only then suspect the model itself.

This order matters. Many "the model won't listen" problems are actually a constraint missing from the context projection, or a tool error written in a way the model can't act on. Pin responsibility first on the deterministic layer you control, and only then on the nondeterministic model, and debugging becomes far more efficient.

Closing

The heart of building a coding agent is not finding a magic prompt; it is placing a nondeterministic model inside deterministic engineering boundaries: clear protocols, controlled tools, recoverable state, observable events, auditable permissions, bounded extensions, and a harness that confines every change to an explainable point in time. Get these right, and gains in model capability naturally become gains in system capability; get them wrong, and the stronger the model, the harder the system is to control. This is also why this course returns, from the first chapter to the last, to the same layered diagram — each layer solves one determinism problem, and together they form a system that can genuinely change your code for you, and that you dare to let do so.