15. Harness Engineering: Assembling a Reliable Runtime

The earlier chapters implemented the agent loop, tools, session log, Context Engineering, safety boundaries, and extension hooks as separate pieces. Putting those objects in one process still does not give you a reliable coding agent. The system must also answer: which request does a model change affect while a run is active? Where does an extension write from an event callback appear in the log? Should a second prompt be rejected or queued? When may the runtime truthfully report that the agent is idle?

Harness Engineering is the work of answering those orchestration questions. The agent loop advances one model-tool-model cycle. Context Engineering decides what the next request can see. The harness assembles the environment, session, model, tools, resources, policies, and events into a runtime with explicit consistency guarantees. Pi Agent is one example of this layered design: the lower-level loop executes, while the harness above it owns state, change boundaries, and settlement semantics.

Failure mode: one shared set of mutable objects

The most naive assembly lets the UI, extensions, and loop share config, messages, tools, and session, with every participant mutating whatever it needs. That can survive a single demo while failing at every timing boundary:

• A model request is already in flight when another callback changes the model or tools. The request, log, and UI now describe different facts for the same turn.
• Two calls to prompt() enter the loop together, interleaving messages and tool results on one session branch.
• A message_end subscriber appends a custom record, and that record reaches storage before the assistant message it was reacting to.
• A resource reload changes the system prompt but not the active tools, so the model-visible capabilities no longer match the executable set.
• The provider stream ends while an asynchronous listener is still writing the log, but waitForIdle() returns early and the process loses the last records on exit.
• Abort clears in-memory state and accidentally discards configuration changes or extension records that have not been persisted yet.

"Try not to let callbacks overlap" is not a solution. The harness has to classify mutable state and define admission and settlement semantics for every operation class.

Design boundary: Loader, Core, Harness, and Session

Start by separating four responsibilities that are easy to blur together:

• The Loader discovers project instructions, skills, prompt templates, and extensions. It applies project trust, tracks provenance and precedence, resolves collisions, emits diagnostics, and produces concrete resources.
• The Agent Core executes one model request, its tool batch, and any subsequent request, emitting the lifecycle events from Chapter 6.
• The Harness owns runtime configuration and phase. It creates turn snapshots, decides which operations reject, queue, or take effect at a safe point, and settles events and session writes in deterministic order.
• The Session Store appends durable entries and rebuilds the active branch. It does not decide when a turn starts or discover tools and project instructions on its own.

That boundary matters. The harness accepts Loader output instead of scanning directories inside prompt(). It calls the Agent Core instead of reimplementing the provider protocol. It owns write ordering without hard-wiring JSONL mechanics into lifecycle code. Moving to a remote execution environment, an in-memory session, or a different product shell then means replacing a dependency, not changing the consistency contract.

Four kinds of state cannot share one state object

A mature harness distinguishes at least four kinds of state:

• Live config: the application's current model, thinking level, tools, resources, system-prompt builder, and provider request options. Getters return the newest values.
• Turn snapshot: the model, tools, resources, system prompt, session projection, and request options used by one concrete provider request. Once created, that request treats the snapshot as immutable.
• Persisted session: entries successfully committed to the source of truth. Recovery and the next context projection trust only these facts.
• Pending writes: session writes requested by setters, extensions, or listeners while an operation is active. They are persisted in request order at safe points instead of being spliced into the current event sequence.

The central rule is that live config may update immediately during a turn, but it cannot reach backward into an in-flight request. The change takes effect when the harness builds a new snapshot at the next safe point. If the corresponding session entry cannot be written safely yet, it joins the pending-write queue.

live config v1
    -> create snapshot v1
    -> provider request 1 -----------------------+
                                                   |
callback sets live config v2                       |
                                                   v
assistant/tool messages -> persisted session -> save point
pending writes -------------------------------> flush
                                                   |
                                                   v
                                      create snapshot v2
                                      -> provider request 2

Snapshot isolation does not mean cloning the whole process. It is often enough to copy arrays and request options and declare tool definitions and resource objects immutable. If extensions may mutate nested values, deep-copy them, freeze them, or replace them with immutable structures. A shallow copy protects the container, not the objects inside it.

Operation admission: reject, queue, or apply at a safe point

Do not force every busy-time operation through one policy. Different operations have different semantics:

• A new prompt, skill invocation, or prompt-template invocation starts a structural run. When the harness is not idle, it should return a stable busy error instead of silently merging the input into the current run.
• Compaction and tree navigation change the session projection or active leaf, so they are idle-only.
• Steering and follow-up are intentionally mid-run inputs. They enter separate queues and are consumed at boundaries defined by the Agent Core.
• A next-turn message can be queued early but is injected only before the next user-initiated turn. Whether it survives abort must be a fixed contract.
• Setters for the model, thinking level, tools, resources, and stream options update live config. The current snapshot stays fixed; the next safe point reads the new values.
• Abort may cancel an active turn, but it still passes through shared cleanup, pending-write flushing, and the settlement barrier.

Represent phase as an explicit discriminated union such as idle | turn | compaction | branch_summary | retry. Structural methods must check and switch phase before their first await, or two calls can still pass the same idle check. settled is not a phase; it is the event that says the run's messages, listeners, and required writes have finished settling.

The save point turns timing into a public contract

A safe point comes after a complete assistant turn: the assistant message and every tool result it requested have finished. A stable order is:

1. On message_end, persist the agent message before notifying observers; observers therefore see committed state.
2. After turn_end, flush pending writes in queue order, removing an item only after its write succeeds.
3. Emit save_point to announce that the turn's facts and auxiliary writes are stable.
4. If the loop will continue, re-read the session and live config, build a fresh snapshot, and only then start the next provider request.
5. After agent_end, settle remaining writes and asynchronous listeners. Emit settled last, then release the completion barrier.

This ordering guarantees that a custom entry requested by an assistant message_end listener appears after the assistant message. It also guarantees that a model changed during the run affects only the next request. Do not clear the pending array up front on write. Commit one entry at a time and call shift() only after success so a mid-flush failure preserves the unfinished tail.

Minimal implementation: wrap the existing loop in a harness

The following code compresses the new contracts into a runnable core. RunLoop and Session are dependencies implemented in earlier chapters. A faux provider can implement RunLoop, so the test does not need a real model.

type Phase = "idle" | "turn";

type Message = {
  role: "user" | "assistant" | "tool";
  text: string;
};

type RuntimeConfig = Readonly<{
  model: string;
  tools: readonly string[];
  resources: readonly string[];
}>;

type SessionEntry =
  | { type: "message"; message: Message }
  | { type: "config"; config: RuntimeConfig }
  | { type: "custom"; value: string };

type Session = {
  messages(): Promise<readonly Message[]>;
  append(entry: SessionEntry): Promise<void>;
};

type TurnSnapshot = Readonly<{
  config: RuntimeConfig;
  messages: readonly Message[];
}>;

type HarnessEvent =
  | { type: "message_end"; message: Message }
  | { type: "save_point"; hadPendingWrites: boolean }
  | { type: "settled" };

type RunLoop = (input: {
  prompt: Message;
  snapshot: TurnSnapshot;
  signal: AbortSignal;
  commit(message: Message): Promise<void>;
  prepareNextTurn(): Promise<TurnSnapshot>;
}) => Promise<Message>;

type HarnessErrorCode = "busy" | "session" | "hook" | "unknown";

class HarnessError extends Error {
  readonly code: HarnessErrorCode;

  constructor(code: HarnessErrorCode, message: string, cause?: Error) {
    super(message, cause ? { cause } : undefined);
    this.code = code;
  }
}

class Harness {
  private phase: Phase = "idle";
  private config: RuntimeConfig;
  private pendingWrites: SessionEntry[] = [];
  private controller?: AbortController;
  private idlePromise?: Promise<void>;
  private listeners = new Set<(event: HarnessEvent) => void | Promise<void>>();
  private readonly session: Session;
  private readonly runLoop: RunLoop;

  constructor(initialConfig: RuntimeConfig, session: Session, runLoop: RunLoop) {
    this.config = this.copyConfig(initialConfig);
    this.session = session;
    this.runLoop = runLoop;
  }

  getPhase(): Phase {
    return this.phase;
  }

  getConfig(): RuntimeConfig {
    return this.copyConfig(this.config);
  }

  subscribe(listener: (event: HarnessEvent) => void | Promise<void>): () => void {
    this.listeners.add(listener);
    return () => this.listeners.delete(listener);
  }

  async setConfig(patch: Partial<RuntimeConfig>): Promise<void> {
    const next = this.copyConfig({
      model: patch.model ?? this.config.model,
      tools: patch.tools ?? this.config.tools,
      resources: patch.resources ?? this.config.resources,
    });
    const entry: SessionEntry = { type: "config", config: next };
    if (this.phase === "idle") await this.session.append(entry);
    else this.pendingWrites.push(entry);
    this.config = next;
  }

  async appendCustom(value: string): Promise<void> {
    const entry: SessionEntry = { type: "custom", value };
    if (this.phase === "idle") await this.session.append(entry);
    else this.pendingWrites.push(entry);
  }

  async prompt(text: string): Promise<Message> {
    if (this.phase !== "idle") throw new HarnessError("busy", "Harness is busy");
    this.phase = "turn";
    const controller = new AbortController();
    this.controller = controller;
    let releaseIdle = (): void => {};
    this.idlePromise = new Promise<void>((resolve) => {
      releaseIdle = resolve;
    });

    try {
      const snapshot = await this.createTurnSnapshot();
      return await this.runLoop({
        prompt: { role: "user", text },
        snapshot,
        signal: controller.signal,
        commit: async (message) => {
          await this.session.append({ type: "message", message });
          await this.emit({ type: "message_end", message });
        },
        prepareNextTurn: async () => {
          const hadPendingWrites = this.pendingWrites.length > 0;
          await this.flushPendingWrites();
          await this.emit({ type: "save_point", hadPendingWrites });
          return this.createTurnSnapshot();
        },
      });
    } catch (error) {
      if (error instanceof HarnessError) throw error;
      const cause = error instanceof Error ? error : new Error(String(error));
      throw new HarnessError("unknown", cause.message, cause);
    } finally {
      try {
        await this.flushPendingWrites();
      } finally {
        this.controller = undefined;
        try {
          await this.emit({ type: "settled" });
        } finally {
          this.phase = "idle";
          releaseIdle();
          this.idlePromise = undefined;
        }
      }
    }
  }

  abort(): void {
    this.controller?.abort();
  }

  async waitForIdle(): Promise<void> {
    await this.idlePromise;
  }

  private async createTurnSnapshot(): Promise<TurnSnapshot> {
    const config = this.copyConfig(this.config);
    const messages = [...(await this.session.messages())];
    return { config, messages };
  }

  private async flushPendingWrites(): Promise<void> {
    while (this.pendingWrites.length > 0) {
      await this.session.append(this.pendingWrites[0]!);
      this.pendingWrites.shift();
    }
  }

  private async emit(event: HarnessEvent): Promise<void> {
    for (const listener of this.listeners) await listener(event);
  }

  private copyConfig(config: RuntimeConfig): RuntimeConfig {
    return {
      model: config.model,
      tools: [...config.tools],
      resources: [...config.resources],
    };
  }
}

This teaching version implements only idle | turn, but it establishes the right extension point. Add compaction, branch summary, and retry as structural phases rather than inventing more boolean flags. A production implementation should also normalize session, hook, auth, and other failures into stable codes while preserving the original cause.

What to observe: prove that changes apply only at boundaries

Script two consecutive responses with the faux provider. The first returns a tool use. In the tool-start event, switch the model, thinking level, resources, system prompt, and active tools to a second set. The second response returns final text. You should observe:

phase=turn request=1 model=model-a tools=read prompt=v1
config_update model=model-b tools=grep prompt=v2
message_end assistant persisted=true
message_end toolResult persisted=true
save_point hadPendingWrites=true
request=2 model=model-b tools=grep prompt=v2
agent_end
settled
phase=idle

The first request must remain entirely on v1; only the second request uses v2. Then make a message_end listener call appendCustom("listener note"). The final entry order must be user, assistant, tool result, listener note, independent of Promise completion timing.

waitForIdle() should await asynchronous listeners and the final flush. Do not call await harness.waitForIdle() from an active listener, however. The listener is itself part of the current run's unsettled work, so it would wait for itself. If an extension needs to do work after the harness becomes idle, expose a scheduler such as runWhenIdle() instead of making callbacks wait on the settlement promise.

Typed hooks, observation events, and error semantics

Harness Engineering is more than adding a mutex. Public events should form a discriminated union, while hooks that may change execution need event-specific result types: a context hook returns replacement messages, a tool-call hook returns a block decision, and a tool-result hook returns a constrained patch. Pure observation events accept no mutation result.

Each control-plane hook needs an explicit failure policy. Permission hooks generally fail closed, while a statistics listener should be isolated and reported instead of terminating the turn. If a listener fails after state has already been persisted, do not pretend to roll back memory while leaving a committed log behind. Public methods should expose stable categories such as busy, invalid_argument, session, hook, and auth, so callers never have to parse error strings.

Production tradeoffs

• Once the Loader and harness are separate, resource reloads must preserve provenance and diagnostics. Name collisions need a deterministic winner instead of depending on filesystem enumeration order.
• Pending writes need a limit or backpressure. Otherwise a runaway extension can exhaust memory during a long turn. At the limit, reject new writes with an observable error rather than dropping old facts.
• Specify abort behavior per queue: which inputs are cleared, which survive, and whether pending writes still commit. A blanket queues = [] hides important semantic differences.
• Provider streams are rarely resumable in the middle. After a process crash, continue from the latest durable boundary or mark the unfinished operation interrupted. Never rerun an unfinished tool automatically unless it declares itself idempotent or retry-safe.
• Events and session entries evolve over time. Stable types, versions, and migration tests are more reliable than serializing the entire in-memory object graph.
• Tests must create races deliberately. Use deferred promises to pause a provider or listener at an exact point, then change config, append a write, or abort. Happy-path tests cannot demonstrate the value of a harness.

Exercise

Assemble the Agent Core, Session, context builder, tools, and faux provider from the earlier chapters behind Harness.

Acceptance criteria:

• Of two concurrent calls to prompt(), the second receives a stable busy error and does not write half a user message into the session.
• Changing the model, tools, resources, and system prompt during a turn leaves the active provider request on its old snapshot; the request after the next safe point uses the new snapshot.
• An entry appended by a message_end listener always follows the agent message. One failed write in the middle of a flush does not discard the rest of the pending queue.
• Compaction and tree navigation reject outside idle, while steering, follow-up, and next-turn inputs follow their separate queue contracts.
• waitForIdle() returns only after the last asynchronous listener and pending write finish. The test does not use sleep to guess at timing.
• After abort, phase eventually returns to idle, and next-turn inputs that should survive are injected before the next prompt.
• A faux-provider test records the model, tools, system prompt, and session-entry order for request 1 and request 2 and asserts every invariant directly.