Learn Agents from Pi
中文
On this page

04. Provider Abstraction and a Unified Message Protocol

When an agent has only one model, you might be tempted to pass the vendor SDK's types throughout the entire system. That gets you started quickly, but it soon binds the runtime to one provider's semantics. Different vendors express tool calls, streaming deltas, errors, usage, stop reasons, system prompts, and image input in different ways. The agent kernel should not have to understand any of these differences.

The mature approach is to establish an internal message protocol and convert in both directions at the provider boundary. Model vendors are plugins; the agent's source of truth is your own types. A real system might integrate a dozen different API shapes and more than forty gateway vendors at the same time, and the kernel knows nothing about any of it — that is the value of the protocol layer.

Why an internal protocol is necessary

Without an internal protocol, the problem spreads step by step:

  • The UI needs to know whether the assistant stopped because of tool use, so it comes to depend on vendor fields.
  • The session log stores raw vendor responses, and can no longer be restored after switching models.
  • The tool result format is coupled to one particular API, and another provider needs adaptation everywhere.
  • Tests must mock the vendor SDK instead of mocking the agent's real boundary.

An internal protocol confines these problems to the provider adapter. The agent loop only knows about Message, ToolDefinition, AssistantMessage, and stopReason. Provider differences exist in exactly two places: conversion before sending the request, and conversion after receiving the response.

What the protocol layer must store at minimum

An assistant message should not store text alone. At a minimum it should store:

  • model: the id of the model that actually responded. The responding model can differ from the requested one (routing gateways select a model automatically), and both are worth recording.
  • provider: an optional provider id, useful for auditing and restoration.
  • usage: beyond input and output tokens, also cache reads, cache writes, and reasoning tokens. The cache fields directly affect cost accounting — a cache read is typically only about one tenth the price of a normal input token.
  • stopReason: the normalized stop reason the loop's control flow needs.
  • content: text, tool calls, thinking blocks, and possibly images or other blocks.
  • errorMessage: the human-readable reason when the stop reason is error or aborted.

The reasons for storing these fields are practical. A user may switch models mid-session; you still need to know where each response came from. Billing multiplies usage by the unit prices in the model catalog. Resumption needs the stop reason. Tool calling needs structured content blocks.

The vendor differences the adapter must absorb

It is worth listing some of the differences that actually exist, so you get a concrete sense of what an adapter does:

  • System prompts: some APIs use a system role, others require a developer role.
  • Output limit: the field might be named max_tokens, or it might be max_completion_tokens.
  • Message ordering: most APIs require a tool result to immediately follow its corresponding assistant message, and several consecutive tool results must be merged into a single message.
  • Empty content: some APIs reject empty text blocks, and the adapter must strip them during conversion.
  • Tool schemas: some APIs offer a strict JSON schema mode, and some proxy gateways reject schemas that carry unknown fields.
  • Reasoning content: models that support chain-of-thought return thinking blocks, often accompanied by an encrypted signature that is only valid for the same model and is meant to be replayed verbatim on the next request.
  • Argument encoding: the format of tool call ids, and whether arguments are an object or a string, differ from vendor to vendor.

Each difference looks trivial on its own; together they add up to thousands of lines of adapter code. The key point is that all of that code is concentrated at the boundary — the kernel and the product layer never change a single line.

Cross-model switching is where the differences concentrate most. When a session switches from model A to model B halfway through, the adapter needs to: discard A's proprietary thinking signatures or downgrade the chain-of-thought to plain text; normalize the tool call ids A generated into a format B accepts and update the tool result references to match; and replace images B does not support with placeholder notes. Without this conversion layer, "switching models" — a feature that looks trivial — turns the session straight into an invalid request.

The shape of a provider adapter

You can write the provider boundary as two directions:

type ProviderRequest = {
  messages: Message[];
  tools: ToolDefinition[];
  systemPrompt: string;
  model: string;
};

type ProviderClient = {
  id: string;
  complete(request: ProviderRequest, signal: AbortSignal): Promise<AssistantMessage>;
  stream(request: ProviderRequest, signal: AbortSignal): AsyncIterable<ProviderEvent>;
};

complete serves non-streaming use and tests; stream serves the product experience. The final assistant messages the two return must be equivalent. Otherwise you will run into "the non-streaming tests pass, but the streaming UI behaves differently."

There is one easily overlooked cleanup step before sending a request: replacing unpaired UTF-16 surrogates in the text. Content the user pasted, or bytes read out of a file, may contain them, and most APIs reject such JSON outright. Normal emoji are paired surrogates and are unaffected. This kind of "protocol-boundary hygiene" work belongs to the adapter.

The faux provider is a first-class citizen

Do not treat the faux provider as a throwaway mock. It should implement the same interface as a real provider and be able to return complete assistant messages, tool calls, usage, and stop reasons. A scripted provider can work like this:

type ScriptedStep = {
  expectLastRole?: Message["role"];
  response: AssistantMessage;
};

class ScriptedProvider implements ModelClient {
  private readonly steps: ScriptedStep[];
  private index = 0;

  constructor(steps: ScriptedStep[]) {
    this.steps = steps;
  }

  async complete(input: { messages: Message[] }): Promise<AssistantMessage> {
    const step = this.steps[this.index];
    if (!step) {
      throw new Error("No scripted response left");
    }
    this.index += 1;
    const last = input.messages.at(-1);
    if (step.expectLastRole && last?.role !== step.expectLastRole) {
      throw new Error(`Expected last role ${step.expectLastRole}, got ${last?.role ?? "none"}`);
    }
    return step.response;
  }
}

This provider can test whether the loop requests the model again after a tool result, and it can also test unknown tools, compaction, steering, and resumption. It comes much closer to real agent behavior than mocking fetch.

The model catalog and capabilities

The agent also needs a model catalog. The catalog is not dropdown data — it is the basis for runtime decisions. For each model, record at least:

  • Context window size and maximum output tokens.
  • Whether it supports tool calling, image input, and reasoning mode.
  • The per-million-token prices for input, output, cache reads, and cache writes.
  • How reasoning-effort tiers map to that vendor's parameters (the same "high" is an entirely different field across APIs).
  • The default provider, endpoint, and authentication method.

Compaction thresholds, tool exposure, UI hints, cost accounting, and error messages all depend on these capabilities. Do not scatter "if the model name contains some string" checks throughout the code. Model capabilities should come from configuration and the catalog.

Production tradeoffs

The provider adapter is the boundary where error handling is densest. The first step is normalization: unify the vendors' assorted exceptions into a handful of categories — authentication failure, rate limiting, retryable server errors, context overflow, content-safety refusal, and network interruption. The second step is classified retry. A real system's retryability judgment is roughly:

  • Retryable: 429 rate limiting, 5xx, gateway timeouts (including a CDN's 524), overloaded, connection refused, socket hang-ups, a stream cut off before its end marker, and any error where the vendor explicitly says "please retry."
  • Not retryable: quota exhausted, insufficient balance, billing issues, authentication failure, and a request that is itself invalid.

Retries need backoff and a cap, and they should respect the wait time the server asks for — but if the server asks you to wait longer than some limit (say 60 seconds), you should hand the error, together with the wait duration, up to the caller and let the user decide, rather than letting the interface silently hang for a minute.

In addition, the streaming adapter must produce a clear state when the stream breaks. Do not let the UI hang on "the model is responding." The assistant message after a stream interruption can be marked stopReason: "error" and keep the text fragments already received, making it easy for the user to decide whether to retry or continue.

Exercise

Implement two providers:

  • ScriptedProvider: returns fixed assistant messages from an array.
  • HttpProvider: only needs to support non-streaming calls to one real model.

Acceptance criteria:

  • The agent loop can switch providers without changing a single line of core code.
  • Both providers return the internal AssistantMessage.
  • Usage (including the cache fields) and the stop reason are not lost.
  • When the real provider returns a context-overflow error, the error is recognized as requiring compaction rather than treated as an ordinary exception.
  • Write an error-classification function that correctly sorts at least six simulated error texts into retryable and non-retryable.