Learn Agents from Pi
中文
On this page

12. Safety Boundaries and the Permission Model

A coding agent can read and write files, run commands, access the network, and call models. As long as it runs on your machine, it has every capability within the scope of its process permissions. The first step of safety design is honesty: don't treat the prompt as a permission system, and don't treat "the model probably wouldn't do that" as an isolation boundary.

Threat model

Consider at least four categories of risk:

  • User error: asking the agent to delete files, overwrite changes, or run expensive commands.
  • Model misjudgment: the model writes a destructive command where a test command was intended.
  • Prompt injection: text in the repository lures the model into leaking environment variables or bypassing rules.
  • Tool vulnerabilities: path escapes, command injection, parallel-write clobbering, secrets leaking into logs.

Different risks call for different boundaries. A prompt can lower the probability of misjudgment, but it cannot stop a malicious tool call. The real boundary sits in front of tool execution.

The permission gate

A tool call can pass through a permission gate before it runs:

type PermissionDecision =
  | { type: "allow" }
  | { type: "deny"; reason: string }
  | { type: "confirm"; prompt: string };

read is usually allowed, edit may be allowed depending on file state, and bash is confirmed or denied based on how the command is classified. The permission gate must run before a tool call reaches the executor. If the call is denied, it should become an error tool result so the model knows why — a denial, to the model, is an observable, adjustable outcome, not a crash.

Don't intercept only in the UI. The SDK, JSON mode, and RPC mode must all go through the same permission gate. Otherwise a user can bypass your safety policy just by switching to a different shell. This is the direct safety corollary of "one kernel, many shells": the permission decision belongs to the kernel, and the interface is only responsible for presenting the confirmation request to the user and returning the user's choice to the kernel.

There is a common misconception to avoid here: the permission gate is not a hardcoded "blacklist of dangerous commands." Baking strings like rm -rf into the kernel neither stops a destructive command written a different way nor avoids false positives on legitimate use. A sturdier model is this: the kernel only provides the mechanism — "there is a decision point before a tool call that can return allow / deny / confirm" — and the actual decision rules are injected as policy, coming from a default policy, user configuration, or the extension hooks of Chapter 14. Command classification is just one implementation of that policy, not a universal list built into the kernel. Safety is therefore two layers stacked together: one is the project-trust boundary discussed below, which decides whether project-local configuration, extensions, and skills load at all; the other is this per-call decision point, which decides whether each concrete tool call may execute. The former is a one-time decision at load, the latter runs before every execution, and neither can be bypassed by any shell.

Project trust

The first gate in a real system is not command classification but project trust. The reason is practical: an agent loads project-local configuration, extensions, custom system prompts, and skill files — all of which are, in essence, code that runs. Opening an unfamiliar repository and loading them unconditionally is the same as running scripts of unknown origin.

So the first time it enters a workspace, the agent should ask whether to trust the project, and offer graduated options: trust for this session only, trust this directory, trust its parent directory (covering all subdirectories), or never trust. The decision is persisted against the normalized path and takes effect automatically next time; global configuration can set a default policy (always trust / always ask / never trust). The trust check walks up the directory tree, and the nearest decision wins.

In the untrusted state, allow only read-only tools, refuse to load project configuration and extensions, and restrict bash, file writes, and reads of sensitive paths. Trust is not a permanent truth; users should be able to revoke it. Collapsing all of this into a single allow/deny toggle leaves users unable to make fine-grained judgments, so trust policy should be concrete:

  • Whether the current workspace is trusted.
  • Whether project extensions and custom prompts may be loaded.
  • Whether project scripts may be executed.
  • Whether network access is allowed.
  • Whether environment variables may be read.
  • Whether paths outside the workspace may be written.

Sandboxes and external boundaries

If you need stronger safety, put tool execution inside a container, a virtual machine, or a remote sandbox. That way, even if the model requests a dangerous command, the damage is confined to the sandbox. A sandbox doesn't have to be part of the first version of a teaching project, but the interface should be reserved for it in advance: tool execution should not be hard-wired to the local filesystem and shell.

This is exactly the safety value of the operations interface that Chapters 3 and 10 return to again and again:

type FileOperations = {
  readText(path: string): Promise<string>;
  writeText(path: string, content: string): Promise<void>;
  realpath(path: string): Promise<string>;
};

Local, SSH, container, and remote runtimes can all implement the same interface. The agent kernel doesn't need to know where tools execute, which means "swap the execution environment for an isolated sandbox" is a matter of replacing one implementation rather than rewriting the tools.

Logs have safety boundaries too

The session log stores user input, tool results, file fragments, and command output. It may contain secrets, private code, and error stack traces. Consider at least:

  • Where logs are stored and who can access them.
  • Whether they are encrypted.
  • Whether they are redacted before export.
  • Whether the user is asked before uploading them to a remote service.
  • Whether the UI collapses sensitive output by default.

Safety is not just about blocking commands. An agent can avoid executing a single dangerous command and still write the contents of .env into the log or the model context — that's a leak all the same. Reading a likely-sensitive file (.env, private keys, credential files) should have a dedicated policy: deny, confirm, or redact.

Exercise

Implement the permission gate and project trust state.

Acceptance criteria:

  • In an untrusted project, write tools and bash require confirmation or are denied by default, and project extensions and custom prompts are not loaded.
  • Trust decisions are persisted against the path, supporting both "trust the parent directory, overriding subdirectories" and "this session only" granularities.
  • A permission denial becomes an isError: true tool result.
  • The UI, CLI, JSON mode, RPC, and SDK all invoke the same permission gate.
  • Reads of likely-sensitive files have a policy: deny, confirm, or redact.
  • Tool operations go through an interface abstraction that can later be swapped for container or remote execution.