Architecture
- Components
- Egress mediation
- Session lifecycle & durability
- Credential flow
- When a session looks terminated but isn’t
- Rewind & edit
- Data & state
- Project layout
Terrarium is a control plane (the orchestrator + console) that spawns one ephemeral sandbox per session. The sandbox runs two cooperating processes — the agent worker and the Warden egress mediator — sharing a network namespace but isolated by uid and a kernel firewall.
┌──────────────────────────────────────────────────┐
│ Operator │
│ Console (Next.js, :3737) · SDK · curl │
└───────────────────────┬──────────────────────────┘
│ HTTPS + bearer token (scoped)
┌───────────────────────▼──────────────────────────┐
│ Orchestrator (FastAPI, :8900) │
│ sessions · agents · schedules · tokens · logs │
│ egress policy · credentials │
│ │
│ CredentialManager SessionRegistry (sqlite) │
│ (rotating refresh, EventStore (jsonl) │
│ sealed at rest) EgressPolicyStore │
└──────┬───────────────────────────────────┬────────┘
spawns / reattaches │ │ owns the real
(Docker container | k8s Pod)│ │ credential (RAM)
┌──────────▼─────────────────────────────────▼───────┐
│ Session sandbox (ephemeral · single-tenant) │
│ │
│ ┌────────────────┐ loopback ┌──────────────┐ │
│ │ worker │ 127.0.0.1: │ Warden │ │
│ │ Agent SDK + │ ◄─── 8888 ──► │ (Rust) │ │
│ │ Claude Code │ │ TLS-term · │ │
│ │ uid 1001 │ │ inject cred │ │
│ │ *decoy* cred │ │ allow-list ·│ │
│ └────────────────┘ │ DLP · audit │ │
│ │ uid 1002 │ │
│ nftables: default-DROP — └──────┬───────┘ │
│ only lo + Warden's uid may leave │ real cred │
└────────────────────────────────────────────┼─────────┘
▼
api.anthropic.com +
console-allow-listed hosts
Components
| Component | Tech | Role |
|---|---|---|
| Orchestrator | Python · FastAPI (orchestrator/) |
Control plane: REST API, session lifecycle, credential ownership, egress policy, durability. Spawns sandboxes via the Docker, k8s, or local runner. |
| Worker | Python (sandbox/worker.py) |
Drives the Claude Code CLI through the Agent SDK; streams typed events; handles interrupts, rewind, and uploads. |
| Warden | Rust (warden/) |
Per-session MITM egress mediator and the single policy brain — see Security. |
| Console | Next.js · Tailwind · shadcn/ui (web/) |
Operator UI: sessions, agents, usage, schedules, tokens, egress, logs. |
| SDK | Python (sdk/terrarium/) |
Typed client for CI/automation — see SDK. |
| Shared protocol | (terracore/) |
The worker↔orchestrator event/command protocol, harness model, personas, event store. |
Egress mediation
Every outbound request from the agent is forced through Warden, checked against the live policy, and re-credentialed at the boundary:
agent (uid 1001) Warden (uid 1002) upstream
│ HTTPS_PROXY=127.0.0.1:8888 │ policy.json (hot-reload) │
│ CONNECT host:443 ─────────────────►│ host allow-listed? │
│ │ ├─ no → DENY + audit ─────────┤
│ │ └─ yes → TLS-terminate, │
│ │ strip decoy header, │
│ │ inject REAL cred ─────► api.anthropic.com
│ ◄───────────────────────────────────│ ◄────────────────────────────────┤
▼ ▼
raw IP / other host → kernel DROP HMAC-signed receipt → audit.jsonl
Policy and credential are both hot-reloaded: editing the allow-list in the console (or a credential refresh) is patched into each running session’s Warden, which re-reads it by mtime — so changes take effect on live sessions, not just new ones.
Session lifecycle & durability
create ─► resolve agent/harness ─► spawn sandbox: [warden sidecar] + [worker]
│ policy · cred · CA · audit
▼
worker connects to Claude ─► EV_READY
│
┌──────────────┬───────────┼────────────┬──────────────┐
▼ ▼ ▼ ▼ ▼
message interrupt rewind upload idle
│ (files / │
▼ convo) ▼
run turn ───────────────────────────────────────► durable (sandbox kept)
│
└──────────► event log (jsonl, append-only source of truth)
│
orchestrator restart ─► rehydrate: reattach RUNNING sandboxes, reap orphans
│
stop ─► reap sandbox + per-session policy/cred/CA
The event log is the durable source of truth — the registry is a cache, and a restarted orchestrator replays/reattaches from the logs plus live sandboxes. Sessions persist across orchestrator restarts (not sandbox restarts).
Credential flow
The orchestrator owns the credential because a subscription token’s rotating refresh chain can’t live in an ephemeral, per-session Warden:
CredentialManager (orchestrator) per-session Warden
├─ refresh before expiry (15-min skew) reads cred (mtime hot-reload)
├─ rotate + persist (sealed: AES-256-GCM, KEK) ▼
├─ exponential backoff on 429 (no hammering) inject at egress
└─ on change → patch every live session's cred ─────────► (sandbox sees a decoy)
A console “re-paste” or an auto-refresh propagates to running sessions, so long-lived agents don’t 401 when their start-time token expires.
When a session looks terminated but isn’t
The orchestrator reads a session’s events over a stream that is a client of the sandbox —
for the Docker runner, a docker attach subprocess — not the sandbox itself. That stream can
end while the container is still running and the agent is still healthy: a Docker daemon
restart, a host suspend, an idle connection dropped.
Treating the first end-of-stream as death is what used to mark healthy sessions terminated
after a day or two of idling. The pump now probes the sandbox before concluding anything, and
reattaches up to five times with backoff when the container is still running. Only a sandbox
that is genuinely gone produces worker_lost.
For a session stranded before that — or past the retry budget — reattach explicitly:
curl -X POST localhost:8900/v1/sessions/<id>/recover
It refuses with 409 unless the sandbox is actually running: recovery is a fact about the
container, never an assumption. A 409 means the sandbox is gone and the transcript is all
that remains — it stays readable and downloadable, but the conversation cannot be resumed,
because the CLI’s own transcript lived inside that container.
Rewind & edit
Built on the Agent SDK’s native checkpointing. A per-turn anchor (the user-message uuid) lets you, from the console:
- Edit this message — truncate the conversation to before that turn and drop its text back
into the composer (transcript truncate +
resume). - Restore workspace to here — roll files back to that turn (
rewind_files). - Rewind + restore workspace — both.
A failed rewind is a no-op that keeps the session alive; it never crashes the worker.
Data & state
All state is the operator’s; nothing leaves your infrastructure except agent-authorized egress.
| Store | Path (under runtime/ or logs/) |
Contents |
|---|---|---|
| Event log | logs/<session>.jsonl |
Per-session, append-only stream of typed events. The durable source of truth. |
| Session registry | runtime/sessions.db (SQLite) |
Session metadata cache: id, agent, status, model, cost, last_seq, memory volume. Rebuildable from the logs. |
| Egress policy | runtime/egress/policy.json |
Console-managed mode + rules + kill switch. Warden’s source of truth. |
| Egress audit | runtime/egress/audit/<session>.jsonl |
HMAC-signed decision receipts, one independently verifiable chain per session. Persists after the sandbox is gone, on every runner, so terra verify-egress works post-mortem. |
| Credential store | runtime/credentials.json |
The subscription/API credential, sealed at rest (AES-256-GCM; KEK = TERRA_KEK, else TERRA_TOKEN). |
| Agent registry | runtime/agents.json |
Reusable harness configs. |
| Schedules | runtime/schedules.json |
UTC, 5-field cron recurring sessions. |
| API tokens | runtime/tokens.json |
Scoped bearer tokens (hashed) for CI/cron callers. |
| Memory volume | per-agent volume (shared) · per-session (isolated) | The agent’s persistent /memory. ReadWriteOnce → single-writer; a concurrent run for the same agent gets its own isolated volume. |
Every durable JSON store is written 0600 through a single JsonStore base.
The Logs console view unifies the event logs and the egress audit into one filterable stream (by agent, session, source, type, and a host/content search). It bounds its fan-out and reports when a cap truncated the result, so a partial view never reads as a complete one.
Project layout
orchestrator/ FastAPI control plane — api, manager, runners (docker/k8s/local),
credentials, secret_store, egress, store, filebridge, registry, schedules
warden/ Rust egress mediator — proxy, policy, inject, ca, audit (+ cargo tests)
sandbox/ The sandbox image — Dockerfile, worker.py, entrypoint.sh, firewall.sh
terracore/ Shared lib — protocol, harness, personas, models, toolset, events
sdk/ Python SDK (terrarium) + CLI
web/ Next.js console — Console, SessionView, EgressView, LogsView, …
docs/ This site
tests/ Unit suite + red-team egress tests