0004: Supervisor, work distribution and sandbox performance¶

Status: deferred (2026-10-15): the supervisor, planner and scheduler v2 are not scheduled; fair-share job slots shipped instead (#89). Was: draft. · Date: 2026-10-12. Author: hangar agent session, for review.

Part of a set. Ownership, cross-spec decisions and the combined build order are in docs/README.md; where this spec conflicts with that page, the page wins. Builds on 0001 (runs, events, queue), 0002 (results, fan-out, landing) and 0003 (grants, scopes). Spec 0005 (traces and decisions) is being written in parallel; section 5.9 states what this spec assumes from it.

1. Summary¶

• A Task is a goal split into a graph of sub-tasks. A deterministic supervisor in the control plane dispatches them, watches 0001's events, and nudges, restarts, reassigns or escalates when a run stalls, loops or overspends.
• An LLM is used in two narrow places: planning (a run in a sandbox) and choosing between enumerated options at decision points (0005's classify). Code validates and executes every action.
• A scheduler layer on 0001's queue adds priority, fair share between users, plan-aware pacing for the shared Claude subscription, and tier-based routing (cheap or strong model per sub-task).
• Sandbox speed work: measure first (scripts/bench_*), then slim images, pre-pull, predictive warm pools, right-sizing, and workspace checkpoints so Spot loss costs minutes, not the task.
• Nothing here can push, merge or approve. Landing stays with 0002 and a human.

2. Problem and evidence¶

Evidence is the first live GKE run (2026-10-11) and the code at c21c598. Numbers marked measured come from that run. Everything else is an estimate or a question.

Research this spec follows (same sources as 0001–0003):

• Jarmak, Engineering Reliable Coding Agents: keep recovery outside the failure domain, make retried steps idempotent, verify the workspace, compare repeats, widen autonomy one action class at a time using measured outcomes. The supervisor's autonomy levels (§7) follow this.
• Willison's lethal trifecta: agent text is attacker-influenced. The LLM parts of the supervisor see it, so they get no tools and choose only among options code enumerates (§6).

3. Goals / non-goals¶

Goals:

1. A durable Task object that outlives the orchestrator's chat. The orchestrator can disconnect and come back.
2. A dependency graph of sub-tasks with file-scope partitioning, so parallel work does not collide.
3. A supervisor that detects stalls, loops, overspend and crashes, and acts through a fixed ladder (nudge, restart from checkpoint, reassign, escalate), with hard stops.
4. A scheduler that gives priority, fairness, plan-aware pacing and tier routing, on top of 0001's queue.
5. Faster, cheaper sandboxes, with a benchmark harness and numbers to hold the line.
6. Every new MCP tool is additive. Existing tools keep working.

Non-goals:

• Replacing 0001's run supervisor (the in-sandbox hangar-run and the watcher). That stays 0001's. This spec's supervisor sits one level up and uses runs.
• Landing, approvals, CI waiting (0002). Credentials, egress (0003). The traces store and decision model (0005).
• Semantic grading with an LLM judge. Ranking stays mechanical (0002).
• Preempting a running run to free a slot. v1 only orders the queue.
• Multi-replica control plane. State stays in SQLite.
• Money accounting on subscriptions. Budgets use minutes and tokens; dollars only when a harness reports them (§5.7).

4. User stories¶

Orchestrator over MCP (e.g. a Claude chat):

• I call task_create(goal, environment, repo). hangar plans it, shows me the plan, and I approve it. I get one task_id and disconnect.
• I return later, call task_status(wait_seconds=60), and see "3 of 5 sub-tasks done, 1 nudged for looping, 1 waiting on a human decision".
• I pass my own sub-task list and skip the planner.
• I call task_intervene to restart one sub-task with a stronger model. I do not hand-manage sandboxes.

Human on the dashboard:

• I see the graph, which node is on which sandbox, and what the supervisor did and why ("nudged run-3fa9: same failing test 4 times, no files changed").
• I get one notification only when a decision is mine: budget raise, scope violation, repeated failure.
• I set a budget once and know the task stops at it.
• Another user's 30-node task does not starve my single run.

Operator:

• I see warm-pool hit rate, queue wait, node scale-up time and cost per task, and I can tell whether a change made things faster.
• A Spot preemption costs a few minutes of one sub-task, not the task.

5. Design¶

5.1 Data model (models.py, store.py)¶

New tables go in store.py:_TABLES as JSON rows, like the rest. Runs stay 0001's table.

class TaskState(StrEnum):
    PLANNING = "planning"            # planner run is working (skipped when subtasks were given)
    PLAN_REVIEW = "plan_review"      # waiting for plan approval
    RUNNING = "running"
    PAUSED = "paused"                # user paused: no new dispatch, running runs finish
    BLOCKED = "blocked"              # nothing runnable and a human decision is open
    COMPLETING = "completing"        # all required sub-tasks succeeded; result roll-up
    DONE = "done"                    # every required sub-task succeeded and verified
    FAILED = "failed"                # a required sub-task failed for good
    STOPPED = "stopped"              # a guard fired: budget, deadline, escalation timeout
    CANCELLED = "cancelled"
TASK_TERMINAL = {DONE, FAILED, STOPPED, CANCELLED}

class SubTaskState(StrEnum):
    PENDING = "pending"              # dependencies not met
    READY = "ready"                  # dependencies met, scope free, not yet queued
    QUEUED = "queued"                # has a Run in 0001's queue
    RUNNING = "running"
    VERIFYING = "verifying"          # 0002 stage pipeline for the winning run
    SUCCEEDED = "succeeded"
    FAILED = "failed"
    SKIPPED = "skipped"              # by a human or by an optional-node rule
    CANCELLED = "cancelled"
    NEEDS_HUMAN = "needs_human"      # an Escalation is open

class Edge(BaseModel):
    frm: str; to: str                # sub-task ids
    kind: Literal["needs", "after"] = "needs"
    # needs: `to` starts from `frm`'s pushed head sha (stacked branch). `frm` must be succeeded.
    # after: ordering only. `to` starts from the task base. Used to serialize overlapping scopes.

class Scope(BaseModel):
    writes: list[str] = []           # globs the sub-task may change, e.g. ["src/hangar/fleet.py", "tests/test_fleet.py"]
    reads: list[str] = []            # informational, never enforced
    shared: list[str] = []           # shared files it must touch (lockfiles, registries). See 5.3.

class Route(BaseModel):              # how to run it; filled by the router (5.5), overridable
    tier: Literal["cheap", "standard", "strong"] = "standard"
    harness: str | None = None
    model: str | None = None
    credential_pool: str | None = None
    attempts: int = 1                # >1 = parallel best-of-N through 0002 fan-out
    max_retries: int = 2             # agent-caused failures
    max_infra_retries: int = 2       # Spot loss, node loss: not the agent's fault

class SubTask(BaseModel):
    id: str                          # "st-" + 6 hex, or a caller-chosen slug (unique in the task)
    task_id: str
    title: str
    prompt: str                      # full instructions, redacted with broker.redact
    kind: str = "change"             # change | test | docs | investigation | refactor (router hint)
    scope: Scope = Scope()
    route: Route = Route()
    verify: str | None = None        # overrides repo.verify (0002 decision 2)
    required: bool = True
    state: SubTaskState = SubTaskState.PENDING
    attempts_made: int = 0
    infra_retries: int = 0
    nudges: int = 0
    run_ids: list[str] = []          # every Run, in order. The last non-terminal one is current.
    fanout_id: str | None = None     # when route.attempts > 1
    winner_run_id: str | None = None
    branch: str | None = None; head_sha: str | None = None
    land_id: str | None = None
    checkpoint_id: str | None = None
    budget: Budget | None = None     # optional per-node cap, inside the task cap
    notes: list[str] = []            # hangar-written, never agent text

class Budget(BaseModel):
    wall_minutes: int | None = None          # since task start (sub-task: since its first run)
    agent_minutes: int | None = None         # sum of run durations. The universal cost proxy on subscriptions.
    tokens: int | None = None                # input + output + 0.1 x cache_read (weights in config)
    usd_est: float | None = None             # API-equivalent, only for harnesses that report it
    max_runs: int | None = None
    max_decisions: int = 30                  # LLM decision calls for this task (5.9)

class Plan(BaseModel):
    version: int
    subtasks: list[SubTask]
    edges: list[Edge]
    source: Literal["caller", "planner", "amended"]
    planner_run_id: str | None = None
    lint: list[dict] = []                    # {code, severity, message, nodes}
    approved_by: str | None = None; approved_at: float | None = None

class Task(BaseModel):                       # table "tasks"
    id: str                                  # "task-" + 8 hex
    goal: str
    owner: str                               # Principal id (0003)
    environment: str; provider: str | None
    repo: str | None; base_branch: str | None; base_sha: str | None
    state: TaskState = TaskState.PLANNING
    state_reason: str | None = None
    priority: int = 5                        # 0 (lowest) to 9
    queue_class: Literal["interactive", "task", "batch"] = "task"
    autonomy: Literal["observe", "nudge", "restart", "full"] = "nudge"   # 5.4
    plan_approval: Literal["auto", "manual"] = "manual"
    plan: Plan | None = None
    budget: Budget; spent: Budget            # spent is a ledger roll-up, not trusted input
    credentials: list[str] = []
    idempotency_key: str | None = None
    escalations: list[Escalation] = []
    created_at: float; started_at: float | None; finished_at: float | None
    trace_id: str | None                     # 0005 trace for this task
    event_seq: int = 0

Fields added to 0001's Run (additive, all optional): task_id, subtask_id, attempt_no, priority, queue_class, owner, model (what was requested), and summary.resources (peak_mem_mb, cpu_seconds, disk_mb; §5.8). They are columns in runs so the dispatcher can sort in SQL.

class Escalation(BaseModel):
    id: str                                   # "esc-" + 6 hex
    task_id: str; subtask_id: str | None; run_id: str | None
    kind: str          # budget_near, budget_hit, loop, stalled, repeated_failure, scope_violation,
                       # question, plan_invalid, escalation_timeout, no_credential
    summary: str       # hangar-built from facts. Never agent text.
    evidence: dict     # event seqs, counters, hashes. Agent excerpts are in `untrusted`.
    untrusted: dict    # {"last_message": "...", "tool_summaries": [...]}, labelled source=agent
    options: list[Option]                     # enumerated by code (5.4)
    human_only: bool = False
    decision_id: str | None                   # 0005 record, if classify was consulted
    state: Literal["open", "resolved", "expired"] = "open"
    resolved_by: str | None; choice: str | None; note: str | None
    opened_at: float; expires_at: float

class Option(BaseModel):
    id: str; label: str; default: bool = False
    action: dict                              # a typed action, e.g. {"type":"nudge","template":"loop_break"}
    needs_scope: Literal["operate", "approve"] = "operate"

class Checkpoint(BaseModel):                  # table "checkpoints"; blobs in data_dir/checkpoints/
    id: str; run_id: str; subtask_id: str; taken_at: float
    head_sha: str; base_sha: str
    bundle_path: str | None                   # `git bundle` of base..HEAD (commits)
    patch_path: str | None                    # `git diff HEAD --binary` (tracked, uncommitted)
    untracked_path: str | None                # tar of non-ignored untracked files, size-capped
    transcript_path: str | None               # harness session file, when the harness has one
    bytes: int

Tables: tasks, checkpoints, usage_ledger(run_id, task_id, owner, credential, ts, minutes, tokens, usd_est), scheduler_samples(ts, credential, running, queued, limit, usage_pct), claim_history(ts, env, provider, wait_ms, warm_hit). The last two feed pacing and pre-warming (§5.5, §5.8). Retention for samples: 14 days.

5.2 Who does what: code vs LLM¶

Models and harness.

• Planner: strong tier, harness claude-code by default (the only harness with a confirmed run, resume and event path today). Other harnesses are allowed once 0001 M7 ships them. It runs as a hangar run, so it uses the same credential, queue and sandbox rules as any agent. It counts against the subscription.
• Classify: the 0005 service. This spec assumes it calls a model API with a control-plane API-key credential (Credential.purpose = "control-plane", 0002 decision 5), a cheap tier model by default, a spend cap, and no tools. It must not use the shared subscription token: README decision 9 keeps that token inside sandboxes until the owner decides on the proxy. With no such credential, classify is unavailable and every "ask" becomes an escalation to a human (§5.9).
• Tier names map to (harness, model) in config, never to hard-coded model ids (§5.5).

What the orchestrator and humans drive. Everything through the same tools and REST routes (§5.6). An orchestrator may play the planner itself (pass subtasks) or approve a plan. It cannot resolve human_only escalations or approve landing.

5.3 Dependency graph, partitioning and merge-conflict avoidance¶

The plan is a DAG. Lint runs on every task_create, planner output and task_amend, and returns Plan.lint. Errors block approval.

Overlap test. Two globs overlap if their literal directory prefixes nest or one pattern can match the other's literal path (fnmatch). It is deliberately conservative: a false "overlap" costs parallelism, a false "no overlap" costs a conflict.

Cross-task check. The dispatcher keeps an in-memory registry of active write scopes per (repo, base_branch) across all tasks. A node whose scope overlaps a running node of another task stays READY with note scope_busy:<task>. It does not fail. Nodes with no declared scope skip this check; 0002's overlaps still catches them at land time.

Stacked branches (needs). The successor's sandbox is created normally, then git fetch + git checkout -b <work_branch> <pred.head_sha> (extends _setup_repo; the predecessor's branch is on GitHub because 0002 publishes it). Landing order is topological. After a predecessor lands, 0002's updating step rebases or merges the successor onto the new base. If that conflicts, 0002 conflict fires and the sub-task gets an escalation (§5.4).

Runtime scope check. When 0002's collector has the diff (or earlier, from summary.files_touched), any path outside scope.writes raises scope_violation. The ladder is: nudge once ("revert changes outside: "), then escalate. A run with an unresolved violation cannot be landed by task_land, and the Inbox card shows the paths. This is detection, not prevention. The gateway does not enforce path scopes (0003 non-goal), and this spec does not add that.

5.4 Supervisor: signals, ladder, guards¶

The supervisor is a control-plane task, TaskEngine, started next to _reaper_loop and 0001's _run_monitor. It is event-driven (it subscribes to 0001's per-run condition) with a 5 s safety tick. All state is in SQLite. Every action is written to an intent log row (task_id, action_id, status) before it is executed, and action_id is the idempotency key passed to resume_agent / cancel_agent. After a restart it replays unfinished intents.

Detectors (pure functions over a run's events and summary; unit-testable with fixtures):

Policy table (the default; one YAML file, data_dir/supervisor.yaml, per-repo overrides). "Count" is how many times this signal already fired for this node.

autonomy caps what the supervisor may do on its own. At observe it detects, logs and notifies only; at nudge it adds nudges; at restart it adds restarts, retries and infra recovery; at full it adds reassign and classify-driven choices. When an action is above the task's autonomy, an escalation opens with that action as the default option. This is how autonomy is widened one action class at a time (§7).

Hard stops. These are deterministic and cannot be overridden by anything but a human raising the budget: task agent_minutes, tokens, usd_est, wall_minutes, max_runs, max_decisions, and a global HANGAR_TASK_MAX_RUNS_PER_TASK (default 40). A hard stop cancels runs with cancel_agent(grace_seconds=10) and verifies with a probe, per 0001.

What the supervisor never does. It never calls exec with content it did not build from templates, never writes to a repo, never creates a credential or grant, never raises a budget or a tier cap, never calls anything in 0002's land path except task_land on a human's request.

5.5 Distribution of work: scheduler¶

0001 owns the per-credential queue and usage tracker. This section changes how that queue is ordered and limited and adds routing. It is a change to 0001 M4 (see §5.11).

Queue key. Each queued run gets (class_rank, -effective_priority, fair_share_rank, enqueued_at).

• queue_class: interactive (a human or orchestrator calling run_agent directly; default for calls that carry no task_id) before task before batch.
• effective_priority = priority + min(3, minutes_waiting / 15). Aging stops starvation of low priority.
• fair_share_rank orders owners with equal class and priority. Each owner has a decayed usage score: agent-minutes from usage_ledger, half-life 6 h, divided by the owner's weight (default 1). Lowest score runs first. A 30-node task by user A therefore does not block user B's one run.
• Per-owner cap, work-conserving: when two or more owners have queued or running work on a credential, one owner may hold at most ceil(0.6 × limit) slots. With one owner there is no cap.

Credential pools. A pool is a named list of agent credentials for the same harness family, in data_dir/routing.yaml:

pools:
  claude-main:   {members: [claude-max], strategy: least_loaded}
  claude-api:    {members: [anthropic-key-1], strategy: least_loaded, spend_cap_usd_day: null}
tiers:
  cheap:    [{harness: claude-code, model: haiku,  pool: claude-main}]
  standard: [{harness: claude-code, model: sonnet, pool: claude-main}]
  strong:   [{harness: claude-code, model: opus,   pool: claude-main}, {harness: claude-code, model: opus, pool: claude-api}]
kinds:
  docs: cheap
  test: cheap
  change: standard
  refactor: strong
  investigation: standard

The alias --model <alias> is confirmed in claude --help 2.1.296 (aliases like sonnet, opus; full names also accepted). I did not verify which aliases map to which Claude 5 model, nor -m equivalents for Codex, Gemini or agy. §10 Q3. The router writes the choice into SubTask.route; a caller can override any field. A tier entry is a fallback list: the first entry whose pool has headroom wins.

Plan-aware pacing for the shared subscription. 0001's gate is a fixed percentage (95) and a fixed concurrency (2). This spec makes both adaptive and class-aware.

• Class gates: batch stops at 80% of the 7-day window, task at 90%, interactive at 95% (0001's default). That keeps headroom for people. Values are config. They read 0001's usage:<credential>:<window> store, so they are only as good as rate_limit_event parsing (0001 OQ1).
• Adaptive concurrency (AIMD): the effective limit starts at the configured value (default 2). On any failed/rate_limited or a gateway 429 (0003 M9, if shipped), halve it (min 1) and hold for 10 minutes. After 30 minutes without throttling and with usage under 80%, add 1, up to a hard cap (HANGAR_MAX_RUNS_PER_CREDENTIAL_CAP, default 4). The samples in scheduler_samples and the live acceptance scenario G (0001 §8.2) calibrate the cap. Nobody knows the true Pro limit yet.
• Burn projection: from the ledger, burn = Δ usage_pct / run-hour over the last 24 h. If usage_pct + burn × hours_to_reset would cross the task gate before resets_at, new batch work is held with reason plan_projection. Projection is shown on the dashboard; it is advisory for task and interactive.
• Pool fallback: a sub-task whose pool is gated is routed to the next pool in its tier if the task allows it (credentials on the task). Otherwise it waits with a clear reason. It does not fail.
• API-key credentials have no 7-day plan window, only rate limits and money. They are used for width: strong tier fallback, and sub-tasks with attempts > 1.

Parallel vs sequential attempts.

• Default attempts = 1.
• Parallel (attempts = 2..3, via 0002 fanout_start with strategy="fresh", graded by 0002) is chosen only if all of these hold: the node is flagged risk: high or is on the critical path; the pool has at least attempts free slots now; usage is under 70%; the task's budget has room for attempts × the node's estimate. Otherwise it is one attempt.
• Sequential retry (§5.4) always changes something: tier, harness, or a handoff prompt with the failure facts. Retrying the same thing at the same tier is not offered. Single runs vary (Jarmak), but identical retries burn the same limit.
• The fanout_id is stored on the sub-task. The winner's run_id is winner_run_id; losers are cancelled once the winner is picked, and their sandboxes destroyed unless the task says keep.

Placement onto nodes. hangar does not schedule pods. Kubernetes does. hangar's levers are the resource requests it writes, the warm-pool size, and the node-pool settings in §5.8.

• Right-sized requests (§5.8) make bin-packing work. n2-standard-8 is 8 vCPU / 32 GB, but allocatable is lower (GKE reserves for system and daemonsets). I did not measure it; M0 records kubectl describe node. With 2 CPU / 4 GiB sandboxes, expect about 6–7 per node.
• The autoscaler optimize-utilization profile makes the default scheduler favor packing and lets underused nodes drain faster. §10 Q8 asks whether it applies to this cluster and node pool.
• Anti-affinity is not used: spreading sandboxes across nodes would block scale-down.
• Docker: one host, no packing problem. max_sandboxes and host cores are the limit. The scheduler's slots_free also checks docker info NCPU against the sum of nano_cpus of running sandboxes and holds new runs when it exceeds 100% (a single laptop thrashes otherwise).
• E2B: placement is E2B's. hangar's limit is the account's concurrent-sandbox limit (unknown; §10 Q13).

5.6 MCP tools and REST¶

All tools are additive and take a task id. Scopes use 0003's names: read, operate, land, approve.

task_create(goal: str, environment: str, repo: str | None = None, branch: str | None = None,
            subtasks: list[dict] | None = None, edges: list[dict] | None = None,
            budget: dict | None = None, priority: int = 5, queue_class: str = "task",
            autonomy: str = "nudge", plan_approval: str | None = None,   # default: "auto" if subtasks given, else "manual"
            credentials: list[str] | None = None, provider: str | None = None,
            ttl_minutes: int | None = None, name: str | None = None,
            idempotency_key: str | None = None) -> dict                  # operate
task_status(task_id: str, wait_seconds: int = 0, include: list[str] | None = None) -> dict   # read; wait <= 25
task_events(task_id: str, after: int = 0, limit: int = 200, wait_seconds: int = 0) -> dict   # read
task_plan_approve(task_id: str, plan_version: int, edits: list[dict] | None = None) -> dict  # operate
task_amend(task_id: str, ops: list[dict], expected_version: int) -> dict                     # operate
task_intervene(task_id: str, subtask_id: str, action: str, message: str | None = None,
               tier: str | None = None, idempotency_key: str | None = None) -> dict          # operate
task_resolve(task_id: str, escalation_id: str, choice: str, note: str | None = None) -> dict # operate; human_only needs approve (REST only)
task_control(task_id: str, action: str, reason: str | None = None) -> dict                   # operate: pause|resume|cancel
task_list(state: list[str] | None = None, owner: str | None = None, limit: int = 50) -> list[dict]  # read
task_land(task_id: str, method: str | None = None) -> dict                                   # land
scheduler_status(credential: str | None = None) -> dict                                      # read

Notes:

• wait_seconds is capped at 25, like agent_events. task_status returns early on any sub-task or task state change.
• task_amend ops: add_subtask, remove_subtask, edit_subtask, add_edge, remove_edge. It bumps plan.version and re-lints. Running nodes cannot be edited (only budget, route); remove-and-add makes a new node.
• task_intervene actions: nudge, restart, reassign (needs tier), retry, skip, cancel. A human or orchestrator may use any of them regardless of the task's autonomy (autonomy limits the supervisor). message is delivered through resume_agent as the nudge body. This is the orchestrator's text, not the model's, so it is allowed to be free text.
• task_resolve for an escalation with human_only: true returns insufficient_scope: requires 'approve' over MCP, as 0002 does for land approval. The REST route requires the approve scope.
• task_land calls 0002 land_request for each succeeded node in topological order and waits for each predecessor to be landed before requesting the next. It needs land. It never approves.
• task_control(cancel) cancels runs, cancels queued runs, leaves sandboxes to their TTL unless destroy_sandboxes is set (REST only).

Example: create with a caller-provided plan.

{"goal": "Add retry with backoff to the GitHub client and document it",
 "environment": "hangar-dev", "repo": "Sean-Koval/hangar", "branch": "main",
 "budget": {"agent_minutes": 90, "tokens": 4000000, "wall_minutes": 120},
 "priority": 5, "autonomy": "restart",
 "subtasks": [
  {"id": "client", "title": "Retry in github client", "kind": "change", "prompt": "In src/hangar/hosts/github.py add ...",
   "scope": {"writes": ["src/hangar/hosts/github.py", "tests/test_github_host.py"]}},
  {"id": "docs", "title": "Document retry settings", "kind": "docs", "prompt": "Add a section to docs/landing.md ...",
   "scope": {"writes": ["docs/landing.md"]}, "route": {"tier": "cheap"}},
  {"id": "cfg", "title": "Config knobs", "kind": "change", "prompt": "...",
   "scope": {"writes": ["src/hangar/config.py"], "shared": ["uv.lock"]}}],
 "edges": [{"frm": "client", "to": "docs", "kind": "needs"}]}

Response:

{"task_id": "task-5b0c91de", "state": "running", "plan": {"version": 1, "source": "caller", "lint": []},
 "ready": ["client", "cfg"], "pending": ["docs"], "warnings": ["claude-max 7-day at 91%, resets 2026-10-16T14:00Z"],
 "events_cursor": 0, "dashboard_url": "https://hangar.example.com/#task/task-5b0c91de"}

task_status (trimmed):

{"task_id": "task-5b0c91de", "state": "running", "elapsed_s": 912,
 "budget": {"agent_minutes": {"used": 31, "cap": 90}, "tokens": {"used": 1210000, "cap": 4000000}, "wall_minutes": {"used": 15, "cap": 120}},
 "subtasks": [
  {"id": "client", "state": "running", "attempt": 1, "run_id": "run-3fa91c20", "tier": "standard", "phase": "running_tests", "nudges": 0},
  {"id": "cfg", "state": "needs_human", "attempt": 2, "escalation": "esc-a1b2c3"},
  {"id": "docs", "state": "pending", "waiting_on": ["client"]}],
 "escalations": [{"id": "esc-a1b2c3", "kind": "repeated_failure", "human_only": false, "summary": "cfg failed verification twice (uv run pytest -q: 3 failed)",
   "options": [{"id": "retry_stronger", "default": true}, {"id": "skip"}, {"id": "fail_task"}]}],
 "supervisor": {"last_action": {"ts": 1791766900.1, "type": "nudge", "subtask": "client", "why": "loop_test: same 2 failing tests x3, no diff change"}},
 "events_cursor": 57}

task_events returns the supervisor timeline: task.state, plan.*, subtask.state, signal, action, decision, escalation.*, budget. Each has seq, ts, type, data; fields that quote agent text are under untrusted.

REST (bearer or UI ticket, same as app.py):

5.7 Budgets and cost per task¶

• Minutes and tokens are the primary caps. They exist for every harness that 0001's parser supports and make sense on a subscription.
• usd_est uses the harness's reported cost_usd (Claude's total_cost_usd). On a subscription it is an estimate (0001 non-goal). It is shown labelled "API-equivalent".
• --max-budget-usd exists in claude --help 2.1.296 ("only works with --print"), but I did not verify how it behaves with a subscription token. hangar does not rely on it. The in-control-plane guard is the cap. It may be passed as a second line of defence for API-key credentials (§10 Q4).
• Cost per task = agent_minutes, tokens, usd_est (if any), sandbox-seconds × a configured HANGAR_NODE_USD_PER_HOUR_PER_VCPU (unset by default; I do not assume prices), plus the decision-call cost reported by 0005. All rolled up from usage_ledger.

5.8 Sandbox performance¶

The principle: measure with scripts/bench_* first (M0), change one thing at a time, and keep a result file per run.

Benchmark harness (scripts/bench_*.py)¶

Each script runs against any hangar URL and token, prints one JSON object per sample to stdout, and appends to bench/results/<date>-<gitsha>-<provider>.jsonl. scripts/bench_report.py compares two result files (median, p95, n) and exits non-zero when a metric regresses past its threshold. Live scripts are run by hand or nightly. They never run in the unit suite.

Metrics tracked (also exported at GET /api/perf/summary from the same tables):

Targets¶

Baselines marked measured come from the 2026-10-11 run. The others are targets until M0 records a baseline; M0 replaces guesses with numbers and this table is updated in that PR.

Image size and layers¶

• 0001 M9 already orders layers (harnesses and tools before the repo bake layer, lock digest first inside it). This spec adds a shared base: a hangar-base image (Ubuntu + git/tmux/rg + mise + python/uv/node) published once, with env images as thin layers on top. Nodes then pull the shared layers once and reuse them across environments. Envs already extends: base (envs/base.yaml), so this is a build change, not a spec change. Expected gain: second and later environments pull only their own layers. Unmeasured.
• Slimming in envspec.render: apt-get install --no-install-recommends, remove /var/lib/apt/lists, no pip/uv cache in the final layer unless deps.cache asks for it, mise cache removed, no docs/man pages, one RUN per install group so a layer is not half-deleted. bench_image.py reports the biggest layers first; the target is set after reading them.
• Pre-compile bytecode in the bake layer (uv sync --compile-bytecode) so the first import in a sandbox does not write .pyc files into the memory-backed root. Flag exists in uv; I did not time it here.
• Do not strip the harnesses. Claude Code's installer size is unknown to me; it is probably the largest single layer.

Image distribution on GKE¶

Options, in order of confidence. None is verified on gVisor nodes.

HANGAR_PREPULL=on|off renders the DaemonSet from the set of images currently referenced by SandboxTemplates (sync_warm_pool already records dev.hangar/image).

Warm pools and predictive pre-warming¶

warm_pool in the env spec becomes the minimum. A WarmPoolController loop (every 30 s, in Fleet) computes the target per (provider, env) and calls provider.sync_warm_pool(spec, image, replicas=target) (new optional argument; today it uses spec.warm_pool).

target = clamp(max(spec.warm_pool, forecast, pending_demand), 0, warm_max[env])
forecast       = p90 of claims per 5-minute window at this hour-of-week over the last 4 weeks (claim_history)
pending_demand = ready sub-tasks of approved tasks that are not yet queued, minus idle warm sandboxes, capped by credential headroom

• Scale up immediately, scale down slowly: a target below the current size holds for 15 minutes first (no oscillation). Scaling down never kills a claimed sandbox, only unclaimed pool pods.
• Task-aware: approving a plan with k ready nodes raises pending_demand by min(k, headroom) before the first create. That is also the "balloon" for node scale-up. Warm pods ask the autoscaler for a node before any run needs it, so the 1–2 min scale-from-zero overlaps with plan review.
• Cost cap: warm_max[env] (default 4) and HANGAR_WARM_MAX_TOTAL_VCPU bound idle spend. Everything is reported in perf/summary.
• Rollout mode HANGAR_WARM_AUTOSCALE=off|observe|on. In observe it logs what it would do.
• Docker: same loop, sync_warm_pool already tops up containers. Warm containers cost RAM on a laptop, so warm_max defaults to 1 there. E2B has no warm pools (Capabilities.warm_pools=False); the lever is template size and keeping sandbox timeout short.

Node autoscaling and Spot¶

• min_node_count: keep 0 as the default. Add a Terraform variable sandbox_min_nodes and a schedule helper (hangar node-schedule, or Cloud Scheduler hitting gcloud container clusters resize) for teams that want 1 node during working hours. A warm node costs money all day; the warm-pool controller usually makes it unnecessary.
• autoscaling_profile = OPTIMIZE_UTILIZATION on the cluster, evaluated with bench_start (§10 Q8).
• Machine shape: compare n2-standard-8 against 2 × n2-standard-4 on node-scale time and packing. Bigger nodes pack better but a Spot reclaim kills more sandboxes at once. Decide from bench_preempt and the right-sizing data. No prices are assumed here.
• Spot preemption. A Spot VM gets a short notice before reclaim (GCE documents a 30-second ACPI notice; I did not confirm what GKE passes to pods on gVisor). The design does not depend on any notice:
• Checkpoint on a timer: every checkpoint_every_minutes (default 10) the watcher takes a checkpoint by exec: git bundle create base..HEAD, git diff HEAD --binary, a size-capped tar of untracked non-ignored files, and the harness session file when the harness has one (Claude: ~/.claude/projects/<slug>/<session>.jsonl). It is read-only toward git, pushes nothing, and writes to data_dir/checkpoints/. Cost: one exec of a few seconds on a sandbox with a small diff. Skipped if nothing changed.
• Detect: 0001's crashed/sandbox_gone. A new provider hook explain_loss(sandbox) -> "preempted" | "evicted" | "oom" | "unknown" reads the pod's last status and events (Kubernetes) or the Docker OOMKilled flag. Exact Kubernetes reasons for Spot reclaim on GKE are unverified (§10 Q7). unknown is treated as infra loss.
• Recover: infra_loss policy (§5.4): new sandbox from the warm pool, git fetch the base, git bundle unbundle, git apply, untar, then resume_agent equivalent if the session file was restored, else a new run with a handoff prompt built from facts: the task prompt, git log --oneline base..HEAD, git diff --stat, and "continue; do not redo finished work".
• Budget: loss does not consume an attempt, but it does consume agent_minutes. After max_infra_retries, escalate: the node may be too large for Spot.
• Restoring a session file across sandboxes of the same environment is untested; 0001 M10 owns the transcript archive. If the session cannot be resumed, the handoff prompt path is the fallback and is harness-agnostic.
• Sandboxes on a Kata/Docker host that is not preemptible do not need this, but the same checkpoint also serves restart and reassign.

Dependency caches: baked vs shared read-only volume¶

0001 M9 rejects shared writable volumes (poisoning path, docs/security/README.md). This spec keeps that and compares the read-only options.

Memory pressure: because the root is memory-backed, baked dependencies help twice. They are in the image's lower layer, not in the sandbox's writable layer. A claim-time uv sync writes the whole venv to RAM. Where scratch space is needed (build outputs, TMPDIR, caches), HANGAR_K8S_SCRATCH=emptydir mounts a size-limited emptyDir at /scratch and sets TMPDIR, UV_CACHE_DIR, npm_config_cache, CARGO_TARGET_DIR to it. Whether the hardening admission policy allows emptyDir, and whether it bypasses the memory-backed root under gVisor, is exactly what bench_gvisor_io.py must show (§10 Q5).

gVisor tuning¶

• On GKE the sandbox runtime flags are managed by the platform. I found no documented way to pass runsc flags there, so overlay medium and directfs are not tunable (§10 Q5). The lever is what hangar writes: memory limit that includes expected writes, scratch emptyDir, baked dependencies.
• On the Docker provider with runsc, the knobs are daemon-level: runsc's --overlay2=root:self (file-backed upper layer instead of memory) and --directfs. Flag names and defaults are from my memory of gVisor docs and depend on the installed runsc version. Verify with runsc --help on the host before use. deploy/terraform/single-vm is where this would be set. Test with bench_gvisor_io.py: same script, two settings.
• Do not turn on features that weaken isolation for speed without a recorded measurement and a note in docs/security/README.md.

CPU and memory right-sizing¶

• Sampling: 0001's in-sandbox supervisor already loops every 10 s. This spec asks it to also write <run>.res with the peak memory and CPU seconds of the run's process group, and disk used under the writable layer. Source order: cgroup files if present, else summed RSS from /proc, else ps. gVisor's cgroup and /proc coverage is unknown (0001 OQ7; §10 Q6). The watcher reads the file in its normal probe, so there is no extra exec.
• Store: Run.summary.resources, plus a rolled-up env_profile(env, kind) of p50/p95/p99 peak memory, CPU p95 and run time.
• Recommend: requests.cpu = ceil(cpu_p95), memory = p99 peak × 1.3, never below the GKE admission rule of 1 GiB per vCPU (docs/gke.md), with limits == requests as today (_pod_template). Shown by hangar env profile <name> and on the dashboard.
• Apply: HANGAR_RIGHTSIZE=off|recommend|apply. apply only raises memory automatically (after an OOM or p99 over 85% of the limit); it lowers resources only on an explicit command, since a lower limit risks OOM in the middle of a run. A resource change rebuilds the SandboxTemplate and recreates the warm pool.
• Per-sub-task override: task_create accepts resources per node (Resources, as create_sandbox does) for the occasional heavy one.
• kind hints bring defaults: docs and test nodes run on smaller sandboxes only if an env defines them (a second small env). Not automatic in v1.

Claim path¶

create_sandboxes is parallel on the MCP side, but 2 sandboxes took 3.1 s. Hypotheses to test in bench_claim.py (none verified): (a) the warm-pool controller binds claims one at a time; (b) Fleet._wait_running polls on a fixed interval; (c) _apply retries on 409 with sleep(0.2 * attempt) when concurrent creates race on the same template; (d) each claim waits for a refill (_refill) before the next claim sees a ready pod. M1 fixes the cause the benchmark finds. If it is the controller, the answer is pool size, not code.

5.9 Assumed interface from spec 0005¶

Kept minimal. If 0005 names things differently, only the adapter changes.

class DecisionService(Protocol):
    async def classify(self, *, task_id: str, subtask_id: str | None, run_id: str | None,
                       question: str, options: list[Option], context: dict,
                       untrusted: dict) -> Decision: ...

class Decision(BaseModel):
    id: str                       # decision_id
    choice: str                   # must be one of options[].id (hangar validates and rejects otherwise)
    confidence: float             # 0..1
    rationale: str                # stored, shown to humans, never executed
    model: str; latency_ms: int; cost_usd: float | None

class TraceStore(Protocol):
    def emit(self, trace_id: str, kind: str, data: dict, *, parent: str | None = None) -> str: ...
    def record_outcome(self, decision_id: str, outcome: dict) -> None: ...   # e.g. {"accepted": true, "later_failed": false}

Rules on hangar's side:

• question and options are built by code. untrusted carries agent excerpts and is marked as data. context holds facts only (counters, plan titles, scopes).
• A Decision with choice not in options, or confidence under HANGAR_DECISION_MIN_CONFIDENCE (default 0.7), opens an escalation instead. No classify available opens an escalation, with the default option pre-selected for one-click approval.
• record_outcome is called when the run ends, so 0005 can measure whether decisions were good. That data is what lets autonomy widen (§7).
• Events emitted to the trace store: task.created, plan.linted, subtask.dispatched, signal, action, decision, escalation.opened/resolved, checkpoint, budget. trace_id = task_id, so one trace covers planner run, sub-task runs and recovery.
• hangar's own task_events do not depend on 0005. If the trace store is down, emit is best-effort and the supervisor continues.

5.10 Provider differences¶

Everything the supervisor does goes through Provider.exec, read_file, write_file and 0001's run tools. The only new provider methods are explain_loss and the replicas argument of sync_warm_pool. Both have defaults, so the E2B provider needs no change in M-early.

5.11 Changes to other specs¶

State these explicitly, since the README says its decisions win:

5.12 Dashboard UX (ui/)¶

One combined dashboard design is already planned for phase 6 (0001 M8, 0002 M9). These are the additions.

• Tasks page. Table: goal, state, owner, progress (3/5), budget meters, open escalations. Filters by state and owner.
• Task detail:

 task-5b0c91de · running · priority 5 · autonomy restart · owner sean        [Pause] [Cancel] [Land all…]
 Budget  agent-min 31/90 ▓▓▓░░░░░░  tokens 1.2M/4.0M ▓▓▓░░░░░░  wall 15/120 ▓░░░░░░░░░

   client ──needs──▶ docs
   [client  RUNNING  standard  run-3fa9 ● alive 4s  running tests · 17 turns]
   [cfg     NEEDS YOU  standard  attempt 2    ⚠ verification failed twice]
   [docs    PENDING   cheap     waiting on client]

 Supervisor
  14:02  nudged run-3fa9  loop_test: same 2 failing tests x3, no diff change  (template loop_break)
  13:51  infra loss run-77d0 (preempted?)  restored checkpoint ckpt-91 on sb-5e6f   +2m10s
 Decisions needed
  esc-a1b2c3  cfg failed verification twice.   (•) Retry on strong tier   ( ) Skip   ( ) Fail task      [Resolve]

• Agent text appears only inside a collapsed "Agent says (unverified)" block, escaped. Supervisor lines are built from facts.
• Plan review: the DAG, lint results with node highlights, per-node scope globs, tier and estimated tokens, and Approve / Edit. Overlapping scopes are shown red.
• Fleet header strip: claude-max · running 2/3 (limit adapts) · queued 4 · 7-day 91% · warm 2/2 · nodes 1 spot. Click through to the scheduler page: queue by class and owner, pacing reason per held run, warm-pool targets and forecast.
• Performance page: the §5.8 metrics over time, with the targets drawn as lines and a table of the latest benchmark runs.
• Escalations also appear in 0002's Inbox (kind = task.escalation), and go to its notifiers. human_only ones show the buttons only to users with approve.

6. Failure modes and safety¶

Trust model: everything from a sandbox is untrusted (0001 §6). The supervisor adds LLM calls on top of that, so the rule is the model picks from a list; code does the rest.

7. Rollout and migration¶

1. Benchmarks first (M0), no behavior change.
2. Feature flags, all default off until their milestone is proven live: HANGAR_TASKS=off|on, HANGAR_SCHEDULER=fifo|fair, HANGAR_WARM_AUTOSCALE=off|observe|on, HANGAR_RIGHTSIZE=off|recommend|apply, HANGAR_PREPULL=off|on, HANGAR_CHECKPOINTS=off|on.
3. Supervisor autonomy ladder. New tasks default to nudge. Run observe on real tasks first and compare what it would have done with what a human did. Widen to restart after the false-nudge rate is under 10% over 20 tasks, and to full after record_outcome data shows the classify choices were accepted. This is Jarmak's "widen one action class at a time, by measured rate".
4. Scheduler. fair mode with one owner and no tasks behaves like 0001's FIFO with class gates. Turn it on in the same release as tasks. The only visible changes are class gates (batch stops at 80%) and adaptive limits; both are listed in the changelog.
5. Existing clients. No tool changes. run_agent calls without a task are class interactive. The new Run columns are nullable. warm_pool: N still means "at least N".
6. Behavior changes (the complete list): queue ordering is no longer FIFO (§5.11); warm_pool may be raised above the spec value when HANGAR_WARM_AUTOSCALE=on; resource recommendations never change a spec silently (apply only raises memory, and logs it).
7. Docs: new docs/tasks.md (plan format, ladder, budgets), docs/performance.md (benchmarks, targets, tuning), updates to docs/gke.md (pre-pull, autoscaler profile, Spot recovery) and README.md MCP table.
8. Rollback: turn the flags off. Tables stay and are ignored. Tasks in flight finish their current runs and then stop dispatching (PAUSED/flag_off).

8. Test plan¶

8.1 Unit tests (fake provider)¶

Setup: the fake provider from tests/conftest.py (real bash and git, private tmux socket). New: an injectable Clock (0001 §8.1), the scripted harness extended with behaviors (loop, hang, die_at, fail_tests, edit_outside_scope, emit_usage), FakeClassifier (programmable choice, confidence, latency, failure), FakeTraceStore, and a FakePreemptingProvider that deletes a sandbox on cue and returns an explain_loss value.

8.2 Live acceptance on GKE¶

Setup: the 2026-10-11 cluster, one Pro token, a second API-key credential with a spend cap, hangar-testbed repo (0002), baked env with warm_pool: 2, proxy mode if 0003 has shipped. Repeat each scenario 3 times and report median and range. scripted harness where model behavior does not matter.

Pass: all scenarios on one run, repeated twice. Results are written to bench/results/ and summarized in the closing PR of each phase.

9. Milestones¶

Sizes: S ≤ 1 day, M ≈ 2–3 days, L ≈ 4–6 days. Each is one PR. "Slot" is the README build-order phase.

Notes:

• M0 and M1 are useful even if nothing else here ships.
• M2–M5 do not need tasks. They improve every existing run.
• Fold M13's checks into each phase-closing PR as the README asks, rather than one big final PR.
• M6–M8 give a complete deterministic supervisor with caller-provided plans. LLM use starts at M8 (planner) and M9 (classify).

10. Open questions¶

1. Pre-pull DaemonSet under the hardening policies. Do sandbox-core-policy / sandbox-hardening-policy apply to a DaemonSet in the sandbox namespace, and would it need the same security context? If it must live in another namespace, does image sharing with gVisor pods still hold? Not tested.
2. Why did two claims take 3.1 s? Serial controller, poll interval, 409 backoff or refill wait? M0 answers it. The 10-in-6 s target assumes a fixable cause.
3. Model flags per harness. claude --model <alias> is confirmed. I did not verify the alias-to-model mapping for Claude 5, or -m for Codex, Gemini and agy. Also: can tiers include different harnesses safely for the same prompt?
4. --max-budget-usd on subscriptions. Documented as --print only. Does it count anything with an OAuth subscription token? Until known, hangar's own guard is the control.
5. gVisor tuning on GKE. Can overlay medium or directfs be set on the add-on's gVisor? Does it allow an emptyDir volume under the hardening policies, and is it disk-backed? Does writing to it avoid the memory-backed root? bench_gvisor_io should settle it; flag names for runsc on the Docker side are from memory.
6. Resource sampling under gVisor. Is cgroup memory accounting or /proc readable enough for peak memory? 0001 OQ7 covers /proc/<pid>/stat; this needs a peak-memory source.
7. Spot reclaim signals on GKE + gVisor. What notice reaches the pod, and which pod/node event reason identifies a Spot reclaim vs an eviction vs an OOM? explain_loss is written against guesses until M4 captures real events.
8. Autoscaler profile optimize-utilization. Does it apply to this Standard cluster and the gVisor Spot pool, and does it hurt scale-up time? Measure with bench_start.
9. Image streaming and secondary boot disks on gVisor nodes. Supported or not? (Same as 0001 OQ9, now with a test plan.)
10. Checkpoint session restore. Can a Claude Code session file copied to another sandbox be resumed with --resume <id>? If not, the handoff prompt is the only path.
11. Path-scope enforcement. Should the gateway reject pushes touching paths outside scope.writes? It would turn §5.3's detection into prevention, but it needs the scope in the grant and cheap tree inspection of a pack. Not proposed here.
12. Where classify gets its credential. An API key with a spend cap held by hangar is assumed. Is the owner willing to hold one, given README decision 9?
13. E2B and Docker limits. E2B concurrent-sandbox and lifetime limits for the account, and its start latency, are unmeasured. On Docker, is NCPU gating enough on a laptop?
14. Task-level integration check. After all PRs land in order, per-PR CI proves each head, not the combination. Add an optional integration_verify on the merged result, or leave to CI on main?