Run capture: the outbox (hangar.run/v1)¶

hangar writes every finished agent run, once, as immutable objects to a bucket. The sister tool (flightdeck) reads them into its own tables (flightdeck.parquet/v1) without asking hangar anything, so it works while the sandbox pools are off (01:00-10:00 ET) and after hangar's own retention has let a run go. hangar pushes; the sister tool owns the tables and the compactor that builds them. This page is the contract between the two. It is the fourth of hangar's interfaces, next to the manifest, the trace export and experiments (spec 0014).

Versioned like the others: within v1 hangar only adds optional fields, objects and event kinds; anything removed or renamed is v2, under a new top-level prefix.

Turning it on¶

On GKE, terraform creates the bucket (terraform output run_bucket, gke.md). A local directory gets the same layout and the same bytes a GCS reader gets (objects are stored gzipped in GCS with Content-Encoding: gzip, which GCS decompresses on download unless the reader asks for the raw bytes).

Layout¶

v1/runs/{date}/{session_id}.{run_id}/run.json          hangar.run/v1, written last
                                     manifest.json     hangar.manifest/v1, as captured when the turn started
                                     trace.otlp.json   hangar.trace/v1 of this run only (the turn and its children)
                                     tests.json        hangar.tests/v1: the turn's test runs (when tests ran)
                                     grade.json        experiment trials, once the grader has run
                                     canary.json       canary shadows (#184): the canary and whether it was obeyed
                                     patch.diff        the turn's diff against the commit it started on
v1/jobs/{date}/{job_id}/events/{ts_ms}-{kind}.json     hangar.job_event/v1
v1/tapes/{date}/{session_id}.{run_id}/tape/{seq}.json  hangar.tape/v1 entries (#177), expired after 30 days
                                          tape/index.json
v1/determinism/{date}/{repo}.json                      hangar.determinism/v1: the nightly determinism report (#179)

• {date} is the UTC date (YYYY-MM-DD) the run finished, or the event happened.
• The run key is {session_id}.{run_id}: a run-[0-9a-f]{8} id alone isn't unique across months. run_id is hangar.run.id in the trace and subject.run_id in the manifest; flightdeck's runs.run_id.
• Complete runs. run.json is uploaded only after every other object of its run is, so its presence is the completion marker: a run directory without it is still arriving. Its objects lists what is there, withheld what never will be (below), and expects what arrives later: grade.json for a trial of a graded experiment, written when the grader finishes, possibly after run.json.
• Ordering. Nothing may rely on arrival order. Runs and events carry their own timestamps and ids; a late object for an old date lands under that old date. A compactor looks back a few days (flightdeck: 3).

What is captured, and when¶

A job turn cut off by the power-down or a restart (it resumes later) is captured when it really ends. Capturing the same run again uploads nothing: every object is write-once.

run.json (hangar.run/v1)¶

Schema: run.schema.json (validated in CI against real runs). The fields a reader most needs:

PR review data is not in run.json. Flightdeck reads the share of agent-authored PRs from GitHub into its own prs table (Sean-Koval/flightdeck#6). The lifecycle facts hangar knows are job events: the PR ready for review, its first human review, reverts and abandon reasons (#168, metrics.md).

The other objects¶

• manifest.json: the turn's stored hangar.manifest/v1, as captured when the turn started (its digests included), redacted again.
• trace.otlp.json: hangar.trace/v1 for this run only, as GET /api/sessions/{name}/trace?turn=N returns it: the turn span with its tool calls, subagents, approvals and events.
• tests.json (hangar.tests/v1, $defs.tests in the schema): runs are the turn's TestRun records (jobs/test-results.md) with cases, every case's outcome by test id (not only the failures); events are their test.run and test.case span events in OTLP/JSON span-event form (name, timeUnixNano, attributes as OTLP key-values), exactly what GET /api/jobs/{id}/tests?format=otel describes, so a trace flattener reads them unchanged. Each run has counted (traces.md: the field predicates use) and agent_run_id; a session turn's runs are those recorded with its run id (#164).
• grade.json (hangar.grade/v1): the trial's grade as the experiment index has it (command, exit_code, applied, seconds, trust, hidden_tests, data from $HANGAR_GRADE_OUT, ...) with experiment_id, trial_key, job_id, session_id and run_id. test_run names the grader's grader TestRun and tests holds it in tests.json's shape (runs with every case, events), trusted only for trust: isolated (#164).
• canary.json (hangar.canary/v1, #184): a canary shadow's injection canary (security/taint.md): token, placement, template_id, expected_signals, signals (each signal and whether hangar saw it), obeyed, experiment_id, shadow_of, harness_digest, job_id, session_id, run_id. Staged as the shadow's turn ends, before its run.json, and not listed in objects.
• patch.diff: the run's work against the commit the turn started on (start_sha): commits, uncommitted changes and untracked files (as get_session_patch computes it), read from the sandbox as the turn ends. Changes left uncommitted by an earlier turn appear again. Absent when the sandbox was gone.

Tapes¶

Runs in a model_gateway: record environment (model-gateway.md) also leave a tape of their model calls, staged when the turn ends under v1/tapes/ (a lifecycle rule can only match a name's prefix, so tapes live apart from their run and the bucket deletes them after run_tape_days, 30). {date} is the day the turn finished; the run key is the run's.

• tape/{seq:05d}.json, one per model call, in order: seq (from 1), req_hash, req_delta, resp, resp_hash, prev_hash, at (epoch seconds) and usage (the provider's, from the final events). req_delta is {"full": request} or {"base": seq, "append": [...], "set": {...}, "unset": [...], "outer": {...}}: the messages (or input items) the request adds to entry base. A request is {method, path, headers, body}; resp is {status, headers, sse: [{event, data}]} for a stream or {status, headers, body}, plus truncated: true when it wasn't kept whole. Both are normalized and redacted before hashing.
• tape/index.json: schema (hangar.tape/v1), run, session_id, run_id, sandbox_id, provider, model, harness, manifest (the turn's manifest digest), harness_digest, entries, genesis, head, started_at, finished_at, untrusted and untrusted_reasons.
• Checking one: hashes are SHA-256 over canonical JSON (sorted keys, , and : separators, UTF-8). Rebuild each request from its delta, check req_hash and resp_hash, check that prev_hash is the previous entry's hash (genesis for the first, sha256("hangar.tape/v1\n" + run)), and that the last entry's hash is head (hangar.modelgw.validate does it).

Job events (hangar.job_event/v1)¶

Facts about a job that arrive after its runs, each its own immutable object, never a rewrite of run.json. Schema: $defs.job_event. Every event has kind, ts, job_id, job_source, source, repo, person, environment, session_id, run_ids, pr, untrusted and a data object:

hangar scans jobs changed in the last minute for new facts and writes each once.

Guarantees¶

• Write-once. Every upload is a create with ifGenerationMatch=0; a 412 means the object is already there and counts as done. hangar's service account has roles/storage.objectCreator on the bucket and nothing else there, and an IAM deny policy takes objects.delete and objects.update away from it (a backstop): it can't delete or overwrite a run. Readers get roles/storage.objectViewer.
• Redacted. Every object passes the broker's redaction (stored credentials, verbatim and in common encodings) and secretscan's masking (token formats: GitHub, cloud keys, ...) before it is staged, on top of the redaction the manifest and trace already apply. The redacted text is then scanned again; an object that still holds a secret is marked failed, listed in run.json's withheld, and never uploaded. A test seeds a stored secret, its base64 form and a token-shaped string in the agent's output and work, and checks no object holds any of them.
• Never on a turn's path. A turn's end only starts a task. The objects are built, redacted, gzipped and staged on the data volume (<data>/outbox/) with one queue record each (OutboxItem: key, kind, run, state: pending|done|failed, attempts, next_at, error); a drain loop uploads them. A test holds every upload for a minute and checks the job finishes as fast as without the outbox.
• Retried until it lands. A failed upload is retried with exponential backoff, 1 minute doubling to 6 hours, for as long as it stays pending. The queue and the staged payloads survive restarts. The control plane stays up while the pools are off, and GCS doesn't depend on the cluster, so the queue drains overnight too.

Operating it¶

• hangar outbox status: pending, done and withheld objects, the oldest pending one, the last error, and retention held for the export (GET /api/admin/outbox). Admins.
• hangar outbox retry: everything pending, now, instead of waiting out the backoff (after an outage).
• hangar outbox backfill --since 7d: runs finished in the window with nothing queued, captured again from what hangar still has on disk (manifests, event logs, test runs; the patch only while the sandbox is up).
• /metrics: hangar_outbox_pending. hangar doctor warns past 1,000 pending objects or 24 hours of age, lists withheld objects, and reports the bytes deleted sessions hold only because their export is pending.

Retention and the export (#189)¶

hangar's retention never deletes what the outbox hasn't exported yet:

• A session's event log and manifests go with the session record today. Now the record goes as before, but its files are deleted only once every run of it is exported; until then they wait in <data>/pending-delete/<session>/ and a pass every 10 minutes deletes them once the export is done, or after HANGAR_EXPORT_GRACE_DAYS (30) with an audit event retention.unexported_deleted (a GCS outage longer than that loses those runs, and the event says so).
• Test runs and cases of a job that failed, was cancelled (abandoned), had its PR closed unmerged, or carries a failure label (#169, job.failure_label) outlive testresults' windows (cases 3 days, runs 14) until the job's runs are exported.
• Best-of-N attempt sessions are archived, never deleted, so their event logs stay.
• With the outbox off, retention is exactly as before.

A run is exported when nothing of it is being built and every object of it is uploaded. A run with nothing queued (it ran before the outbox was turned on, or jobs_only leaves its session out) counts as exported: nothing will ever upload it.

Deletion requests. Objects are immutable. To delete a run (personal data): delete its objects with an admin credential (gcloud storage rm -r gs://<bucket>/v1/runs/<date>/<session_id>.<run_id>/, and its job's event objects), then have the sister tool recompact the affected dates (flightdeck: flightdeck compact --date D --rebuild). Locally, hangar session delete <session> --force-now deletes its event log and manifests without waiting for the export (audited session.delete_now and retention.unexported_deleted).

Out of scope¶

The GCS → Parquet compactor (flightdeck, Sean-Koval/flightdeck#5); replaying tapes from the bucket (replays read the control plane's local copy, model-gateway.md); exporting anywhere other than GCS or a local directory.