Data plane¶

flightdeck reads hangar's runs from Parquet files, never from a live hangar, so reports work while hangar's sandbox pools are off. The contract is flightdeck.parquet/v1.

hangar writes each finished run once to a GCS outbox (hangar.run/v1, hangar docs/api/outbox.md). The nightly compactor turns it into these tables, and DuckDB queries them (ADR-0001 point 3). For one experiment's results right away, the same flatteners run over hangar experiment export and REST exports (ingest).

Code: src/flightdeck/dataplane/. schema.py holds the tables, flatten.py the flatteners, duck.py the views, compact.py the compactor and github.py the prs table.

Layout and versioning¶

<root>/<table>/dt=YYYY-MM-DD/part-*.parquet      zstd; dt is in the path (hive partitioning), not the file

<root> is FLIGHTDECK_DATA_URI: a local directory or a private gs:// bucket (ADR-0002). Each file's schema metadata carries flightdeck.schema = flightdeck.parquet/v1 and its table name.

Within v1, columns are only added; removing or renaming one is v2. Rows are wide and denormalized: resource attributes (job, session, environment, person) are copied onto every span row.

Every row has run_id, source and dt. source is job, session, experiment, shadow, or local for the local backend.

Tables¶

What the outbox adds¶

From the outbox, runs also carries run.json's outcome_class, infra_reason, job_source, job_turn, job_state, steers (steers.n) and questions. Two flags come with them: untrusted (the job's task text came from outside the trusted list) and tainted (the turn read untrusted content). Promotion evidence leaves out runs with either flag set.

Tapes stay objects under hangar's v1/tapes/; they are not compacted.

What flightdeck writes itself¶

On collecting its trials, flightdeck stamps runs.task_digest, the task directory's digest (Tasks). It also stamps runs.variant_digest: the variant as planned, its hangar overlay and environment. They key the trial cache.

trial_reuse is flightdeck's too: one row per trial the cache reused for a run set. run_id is the reused trial's, experiment_id the run set's, with age_days.

judge_labels holds one judge label per run, with the rubric version and digest, the judge model, the rationale's digest and the rubric's calibration status (Judges).

spec_results holds one row per spec or ADR check and unit (a turn, job or session id in run_id). Its columns: spec_id, spec_source (spec or adr:<repo>/<adr>), scope, result (pass, fail, not_applicable, unknown), repo, harness_digest and knowledge_snapshot (Behavior specs).

Nullable until hangar emits them¶

harness_digest, environment_digest, knowledge_snapshot and grader_digest are nullable now and fill with no schema change once hangar emits them.

runs.outcome_class is agent, infra_error, cancelled or human, and null for records from before hangar#166. It and infra_reason come from the trial, or from the turn span's hangar.outcome.class and hangar.outcome.infra_reason. excluded and attempt come from the trial.

Decoding¶

OTLP/JSON per opentelemetry-proto 1.11.1: intValue is a decimal string, lists are arrayValue.values, kvlistValue becomes a dict. Unknown attributes are kept in attrs, never an error, since hangar adds fields within v1. A different hangar.trace.schema, manifest or experiment schema is refused.

hangar redacts every export. The flatteners also mask anything that still looks like a credential (private keys, GitHub, Anthropic/OpenAI, AWS and Slack tokens) as ***. They clip event text to 2,000 characters.

Ingest¶

flightdeck ingest pulls from hangar directly, for an author who wants one experiment's results now. That covers the wait for the night's compaction, and a hangar whose outbox is off. Code: src/flightdeck/dataplane/ingest.py.

flightdeck ingest export DIR                                  # files from `hangar experiment export <id> -o DIR`
flightdeck ingest experiment <id> --hangar URL                # REST; bearer token from HANGAR_TOKEN
flightdeck ingest job <id> --hangar URL                       # GET /api/jobs/{id}/trace + /manifest?turn=N
flightdeck ingest session <name> [--turn N] --hangar URL      # GET /api/sessions/{name}/trace + /manifest?turn=N

All write to --out (default ~/.flightdeck/data, or FLIGHTDECK_DATA_URI), with source = experiment, job or session.

REST and export are one path¶

REST ingest streams the index and each trial's links.manifest and links.trace to <out>/raw/<id>/, in hangar experiment export's layout (index.json, trials/<task>__<variant>__<n>[~<attempt>]/*.json). Then it reads that directory like an export. Both produce identical tables, and nothing large is held in memory.

Experiments¶

One runs row per trial, from the index. A trial's spans, events and test results take the trial's run_id (<experiment>/<task>/<variant>/<n>), and its run row is truncated when its trace was.

A trial hangar retried after an infrastructure error (hangar#166) keeps its row, excluded. The retry is a second row under the same trial_key, with run_id <experiment>/<task>/<variant>/<n>~<attempt>.

Pins that break hangar's model rule are warned about, as experiment status reports them. A trial still pending or running has outcome null, and statistics leave it out.

Drifted trials (trials[].drift) are kept, with drift set; flightdeck stats compare excludes them unless --include-drift. hangar's #127 anti-forgery fields land in runs.task_invalid, runs.contaminated and runs.grade_trust.

Idempotent¶

Each unit (an experiment, a job, a session) owns one Parquet part per table and day, part-<unit>.parquet. Re-ingesting replaces the unit's rows by run_id, so it adds no duplicates. A trial that finished on a later day moves partitions instead of staying behind.

Validation¶

The index and manifests are checked against hangar's experiment.schema.json and manifest.schema.json. Those are vendored in src/flightdeck/dataplane/hangar_schemas/ with the hangar commit they came from (SOURCE.json); python scripts/sync_hangar_schemas.py --hangar ../hangar refreshes them.

A mismatch warns and the ingest continues. The flatteners keep what they understand, and the raw JSON stays in the export or raw/ directory.

Tokens¶

hangar scopes every record to its owner (spec 0014 §1). A member token sees its own experiments, jobs and sessions; an admin token sees everyone's; a jobs token sees what it submitted. A 404 means missing or not yours, and is reported that way.

A report on your own experiment needs only your member token. A team-wide report needs an admin token, kept in the operator's environment.

Compactor¶

flightdeck compact turns hangar's outbox into the tables above. Code: src/flightdeck/dataplane/compact.py; deployment: deploy/compactor/.

It runs as a Cloud Run job at 02:30 ET, inside the pools-off window. The control plane and GCS stay up; the cluster doesn't need to.

flightdeck compact --input gs://<runs-bucket> --out gs://<team>-flightdeck/parquet/v1     # today and 3 days back
flightdeck compact --date 2026-11-03 --rebuild                                            # rewrite those dates
flightdeck compact --input file:///tmp/outbox --out /tmp/plane                            # a local outbox tree

--input (FLIGHTDECK_RUNS_URI) and --out (FLIGHTDECK_DATA_URI) take gs://, file://, a plain path or any fsspec URL. gs:// needs the gcs extra.

Complete runs only¶

It reads v1/runs/{date}/{session_id}.{run_id}/ directories that have a run.json. hangar uploads that file after the run's other objects, so a directory without one is still arriving and waits for a later night.

A run's rows come from run.json (the runs row, laid over the trace's turn row), trace.otlp.json (spans, events), tests.json (test_cases), manifest.json and grade.json. Job events come from v1/jobs/{date}/. Everything passes through the flatteners' credential masking again.

One partition per date¶

Each outbox date is one dt= partition per table, part-0.parquet, rewritten whole. It is written as part-0.parquet.tmp, then renamed, so readers never see half a file.

Rows are sorted by run_id, seq, so a rebuild of the same inputs is byte-identical. Row groups are sized to about 64 MB.

Late arrivals¶

<out>/_compactor/{date}.json records the date's input objects: path, size, GCS generation and, for what was read, sha256. Each night looks back --lookback days (3) and recompacts a date only when its listing changed. That is a late run.json, a grade.json that followed its run, or a deleted run. One prefix listing per date, no per-object metadata calls.

Identity¶

The job's service account has storage.objectViewer on the outbox bucket and storage.objectAdmin on the flightdeck bucket, nothing else. No hangar endpoint or token; a test asserts the module imports no HTTP client.

Runbook¶

Rebuild a date, after a fix to the flatteners or to pick up something older than the lookback: gcloud run jobs execute flightdeck-compactor --region <r> --args=--date,2026-11-03,--lookback,0,--rebuild. Or run flightdeck compact --date 2026-11-03 --lookback 0 --rebuild locally with read access to both buckets.

Delete a run (personal data): delete its objects from the outbox with an admin credential (gcloud storage rm -r gs://<runs-bucket>/v1/runs/<date>/<session_id>.<run_id>/, and its job's event objects). Where objects can't be deleted yet, list it in gs://<runs-bucket>/v1/_deletions/<request>.json as {"runs": ["<session_id>.<run_id>" or "<run_id>", ...]}. Then rebuild its date.

Within the lookback window the next night does this on its own, so a deletion is out of Parquet within 24 hours.

Deploy: sh deploy/compactor/setup.sh --project <p> --runs-bucket <hangar's run_bucket> --data-bucket <team>-flightdeck (its README; --dry-run first). hangar's GKE terraform makes the outbox bucket (output run_bucket), and its deploy/gke/up.sh turns the outbox on.

Check a night: the job's log prints rows per table per date, or unchanged. flightdeck query --data gs://<team>-flightdeck/parquet/v1 "select dt, count(*) from runs group by dt" reads what it wrote.

flightdeck.weekly/v1¶

flightdeck metrics weekly writes <reports>/weekly/<week>.json, <week>.md and latest.json, with FLIGHTDECK_REPORTS_URI = gs://<team>-flightdeck/reports. hangar's standup line and digest read latest.json and never recompute.

JSON Schema: src/flightdeck/metrics/weekly.schema.json, additive within v1.

Querying¶

flightdeck query "select variant, count(*) from runs join grades using (run_id) group by 1" --data ./plane

duck.connect(uri) registers one view per table (read_parquet(..., hive_partitioning=true, union_by_name=true)). A table without files is an empty view of its schema. Timestamps are shown in UTC.

For gs://, install the gcs extra (uv sync --extra gcs, or pip install 'flightdeck[gcs]'). DuckDB then reads through gcsfs with your Google application-default credentials (gcloud auth application-default login). Without it, DuckDB's httpfs is loaded and the caller supplies a DuckDB GCS secret (HMAC keys). flightdeck holds no credentials either way.

uv sync --extra gcs
flightdeck query "select count(*) from runs" --data gs://<private-bucket>/plane

Truncated exports (hangar.export.truncated, hangar's 20,000-item cap) set runs.truncated. Process metrics (tool calls, cost, wall time) leave those trials out by default and say so; outcome metrics keep them.

Size¶

About 1.5 MiB of OTLP per run gives roughly 1.5 GiB a month as zstd Parquet at the team's volume. DuckDB over Parquet is enough until about 1 TB.

See also¶

• Contracts: the column tables of flightdeck.parquet/v1, generated from the code.
• Running experiments: how collect fills the plane.
• Behavior specs: the SQL evaluator over these tables.
• Weekly metrics: the scorecard computed from them.