Skip to content

Agent factory · Judge

Did that harness change actually help?

flightdeck answers that from the runs hangar already records: paired statistics, behavior specs and a verdict posted on the pull request where the change is reviewed.

Get started Suite overview

A terminal session on the fixture experiment: flightdeck ingests the export, compares the variants paired by task and writes a report, printing its verdict, candidate is better than control on resolved: +21.2 pp (95% CI +1.5 to +43.9, margin −5.0). It opens on the finished session and then replays the typing.
The fixture experiment in a terminal: compare the arms, write the report, read the verdict. Watch it end as a pull-request comment (18 s): Night, Day.

The verdict

One of five words: "better", "non-inferior", "worse", "No decision" or "No verdict". It says how the candidate arm did against the control arm on one metric, with the paired difference and its 95% interval beside it: "candidate is better than control on resolved: +21.2 pp (95% CI +1.5 to +43.9, margin −5.0)". The rule that picks the word (decision.yaml: metric, direction, margin, minimum tasks) is committed before the first trial, so nobody moves the goalposts. Concepts has the words; Reports has the rule.

  • Is: the judge. It reads what hangar recorded and says whether a change helped.
  • Isn't: a runner, a dashboard, a leaderboard or a credential holder.
  • Sees: changes of about 15 points at 500 trials a month; smaller ones through cost, tool calls and behavior specs (what flightdeck can tell you).

See it work

The verdict, posted as one pull-request comment that is updated in place. This one comes from the synthetic fixture experiment (12 tasks × 2 variants × 3 trials):

The top of the flightdeck report as a pull-request comment: the headline is the verdict, candidate is better than control on resolved, +21.2 pp (95% CI +1.5 to +43.9, margin −5.0); under it the experiment FDK-EXP-0531 with its real ID job-expdemo, what differs between the arms, the pre-registered rule, and the Outcome table with 10 of 32 and 16 of 31 resolved, Wilson intervals and dollars per resolved task The top of the flightdeck report as a pull-request comment: the headline is the verdict, candidate is better than control on resolved, +21.2 pp (95% CI +1.5 to +43.9, margin −5.0); under it the experiment FDK-EXP-0531 with its real ID job-expdemo, what differs between the arms, the pre-registered rule, and the Outcome table with 10 of 32 and 16 of 31 resolved, Wilson intervals and dollars per resolved task
The verdict as a pull-request comment: the headline, then experiment FDK-EXP-0531 and its rule

The recording above is the run that produced it: ingest → stats compare → report. Getting started has the six commands and what each printed.

Where it sits

Where flightdeck sits · run / judge / decideRunhangarruns coding agentsin sandboxes;records manifests,traces and theexperiment index.Runs the trials itis asked for, pinned;does no analysis.JudgeflightdeckingestREST or an exportdata planeParquet · DuckDBstats · checkpaired by task · specsreporta pre-registered ruleDecidethe pull requestone comment, updatedin place: verdict,pins, outcome, cost,behavior specs.A reviewer decides;the approver isnever the author.runs, traces,the indextrials to run:tasks × arms × nthe reportGovernTower: policy decisions, approvals, signed receiptspromotion approval · planned

hangar runs the trials, flightdeck judges them, the pull request decides; Tower governs risky actions

flightdeck holds no sandbox code and no credentials (ADR-0001). It reads hangar's versioned interfaces and asks hangar to run the trials it plans.

Use cases

  • Bump the CLI lock with evidence. Run the regression suite on the old lock and the new one; the canary PR gets one comment with the pre-registered verdict and the cost per resolved task of each.
  • Decide whether a workflow plugin pays for itself. A/B the plugin on and off across 30 tasks. Pairing by task means far fewer runs than an unpaired comparison, and the rule was committed before the first trial.
  • Catch a behavior regression nobody tested for. Write "never push to main" as a behavior spec. Nightly, check run scores every turn and opens an issue when the violation rate jumps, naming the harness digest that changed.
  • Getting started: six commands on the fixture, what they showed, then a real hangar.
  • Concepts: the vocabulary, and what is shipped.
  • Reports: the pre-registered rule and the report's anatomy.