Skip to content

Agent factory ยท Judge

flightdeck

Judges harness changes with paired statistics and behavior specs over every run.

flightdeck's report as GitHub shows the comment on the harness pull request: the verdict, candidate is better than control on resolved (+21.2 pp), experiment FDK-EXP-0531, the rule, the outcome table and the paired difference flightdeck's report as GitHub shows the comment on the harness pull request: the verdict, candidate is better than control on resolved (+21.2 pp), experiment FDK-EXP-0531, the rule, the outcome table and the paired difference
GitHub's comment on the harness PR, written by flightdeck: candidate beats control on resolved, with the interval and the outcome of both variants.
Owns The Parquet data plane, paired statistics, behavior specs, trial sampling, experiment reports.
Does not own Sandboxes, credentials or provider code. It reads hangar's exports and the outbox, never its database.
Entry points The flightdeck CLI; reports posted as GitHub comments; a nightly check run.
Used by The harness owner and lead deciding promotions, engineers reading reports on harness PRs, nightly CI.
Status The data plane, statistics, reports and the spec language are shipped; the nightly runner and one real decision are next. See P3 on the roadmap.
The first frame of flightdeck's clip: flightdeck stats compare in a terminal, with the two arms' resolved rates and the paired difference The first frame of flightdeck's clip: flightdeck stats compare in a terminal, with the two arms' resolved rates and the paired difference
Play the clip (18 s): stats compare, then report and its verdict, then the same report as the pull-request comment.

Docs site Repository