Skip to content

Roadmap

What the suite builds next, in four phases and a backlog, across hangar, flightdeck, ai-dlc, Tower, Spaceport and this repo. Phases are GitHub milestones of the same name in every repo; themes in roadmap.toml group the issues that deliver one outcome. Open a theme's row to see its issues.

Each day the site build reads those milestones and issues from GitHub and recounts them. Day-to-day work is tracked in Jira (KAN), which this page does not read. To move an issue to another theme, edit roadmap.toml.

Four decisions that shape the plan
  • No big harness spec. A harness is native files plus a small lock; no new format or compiler in the next six weeks.
  • flightdeck waits for evidence. No flightdeck code before week 5 and twenty labelled failures; v0 must change one real decision before it gets a UI.
  • Shadows wait for reviewers. Shadow runs stay off until review wait p50 is under four hours.
  • Spaceport is frozen. Cleanup only, until its MCP tools ask Tower's decision point.
  • Choices that bind more than one repo are recorded on Decisions.

Progress

Phase Done when hangar flightdeck ai-dlc Spaceport agent-factory All
P1 · Safe release
weeks 1-2		 Security fixes, credential policy and grader isolation are in; CI is green; two engineers use hangar on one repo. 20 of 21 – – 1 of 1 – 21 of 22
P2 · Adopt and capture
weeks 3-4		 All six engineers on three repos through the Jira door and the Slack inbox; every run lands in the outbox and Parquet; weekly metrics; a team harness frozen from a good run. 32 of 43 6 of 6 19 of 20 3 of 3 3 of 3 63 of 75
P3 · flightdeck v0
weeks 5-6		 flightdeck's stats and reports run over real runs and change one real decision. 5 of 7 11 of 18 1 of 2 – – 17 of 27
P4 · Frontier
week 7 on		 Record-mode gateway, replay-as-fork, pin-set trials, shadows, governed tool calls. 10 of 14 2 of 10 – – – 12 of 24
Backlog
unscheduled		 1 of 6 0 of 4 1 of 5 1 of 2 0 of 1 3 of 18

Tower: private or not found, not shown.

The 6 epics that track this work
  • hangar#119 Epic (part A): safe release, harness definition, MCP split and cleanup for the Run/Judge/Govern plan
  • hangar#153 Epic (part B): team adoption, inbox, capture, governance integration and frontier runtime
  • flightdeck#1 Epic: flightdeck v0, the Judge plane over hangar's interfaces
  • ai-dlc#189 Epic: Refocus ai-dlc as workflow plugin author, machine provisioner and CI-gated work lifecycle
  • ✓ spaceport#10 Epic: freeze Spaceport; clean up naming and licensing; later, govern its MCP tools through the workspace
  • workspace#7 (private repository)

Built 2026-11-10 14:58 UTC from GitHub issue state. Counts are closed of filed issues per milestone; epics and pull requests are not counted.

P1 · Safe release

weeks 1-2 · 21 of 22 closed

Security and credentials10 of 10 closed
  • ✓ hangar#121 Security: scope approve_job/reject_job and refuse self-approval on every approval path
  • ✓ hangar#122 Security: REST/MCP scope-parity test mapping every tool and route to its required scope and owner rule
  • ✓ hangar#123 Security: intake posture — intake-triggered jobs require the git gateway and an egress allowlist floor
  • ✓ hangar#124 Security: sudo off by default; no-new-privileges for job sandboxes on Docker
  • ✓ hangar#125 Security: refuse subscription logins (oauth_token/auth_file) for jobs, triggers, experiments and replays; warn in interactive sessions
  • ✓ hangar#126 Security: sealed sandbox profile for graders (egress none, no credentials, fetch-only) and hidden tests run from a pristine copy
  • ✓ hangar#127 Security: grade anti-forgery — hangar-owned grade output, base-must-fail sanity check, canary strings in sealed tasks
  • ✓ hangar#128 Security: harden the Jira Automation intake path (recent-transition check, token rotation, alerting, prefer signed webhook)
  • ✓ hangar#129 Security: taint tracking — record when a turn has read untrusted content
  • ✓ hangar#130 Security: taint gates — after an untrusted read, push, PR open, landing and egress approvals need a different human
Harness identity and readiness3 of 3 closed
  • ✓ hangar#131 Harness identity: add harness_digest and environment_digest to hangar.manifest/v1 (additive) with schema docs
  • ✓ hangar#132 Harness merge rules: component lists replace on extends; add skills_remove; reconcile with experiment Variant
  • ✓ hangar#155 hangar repo check <repo>: one readiness report for landing, gateway, CI and doors
CI green5 of 5 closed
  • ✓ hangar#120 CI: deflake the background-process MCP tests behind #118 and gate P1 on a green main streak
  • ✓ hangar#205 CI: test_opstatus nightly-slot test fails between 23:30 and 00:30 New York time
  • ✓ hangar#207 CI: test_envd_e2b_sdk wrong-token test flakes with a broken pipe
  • ✓ hangar#245 Flaky: test_steer harness-exit steer test
  • ✓ hangar#250 Flaky: test_streams close-on-destroy test
  • Also in private repositories: workspace#27
Release and first repo2 of 3 closed
  • ✓ hangar#151 Docs: P1 safe-release upgrade guide, README security posture, and CHANGELOG for H-03…H-14
  • ✓ hangar#274 Tag the safe release and adopt release tags
  • hangar#154 Runbook: onboard 2 engineers on 1 repo, then 6 engineers on 3 repos
Framing cleanup1 of 1 closed
  • ✓ spaceport#11 Freeze statement in README and ROADMAP; decide what happens to draft PR #8
  • Also in private repositories: workspace#8, workspace#9, workspace#10, workspace#11, workspace#12

P2 · Adopt and capture

weeks 3-4 · 63 of 75 closed

Adoption0 of 2 closed
  • hangar#275 Onboard repos 2 and 3 and all six engineers
  • hangar#276 Friday harness review: name a facilitator, calendar it, label the first 20 failures
Team harness11 of 13 closed
  • ✓ hangar#133 harness.lock format + hangar harness freeze <job|session> --turn N (native files + lock from a captured manifest)
  • ✓ hangar#134 hangar harness thaw (claim from a lock) and drift detection on turn 1
  • ✓ hangar#136 Claude Code prefix layer: plugin seed dir (CLAUDE_CODE_PLUGIN_SEED_DIR) mounted read-only; plugin autoUpdate forced off in sandboxes
  • hangar#138 Team harness repo: propose its home and bootstrap team-default frozen from a good run
  • hangar#139 Consume ai-dlc's workflow plugin from the team harness repo as an optional, A/B-able component
  • ✓ hangar#137 Harness-Digest: PR trailer and review-packet harness field
  • ✓ hangar#221 Harness lock: plugin components (plugin:<name>@<marketplace>) and thaw mounts the locked files read-only
  • ✓ hangar#228 Harness thaw: resolve history when HANGAR_HARNESS_REPO is a subdirectory of a repo
  • ✓ hangar#244 Trust Codex plugin hooks in sandboxes
  • ✓ ai-dlc#191 Package ai-dlc's workflow guidance, hooks and team skills as one Claude Code plugin with a content digest
  • ✓ ai-dlc#192 Make the workflow-plugin hooks safe without the ai-dlc CLI (sandboxes) and harmless under hangar
  • ✓ ai-dlc#193 Codex equivalent of the workflow plugin and publication into the team harness repo for A/B
  • ✓ agent-factory#4 Example team harness repo: a clean, public build around a typical team stack
Review throughput3 of 3 closed
  • ✓ hangar#159 Review-aware admission: hold new jobs as awaiting_capacity past N open agent PRs per reviewer
  • ✓ hangar#160 Self-approval rule: the requester may be sole approver only for low-risk packets with an independent AI review
  • ✓ hangar#161 Independent AI first-pass review on every agent PR (a different harness from the author)
Doors and inbox4 of 4 closed
  • ✓ hangar#156 Jira "Ready for agent" as the default door, GitHub label/Projects parity, and an agent ticket template
  • ✓ hangar#157 Inbox reaches people: native links, stable item ids, and an immediate Slack DM for questions and approvals
  • ✓ hangar#158 Per-person Slack DM digest at 10:15 and 16:00
  • ✓ hangar#162 Steer classification (correction / redirect / new_info) in the Slack and Jira reply paths, recorded as events
Capture and metrics14 of 16 closed
  • ✓ flightdeck#2 Bootstrap: uv package, src layout, CI (lint, test, loc-budget), Apache-2.0 license
  • ✓ flightdeck#3 README with the hangar boundary, CONTRIBUTING, ADR-0001 (boundary) and ADR-0002 (public code, private data)
  • ✓ flightdeck#4 Data plane contract: flightdeck.parquet/v1 tables (runs, spans, events, manifests, grades, test_cases) and flatteners
  • ✓ flightdeck#5 GCS → Parquet compactor on Cloud Run (nightly; runs while the cluster is down)
  • ✓ flightdeck#6 Weekly team metrics: one SQL definition over Parquet, rendered for Slack/standup (shared with hangar, not duplicated)
  • ✓ hangar#163 GCS outbox: write-once, redacted, retried per turn, with a source field (hangar.run/v1)
  • ✓ hangar#164 Trusted test.run events everywhere: grader runs, session exports, and a counted rule
  • ✓ hangar#165 Per-edit git tree hashes in the event log and trace
  • ✓ hangar#166 infra_error outcome class: auto-retried, excluded from statistics
  • ✓ hangar#167 Refuse floating model aliases in pin sets; mark unpinned models in manifests
  • ✓ hangar#189 Retention: never delete failures or run artifacts before the outbox exported them
  • ✓ hangar#168 Weekly adoption metrics and the standup bot line, computed from hangar's run store
  • ✓ hangar#169 Friday harness review: label the first failure on failed or abandoned jobs, stop at 20
  • ✓ hangar#170 @hangar adr in a Slack thread drafts an ADR PR via the untrusted-context path (agent-drafted, reviewer ≠ requester)
  • hangar#273 Turn on the outbox and the compactor in the GKE deploy
  • hangar#278 Weekly metrics: reviewer attention per merged PR
Governance shadow1 of 5 closed
  • ✓ hangar#171 Spike: hangar ↔ governed workspace decision-point contract (client interface, fail-closed, timeouts, offline mode)
  • hangar#172 Job release and landing approvals ask the workspace decision point (shadow → enforce)
  • hangar#174 Per-sandbox workload identity for passport exchange (RFC 8693) instead of static keys on workspace /hooks
  • hangar#176 Correlation key + OTel export of execute_tool spans to the workspace /v1/traces for reconciliation (P3)
  • hangar#277 Slack digest reads Tower's inbox feed
  • Also in private repositories: workspace#13, workspace#14, workspace#15, workspace#16, workspace#17, workspace#18, workspace#19, workspace#21, workspace#28, workspace#52
Security follow-through1 of 2 closed
  • hangar#223 Next release: remove the warn switches for #123–#125 (HANGAR_INTAKE_POSTURE=warn, HANGAR_SUBSCRIPTION_LOGINS=any)
  • ✓ hangar#236 hangar up: a fresh laptop install fails doctor's intake posture check
Work lifecycle fixes2 of 3 closed
  • ai-dlc#160 work archive: refuses a change whose directory is not named after the work id
  • ✓ ai-dlc#226 work finish: tracker step needs a REST fallback when GraphQL is unavailable
  • ✓ ai-dlc#227 project init leaves the generated check failing until project setup
Show, don't tell5 of 5 closed
  • ✓ hangar#219 Front page: show what hangar does (recording, UI screenshot, diagram, use cases, quickstart)
  • ✓ flightdeck#44 Front page: show what flightdeck does (recording, report screenshot, diagram, use cases, quickstart)
  • ✓ ai-dlc#213 Front page: show what ai-dlc does (recordings, plugin in Claude Code, diagram, use cases, quickstart); fix site links that 404
  • ✓ agent-factory#1 Roadmap page on the suite site, kept current from issue state
  • ✓ agent-factory#2 Theme sync: every tool site takes Airframe from here, automatically
  • Also in private repositories: workspace#37
Cleanup and docs22 of 23 closed
  • ✓ hangar#141 Specs index: replace the stale phase plan and "this PR" links; mark 0001/0004/0005/0008 superseded or deferred
  • ✓ hangar#142 Resolve spec contradiction: decision 16 (ATIF canonical, Harbor-only graded format) vs 0014 (OTLP canonical)
  • ✓ hangar#143 Stale guidance sweep: docs/a2a.md output_schema, repo-conventions skill hand-off (RESULT.md vs final message), dead test branch
  • hangar#187 Docs pass for part B: guide, team-scale, jobs, intake/slack/triggers, observability, runbooks
  • ✓ hangar#220 Docs consolidation: inventory, merge or archive, and fix links that 404 on the site
  • ✓ hangar#191 flightdeck: Create the repo Sean-Koval/flightdeck: settings, branch protection, labels, milestones (owner actions)
  • ✓ hangar#246 CHANGELOG entries for the P2 merges
  • ✓ ai-dlc#190 Publish the ai-dlc feature freeze and in-scope list (README, AGENTS.md, product-direction, ADR 0001)
  • ✓ ai-dlc#194 Freeze and hand off the eval runner and skill evaluation to flightdeck (tag, inventory, port notes)
  • ✓ ai-dlc#196 Freeze per-runtime rendering: remove the unused render_agents(target=) parameter and dead cloud targets
  • ✓ ai-dlc#197 Drop exact client-version hook gating and align client pins with hangar's harnesses.lock.yaml
  • ✓ ai-dlc#199 Remove Linear from every default: base profile, Copier, template defaults, module catalog, example profiles, repo sandbox config
  • ✓ ai-dlc#200 Remove the Linear adapter, onboarding and conformance target; mark Plane unsupported
  • ✓ ai-dlc#201 Remove the stale root copier.yml, legacy templates/ + ai-dlc scaffold, and fix the stale playbook skill count
  • ✓ ai-dlc#202 Stop committing duplicate rendered skill trees (.agents/skills, .claude/skills) in the ai-dlc repo
  • ✓ ai-dlc#204 Prune docs/archive and stale OpenSpec changes; freeze OpenSpec growth
  • ✓ ai-dlc#205 Document the tracker boundary with hangar intake; keep adapters for the local work lifecycle only; re-scope #85
  • ✓ ai-dlc#206 Adopt the knowledge-note frontmatter schema for learnings; add provenance fields to work finish --learning
  • ✓ ai-dlc#207 ADR convention: docs/decisions/ with knowledge-note frontmatter and an optional checks: block
  • ✓ ai-dlc#230 Docs site: enable GitHub Pages (deploy has failed on every push)
  • ✓ spaceport#12 Naming cleanup: data-product-gateway / data-product-publisher / dpg identifiers vs Spaceport
  • ✓ spaceport#13 Licensing: resolve the README's open item to match the other work repos
  • ✓ spaceport#15 Docs: index the frozen state; fix stale links and claims
  • Also in private repositories: workspace#23, workspace#25, workspace#26, workspace#36

P3 · flightdeck v0

weeks 5-6 · 17 of 27 closed

v0 core6 of 6 closed
  • ✓ flightdeck#7 v0 ingest: pull experiments, jobs and sessions from hangar REST / hangar experiment export into local Parquet
  • ✓ flightdeck#8 v0 stats core: paired bootstrap over tasks, Wilson, pass^k, $ per resolved task (infra_error excluded)
  • ✓ flightdeck#9 v0 report: markdown experiment/suite report, posted as a GitHub PR or issue comment
  • ✓ flightdeck#10 Task format: Harbor-compatible task directory (task.toml, instruction, environment, tests/), loader, and the semver smoke task
  • ✓ flightdeck#11 Experiment submission client: build hangar.experiment/v1 specs from flightdeck task sets (batch wrapper), with a cost gate
  • ✓ flightdeck#14 Behavior spec language v0: predicates over hangar.trace/v1 (tool.kind, shell.ops, seq, paths; before / never / within N), trusted-test rule
One real decision0 of 3 closed
  • flightdeck#20 Change one real decision with flightdeck v0 (next CLI lock bump, or ai-dlc workflow plugin on/off)
  • flightdeck#12 Regression suite v1: mine tasks from the 20 labelled first failures (Friday harness review)
  • flightdeck#13 Sealed holdout custody: sha256 manifest, canary strings, blind writers, scored once on explicit go-ahead
Ports and boundaries10 of 13 closed
  • ✓ flightdeck#17 Port ai-dlc eval contracts (Scenario, Assertion, ArmReport) and assertion kinds into flightdeck
  • ✓ flightdeck#18 Port ai-dlc's Docker attempt runner as runner=local (fixtures and CI only)
  • ✓ flightdeck#19 Port hangar evals/semver analysis code (spans, classify, metrics) onto the data plane
  • flightdeck#23 Cleanup after the ports and v0: remove shims and duplicates, close cross-repo loops, check the loc budget
  • ✓ flightdeck#45 v0 follow-ups: collect via ingest, hangar's real infra_error, one model-alias rule, a behavior-spec section in reports, long-report attachments
  • ✓ hangar#135 Experiment Variant shrinks to {harness: name@sha256, environment} once locks exist; deprecate overlay fields
  • ✓ hangar#144 Rename src/hangar/supervisor.py (in-sandbox service supervisor) to end the collision with the 0001/0004 'supervisor'
  • ✓ hangar#145 Move evals/semver (spans.py, classify.py, metrics.py + the semver task) to flightdeck; decide what stays
  • hangar#146 Canary boundary: keep contract/protocol/provenance in hangar; gate CLI lock bumps on flightdeck's behavior suite
  • ✓ hangar#147 Audit analysis code against spec 0014: what stays in experiments.py / testresults.py / modelpolicy.py, and a guard against new analysis
  • ✓ hangar#222 Plugins from git or remote marketplaces in environment specs
  • ✓ ai-dlc#195 Remove the eval runner and skill evaluation from ai-dlc; stub ai-dlc eval with a pointer to flightdeck
  • ai-dlc#208 Docs sweep after the refocus: README, AGENTS.md, CLAUDE.md, architecture, product-direction, targets notes, CHANGELOG, migration notes
Judge depth1 of 4 closed
  • flightdeck#15 Spec runner over the data plane + ADR checks: nightly with violation change-point alerts
  • flightdeck#16 Mutation scoring of behavior suites: mutate the harness, check the suite catches each mutant
  • flightdeck#21 Knowledge gold-set suite: 40 questions scored on correctness and citation validity; baseline = repo search + Claude Projects
  • ✓ flightdeck#22 Docs: concepts.md (glossary), statistics.md, behavior-specs.md
  • Also in private repositories: workspace#20, workspace#24

P4 · Frontier

week 7 on · 12 of 24 closed

Governed tool calls0 of 3 closed
  • hangar#173 Risky tool-call approvals (ask, driver.risky) and egress/secret requests ask the workspace
  • hangar#175 Managed, fail-closed workspace hook installation in sandboxes (Claude Code; Codex where possible)
  • hangar#190 Workspace webhooks: suspend/resume turns on approval events and accept workspace stop requests
  • Also in private repositories: workspace#22
Replay, trials and shadows12 of 21 closed
  • ✓ hangar#140 MCP surface split: hangar (work tools) and hangar-sandbox (raw drive tools); per-server size budgets replace the 24,700-byte cap
  • ✓ hangar#177 Record-mode model gateway for API-key traffic: normalized, hash-chained, redacted tapes with taint
  • ✓ hangar#178 POST /api/trials with immutable pin sets (ids); batch experiments become a wrapper
  • ✓ hangar#179 fork(at_seq) as replay-prefix, plus a nightly per-repo environment determinism report
  • ✓ hangar#180 Trial-result cache keyed by pin set and task, with a nightly drift sentinel
  • ✓ hangar#181 Eval queue class: low priority from 20:00, TPM-capped concurrency, power-down drain → infra_error
  • ✓ hangar#182 Team monthly eval budget with per-author sub-caps
  • ✓ hangar#183 Shadow intake policy: shadow: {experiment, p} with no write-back, PR or Slack post
  • ✓ hangar#184 Injection canaries in shadow runs (hangar side): append a benign canary instruction to the untrusted context
  • hangar#185 Spike: attention-priced scheduling (per-job attention cost, per-engineer daily budget, "approvals too fast" sensor)
  • ✓ hangar#186 Overnight lane: one Spot node 01:00-10:00 for agent jobs and evals (power.sh + terraform)
  • flightdeck#24 Trial-result cache keyed (harness_digest|config_digest, task_digest, model_id, base_sha, grader_digest) + nightly drift sentinel
  • flightdeck#25 Pin-set backend: submit trials to hangar POST /api/trials {pin_set, task, variant, n}; batch experiments become a wrapper
  • flightdeck#26 Adaptive allocation: group-sequential (3 looks, O'Brien–Fleming), extra trials only on discordant pairs
  • ✓ flightdeck#27 Rubric judges: human-labelled calibration sets, κ ≥ 0.6 gate, prediction-powered inference; never gate on raw judge averages
  • flightdeck#28 Predictive-validity metric: rank correlation of offline deltas with online merged-PR-yield deltas; offline stays advisory until CI excludes 0
  • flightdeck#29 Steer-point replay analysis: steer-avoidance rate on real human corrections, with an A/A check
  • flightdeck#30 Determinism-report consumer: per-repo non-reproducible steps, eval-grade task certification
  • flightdeck#31 Shadows analysis: grade shadow trials by merged truth; CUPED + anytime-valid confidence sequences
  • flightdeck#32 Injection-canary susceptibility rate per harness_digest (from canaries hangar plants in shadows)
  • ✓ flightdeck#37 Spike: steer and interrupt events need hangar.seq and a class (correction / redirect / new info) in hangar.trace/v1

Backlog

unscheduled · 3 of 18 closed

Spikes1 of 7 closed
  • hangar#148 Spike: evaluate removing the A2A endpoint (no consumer) — evidence first, no decision assumed
  • hangar#149 Spike: evaluate removing the E2B-compatible API surface (/e2b + sandbox router) until an external client asks
  • hangar#150 Spike: evaluate removing the microVM tier (e2b provider flavours cloud/selfhosted/cube, tiers fast) until fork/rollout demand is measured
  • hangar#152 Spike: grader/replay isolation on GKE, where the uid split doesn't exist (gap the plan didn't cover)
  • hangar#188 Spike: test-first acceptance for agent:test-first tickets (hidden tests by a different harness)
  • flightdeck#36 Spike (not in 2026): fractional factorial attribution and harness proposers
  • ✓ ai-dlc#198 Spike: extract ownership-digest managed sections as a small reusable module for the team harness repo
Qualification (needs environments or accounts)0 of 3 closed
  • ai-dlc#53 Qualify ai-dlc's remaining surfaces: machine bootstrap (macOS/Linux) and the workflow plugin in Claude Code
  • ai-dlc#85 Qualify ai-dlc's Jira adapter for the local work lifecycle (work finish against Jira Cloud)
  • ai-dlc#172 Native Windows: install and set up projects from PowerShell
Later1 of 7 closed
  • flightdeck#33 Trace-mined skill micro-evals (fork at real skill invocation points; present / absent / edited)
  • flightdeck#34 Flightdeck MCP server: run_suite, compare, explain_divergence
  • flightdeck#35 ATIF import/export adapter for tasks' trajectories
  • spaceport#14 MCP tools ask the workspace's decision point and accept act-chain tokens
  • ✓ spaceport#21 Front page: one annotated screenshot and the use case (frozen)
  • agent-factory#3 Finalize Airframe from the design session and roll it out
  • ai-dlc#203 Disambiguate the five meanings of "capabilities": glossary plus file and option renames

Unsorted

Open issues that roadmap.toml does not place in a theme yet.

3 open issues
  • hangar#419 Harness canary: a candidate CLI fails the contract (no milestone)
  • hangar#430 Control-plane image: newer apt packages (no milestone)
  • hangar#452 CI: main is red at f2c11e893b5c (no milestone)