Team tooling

Agent factory

Hand work to cloud coding agents and keep every change reviewable. Four planes, five tools and a team harness repo, for a team of six running about sixty agent sessions.

Team harnessDefines what the agents runDefine
hangarRuns the agentsRun
flightdeckJudges the runsJudge
TowerGoverns risky actionsGovern
ai-dlcSets up machines and the workflow pluginSetup
SpaceportServes data to agentsCapability
sean-koval.github.io/agent-factory1 of 14
The problem

Sixty agents at once breaks the way a team works

Live sessions~60

Six engineers, about ten streams each. Nobody can watch them all.

Review1gate

Every change still needs a human. Reviewers become the bottleneck.

Harness changes0evidence

A new skill or prompt is judged by impressions, not paired runs.

Untrusted inputEveryticket

Ticket text and web pages can steer an agent toward a push.

Nobody can watch sixty streams; the tools must bring what needs a person to them2 of 14
The idea

Four planes, one loop, one gate

DoorsMany doorsJira, GitHub, Slack, MCP,schedulesTeam harness repoDefineNative files, harness.lock,one digesthangarRunSandboxes, sessions, jobsand trials on GKEVerify, push through thegateway, land on approvalTowerGovernRules, human approvals,signed receiptsGitHubOne gate: the PRA second human approvesthe exact commitflightdeckJudgePaired statistics andbehavior specs on every runworklockdecidePRevery runtrialspromote the next harness
The agent factory at a glanceDefine · Run · Judge · GovernThe team harness repoDefineNative files and harness.lock,frozen from a good hangar runOne identity: harness_digestpinned as harness.lockhangarRunWork arrives through the doors: Jira,GitHub, Slack, MCP and schedulesSandboxes, sessions, jobs and trialsVerify, push through the gatewaydecide, then a receiptTowerGovernRules and human approvals; theapprover is never the requesterSigned receipts; fails closedpush one commit, open the PRGitHubOne gate: the PRTainted work opens as a draftA second human approves the SHAevery run, through the outboxflightdeckJudgeReads runs from a Parquet data planePaired statistics, behavior specsReports and evidence on PRsreport, then promote the next harnesstools over MCP, bound into each sandboxSpaceport and MCP serversCapabilitiesData products, internal tools and docs search
Each tool owns one plane and meets the others at versioned contracts3 of 14
How work flows

A Jira ticket to a merged PR

01JiraAn engineer moves the ticket to “Ready for agent”Queued
02hangarTicket text is untrusted, so the job starts taintedAdmitted
03SandboxThe agent works with a pinned harness; every step is tracedRunning
04Gatewayhangar verifies, then pushes one commit as the GitHub AppDelivered
05ReviewerA draft PR waits for a second human to approve the exact commitNeeds approval
06OutboxThe merged run is written once, for flightdeck to judgeCaptured
Only approvals and failures light up; everything else stays quiet4 of 14
What the team sees · hangar

Twenty-six agents. Five need you.

Needs you is the console's first tab: held tool calls and questions, one decision at a time, soonest to expire on top.

  • Decide and the next one rises; the agent continues the moment you do.
  • A deny needs a reason. It goes on the receipt and back to the agent.
hangar's Needs you: the decision deck with a held git push to main on top, its changes, branch and expiry, Deny, Allow for this turn and Approve push
hangar, Needs you: the held push on top
Everything running stays grey; the one card that needs you is the only color5 of 14
What the team sees · hangar

Review the change, reply to the agent

A job's detail carries the agent's summary, a composer and the change as a greyscale diff. Your reply runs the next turn; the PR updates.

  • +5 −1 across 2 files, against the base commit, from the sandbox.
  • Only a failing test lights a hunk; additions and deletions stay grey.
A job's detail in hangar: the agent's summary, the Reply to the agent composer, then Changes: files 2, +5 −1, with the whole diff of checkout/export.py switching encode(ascii, ignore) to encode(utf-8)
hangar, a job: the summary, the reply box and the diff
Review and reply in one place; the PR follows the conversation6 of 14
What the team sees · hangar

The whole fleet, one row per agent

List and Board are two views of the same agents. 25 of 120 slots in use.

  • Each row shows what the agent is doing now, its owner and its stage: plan, edit, test, PR.
  • Open an agent in place to decide what it holds without leaving the Fleet.
hangar's Fleet as a list: eight of base-codex's ten agents, each with its tail number, four waiting on a person with an orange diamond, one denied, the rest grey, with owner, stage rail and stage times
hangar, the Fleet as a list
Agents at work stay grey; only the ones that need a person carry color7 of 14
How the team gets better

Change the harness on evidence, not impressions

harness_digestOne identity from lock to report1 Changea skill, hook or plugin in the harness2 Freezea good run into harness.lock3 Experimentstable vs candidate, paired4 Reportverdict with intervals on the PR5 Promotemove stable; drift checks guard it
  1. Change a skill, hook or plugin in the harness
  2. Freeze a good run into harness.lock
  3. Experiment: stable against candidate, paired
  4. Report the verdict with intervals on the PR
  5. Promote: move stable; drift checks guard it
Infrastructure failures are retried and left out of every statistic8 of 14
What the team sees · flightdeck

The verdict lands on the harness PR

flightdeck posts one comment on the pull request that changes the harness and updates it in place: the verdict, both variants' pins, the outcome and its interval.

  • Candidate is better than control on resolved: +21.2 pp, CI +1.5 to +43.9.
  • Paired by task; infrastructure failures are excluded and listed.
flightdeck's report as GitHub shows the comment on the harness pull request: candidate is better than control on resolved (+21.2 pp, 95% CI +1.5 to +43.9), experiment FDK-EXP-0531, the pre-registered rule, the Outcome table and the paired difference
GitHub's comment on the harness PR, written by flightdeck
A harness change is promoted on paired evidence, by someone other than its author9 of 14
Governance

Risky actions fail closed and leave a receipt

TaintTracked

Reading a web page, a remote MCP server or a ticket marks the run.

PushFetch-only

Sandboxes can't push. hangar delivers one commit through its gateway.

Decision pointAsk first

Release, landing, egress and secrets go to Tower: allow, deny or wait.

ApproverNever you

The requester never approves their own risky work. Every decision is signed.

Rollout: off, then shadow, then enforce, one decision kind at a time10 of 14
What the team sees · Tower

The Desk: what needs you, what finished, who you approve for

Tower's home for a person: start work, approve or deny the cards that need you, and see the day's receipts.

  • Each card says what will run, why it asked and what approving does.
  • Every decision files a receipt, chained into the record.
Needs you on Tower's Desk: approval TWR-APR-A5T2, Breaks asking to create a ticket for Globex, what will run, the risk, the credential and what approving does, Deny and Approve
Tower, the Desk: one approval with Deny and Approve
What will run, why it asked and what approving does, before the button11 of 14
What the team sees · Tower

Business objects by reference

A Jira issue, a pull request, a hangar job: each is a chip, and each chip opens a page with what touched it and what is in play.

  • Tower keeps a reference, never a copy; title and state are read from the source.
  • The receipt carries the reference, hashed into the chain.
Start work about it on Tower's page for Jira issue PAY-412: the composer addressed to Ask Breaks, Ask Service agent or Hand to hangar, the issue as a chip with its state, In progress, and two suggested asks
Tower, Jira issue PAY-412: start work about it
Work starts from the object, and its receipt carries the reference12 of 14
Built on · Airframe

One design system, every screen

Airframe holds the parts a person needs to work with an agent: approval card, question, composer, diff, object chips. Every screen in this deck is built from it.

  • Tokens and af- classes only; a test in each repo fails on a color literal.
  • Night and Day, forced colors and reduced motion, in one stylesheet.
The ApprovalCard from the Airframe gallery: a held git push to main with its Cedar policy, checks, expiry, Deny, Approve for this session and Approve push, and a quorum of 1 of 2 approved
Airframe, the ApprovalCard
One stylesheet, every console and docs site; the tool repos vendor a tagged release13 of 14
Where it stands

Running and measured today, governed next

On main

Safe to run

  • Sandboxes, jobs and sessions on GKE
  • Intake posture, taint gates, sealed graders
  • Manifests and traces with harness digests
  • flightdeck data plane and statistics
On main

Measure everything

  • Harness lock: freeze, check, thaw
  • Run outbox and nightly compactor
  • Reports, behavior specs, pinned trials
  • Slack inbox, Tower's decision API
Next

Governed and decided

  • Tower decisions in shadow, then enforce
  • One real flightdeck decision
  • Team harness, frozen from a good run
  • Outbox and compactor on in the cluster
Docs: sean-koval.github.io/agent-factory14 of 14