The team release: what changes as soon as it starts¶
Every default the team release (#75–#117) changes, before and after, and how to opt out.
To opt out, use hangar config set NAME VALUE (kept in hangar's database) or hangar-site (wins over the database). Settings marked restart apply after kubectl -n hangar rollout restart deploy/hangar.
| What | Before → now | Opt out |
|---|---|---|
| The store (#89) | every lookup scanned every sandbox record ever made → expression indexes on state and name, and stopped or failed sandbox records idle for 7 days move to a sandboxes_archive table (still found by id; deleted after 180 days; never while a snapshot or a credential lease points at one). Both happen on the first start, in the background, 100 rows at a time. |
HANGAR_SANDBOX_RETENTION_DAYS=0, HANGAR_SANDBOX_ARCHIVE_DAYS=0 |
| Fleet size and caps (#89, #90) | HANGAR_MAX_SANDBOXES 50 → 120. Per person: 15 live sessions, 20 sandboxes started directly, half the job slots (at least 4) running jobs; a cap reached is a 429 or an MCP error. Caps count the person hangar records on the work, so all of a person's tokens share them (team-scale.md). |
HANGAR_PERSON_MAX_SESSIONS, _MAX_SANDBOXES, _MAX_RUNNING_JOBS (0 = off) |
| Job concurrency (#89, #90) | 4 → HANGAR_JOB_CONCURRENCY=0 (auto): 4 on Docker, 8 on GKE. A free slot goes to whoever has the fewest jobs running. Step 8 raises it to 24 once budgets are set. |
set a number (restart) |
| Sandbox CPU (#89) | each sandbox pod requested its whole CPU limit → half of it (HANGAR_K8S_CPU_REQUEST_FRACTION=0.5); the limit and all memory stay reserved. Each environment's warm-pool template is rewritten once, so the first create of each after the rollout is cold. The control plane requests 500m/1Gi (limit 3Gi). |
HANGAR_K8S_CPU_REQUEST_FRACTION=1 (restart) |
| People (#88) | every token was root over everyone's work → every token belongs to a person. Existing tokens, the admin token and the bootstrap token belong to owner, an admin, so nothing changes until you add someone (step 7). |
— |
| Your credentials (#91) | an unowned credential went into every sandbox → every credential stored before this release becomes the owner's own on the first start, once (hangar cred migration; control-plane ones excepted). Jobs no person owns need a team credential, which must be an API key: hangar refuses a Claude Pro or Max login or a ChatGPT login as one (step 7). A git token (PAT) you stored for sandboxes is yours too until hangar cred owner NAME team. |
— |
| Subscription logins are interactive-only (#125) | a person's own Claude Pro or Max, ChatGPT or Google login went into their jobs from every front door → jobs, schedules, webhooks, experiments and replays refuse oauth_token and auth_file credentials and use the person's API key, else the team's. A job for someone with only a subscription login fails before it starts, with the hangar cred add … --kind api_key command on its ticket or thread. Sessions and sandboxes a person starts still use theirs, with a warning. Jobs already running keep their lease; new ones fail fast. Add API keys first (step 7), then upgrade. |
HANGAR_SUBSCRIPTION_LOGINS=any on v0.2.0 only; removed after it |
| Who a front-door job belongs to (#91, #92) | GitHub and Slack jobs ran for no one → a sender linked to a person (hangar person link) acts as that person: trusted, their caps, budget and login, and only they continue their jobs from a front door (others get a refusal; an admin uses hangar job follow-up). Unlinked senders' jobs run for the team as before. Nothing changes until you link someone. |
— |
| Control-plane credentials (#81) | an mcp token could map the webhook secret or bot token into a sandbox → signing keys, GitHub App keys, Slack and Jira secrets and the GitHub tokens the settings name never reach a sandbox. A direct-mode environment that attaches the same GitHub token as HANGAR_GITHUB_TOKEN_CREDENTIAL or HANGAR_GATEWAY_CREDENTIAL now fails to start; hangar doctor names it (step 6). Saving an environment that sets secrets, image, base, runtime_class, credential_mode: direct, egress, build_secrets or cloud needs an admin. |
hangar cred exposure <name> sandbox |
| Front-door defaults (#82) | jobs from GitHub issues, Slack, A2A and webhooks had no verify → they take the repo's policy (hangar repo set), then the environment's new verify: field. envs/hangar.yaml (and hangar-jobs) now sets verify: bash scripts/ci/verify.sh. A standing check also runs once per base commit before the agent starts; on a red base only new failures count. |
hangar repo set owner/repo --verify "", or verify: "" in the environment |
| Agent scratch files (#82) | delivery committed everything the agent left, including the HANDOFF.md the repo-conventions skill asked for → the skill no longer asks for it, and new root HANDOFF.md, RESULT.md and NOTES.md files and *.orig/*.rej leftovers beside their source stay out of PRs unless the task names them. Images with the skill rebuild once, since its layer changed. |
name the file in the task |
| Job budgets and daily caps (#85) | none → job_budget caps all of a job's turns together, and a repo's max_usd_per_day or HANGAR_PERSON_MAX_USD_PER_DAY holds new jobs for approval. All unset by default, and a day starts at UTC midnight (20:00 in New York) until HANGAR_BUDGET_DAY_TIMEZONE is set (step 8). |
— |
| Secret scanning (#98) | a job's branch was pushed as the agent left it → every new commit's patch, the uncommitted change and the commit messages are scanned first; a stored credential's value or a known token format stops the push and goes back to the agent as a fix turn (2 at most, then secret_found). PR text, comments and text artifacts are masked. |
HANGAR_SECRET_SCAN=warn or off |
| Redaction (#98) | stored values were masked verbatim → also reversed, in hex, base64 and percent-encoded (values of 12+ characters). | — |
| Intake posture (#123) | jobs from GitHub, GitHub Projects, Jira, Slack and inbound webhooks ran with a GitHub token when the gateway was off, and on open egress → HANGAR_INTAKE_POSTURE=enforce (the default): without the git gateway they are refused at submit, with a message on the issue, ticket or thread naming HANGAR_GATEWAY_URL; their sandboxes get at least allowlist egress (the environment's presets and hosts, the repo policy's, the harness's API) and cold-start instead of taking an open environment's warm sandbox (30–60 s). PR auto-fix rounds and env setup started from an issue count too. People's own MCP, REST and CLI jobs see no change (security/front-doors.md). |
HANGAR_INTAKE_POSTURE=warn or off on v0.2.0 only; removed after it |
| sudo off by default (#124) | EnvSpec.sudo was true, so every image wrote NOPASSWD:ALL for the sandbox user → false: the agent isn't root, and the uid split applies on Docker. Every environment image tag changes (images.env_tag covers the Dockerfile), so the first sandbox of each environment after the rollout waits for a build unless you prebuilt (step 3: hangar images plan, then build in CI, #83). Heavy environments imply sudo: true. Under HANGAR_INTAKE_POSTURE=enforce, job sandboxes on Docker run with no-new-privileges (sudo fails even where the image has it), and intake jobs on heavy environments need the repo policy's allow_heavy_for_intake: true. Saved environments that relied on implicit sudo for a runtime apt-get now fail at that step; hangar doctor lists the environments with sudo: true. |
sudo: true on the environment (people's own sessions; not for intake jobs), or move the install to build time (tools.apt) |
| Egress (#97) | sandboxes reached the public internet → unchanged by default for people's own work (HANGAR_EGRESS_DEFAULT=open; intake jobs: above). An environment's egress: {mode: allowlist} sends its sandboxes through hangar's egress proxy (step 10a). The rollout adds port 3128 to the hangar-gateway Service and the proxied NetworkPolicies, which select nothing until an environment is proxied. |
— |
| Egress, closed further (#107) | in proxied environments: a ClientHello with Encrypted ClientHello (GREASE too), several or no server names, or an SSL 2/3 hello is cut off; with the gateway configured, pypi, npm and go are read through hangar's read-only mirror (/pkg/… on the gateway) instead of the registries' own hosts; packages no longer opens GitHub, and crates no longer opens crates.io (the publish API); one person's sandboxes share 640 tunnels; a restart lets open tunnels finish for 20 s. Open environments see no change. |
HANGAR_PACKAGE_MIRROR=false (the registries' hosts again); presets: [packages, github] |
| Pushes through the gateway are scanned (#107) | the agent's own git push went up unscanned → the gateway holds each push, scans it for secrets, and refuses one with a hit (file and line in git's output). Pushes over 64 MB are refused. |
HANGAR_SECRET_SCAN=warn or off; HANGAR_GATEWAY_SCAN_MAX_MB |
| Environments that live with the repo (#93, #99) | a repo's own setup ran in every sandbox, every time → a repo's .hangar/environment.yaml with an install gets versioned builds in the background (first sandbox falls back, with a warning, and schedules one); start, terminals and services run with the sandbox. Repos without the file see no change. |
HANGAR_ENV_BUILDS=false |
| The agent's environment report (#94) | → every harness's MCP config gets a hangar-env server (the environment doctor and requests), and job prompts tell the agent to file blockers with hangar-agent env request and keep going. Requests are never granted on their own: they wait in hangar requests, MCP pending_approvals and the inbox. |
HANGAR_ENV_DOCTOR_MCP=false |
| PRs and the inbox (#96) | → each agent PR gets a review packet comment, and hangar inbox / hangar://inbox / GET /api/inbox lists what waits on each person. Agent PRs keep coming from the control token, and nothing lands by itself, until you set up the App and land (step 9b). |
— |
| Landing needs stale approvals dismissed (#101) | land: auto_after_approval armed auto-merge whatever the base branch's protection → it arms only when the base branch's protection or a ruleset dismisses stale approvals on a push (or hangar can read neither: it doesn't arm), and hangar doctor fails until then. |
hangar repo set owner/repo --no-require-stale-dismissal |
| Auto-fix review rounds (#101) | any collaborator's review started a round on any job, as an instruction → on a person's job only their linked account, an admin's, the policy's approvers or a code owner of every touched path; never a bot; and review text is an untrusted attachment (team.md, rule 6). |
HANGAR_GITHUB_AUTOFIX (ci only) |
power.sh down (#84, #90, #95) |
every node pool and hangar went to 0 at 01:00, so GitHub, Slack and Jira events were lost → it drains (POST /api/admin/drain), checkpoints running turns, then powers down only the sandbox pools (^sandboxes-: gVisor, Kata, microVM). hangar and the system pool stay up: events are queued, jobs wait with waiting: sandboxes power on at 10:00 EDT, creates answer at once (REST 503 with Retry-After). A job turn the pools take away resumes from its checkpoint after power-up. |
POWER_POOLS (a regex) for the script |
power.sh up (#90, #95) |
pools back at their ranges at 10:00, a cold first create (2–4.5 min) → at 09:50 the gVisor pool comes back at a warm floor of 1 node, every image and pack is pulled onto it (the hangar-prepull DaemonSet), then queued jobs start and warm pools are sized from last week's 10:00 hour. It keeps an autoscaling range a terraform apply set while the pools were down, and the microVM pool comes back at its fixed size. |
sandbox_warm_nodes = 0; HANGAR_POWER_WARM_POOLS=off |
| Cost (#95) | the system node runs around the clock, and one Spot sandbox node all powered hours: about +$22/month and up to +$163/month (team-scale.md) | as above |
| Interrupted jobs (#84) | a job caught by a restart or the power-down failed as interrupted → it resumes from its session's checkpoint, at most twice, 20 s apart. |
HANGAR_JOB_RESTART_RESUMES=0 |
| The reconciler (#84) | rows a restart left starting or stopping stayed so, with their leases → a pass at startup and every HANGAR_RECONCILE_INTERVAL_SECONDS (30) finishes them; a GKE pod recreated blank under its claim counts as gone and its session resumes from its checkpoint. Sandboxes from before adopt their current pod's UID. It assumes one hangar process per database (replicas: 1, strategy: Recreate, as shipped). |
— |
| GitHub redelivery (#84) | deliveries that failed while hangar was down were lost → at each start hangar asks GitHub to redeliver them (up to 3 days back). The stored GitHub token needs Webhooks: Read and write, as hangar setup github already asks. |
HANGAR_GITHUB_REDELIVER_ON_START=false |
| GitHub Projects redelivery (#101) | missed projects_v2_item deliveries were lost → the startup pass also asks each HANGAR_GITHUB_PROJECTS organization's webhook to redeliver them. The control token needs the organization's Webhooks: Read and write (classic: admin:org_hook); without it hangar logs a warning. |
HANGAR_GITHUB_REDELIVER_ON_START=false |
| MCP waits (#86) | wait_for_turn waited up to 300 s by default (600 at most), exec as long as its timeout, process attach up to 600 s, terminal_send up to its wait_ms → each at most HANGAR_MCP_TOOL_WAIT_SECONDS (150 s), so claude.ai (which gives up after about 230 s) gets an answer: wait_for_turn says done: false, an exec that runs into the cap points at process. Destroy, pause, resume, extend, snapshot, fork_session and run_agent answer within it too and finish in the background. Any other tool still waiting just past it (165 s) gets an error. |
Raise HANGAR_MCP_TOOL_WAIT_SECONDS only if all your clients wait longer than claude.ai |
| Tool-list notice (#86) | serverInfo.version was 0.1.0 → <version>+tools.<hash>. A claude.ai connector whose tool list is older than the server's gets a one-line note (at most hourly) asking to remove and re-add the connector (step 14). |
— |
| Models (#105, least privilege) | envs/bare.yaml ran Claude Code on the alias opus, pi and Antigravity on flash ids, Codex on its account default, and jobs on whatever the environment said → model: balanced for every harness of bare and of every environment that inherits its model, and jobs default to balanced, as do env setup's agents and the Cloud Doctor's jobs (the owner's decision: an earlier build of this release defaulted to frontier). It is resolved when a sandbox or session is created and pinned for it: Opus on Claude Code (claude-opus-5-5, $4/$20 per MTok), gpt-6.1-sol on Codex. frontier (Fable 5.1, $10/$50) is still there per environment, repo policy or job. Running sessions keep their model. |
model: frontier or a pinned id in the environment, the repo policy or the job |
| Session CLI versions (#105) | → a session records the locked CLI version it was created under. A follow-up on a sandbox running another version hands off (a summary in a new conversation) instead of resuming; sessions from before this release adopt the version of their next sandbox. | — |
| History out of memory (#108) | every job and session ever made was loaded at startup and kept (~30 KB a finished job) → live work and the last HANGAR_RESIDENT_MINUTES (60) stay in memory; the rest is read back by id, name or index. The first start indexes the jobs and sessions tables (~2 s per 20,000 records on 2 vCPUs) before it answers. |
HANGAR_RESIDENT_MINUTES (more memory, fewer reads) |
| Listings page (#108) | list_jobs, list_sessions, GET /api/jobs and /api/sessions sorted the team's newest 500, then filtered by person → pages from the indexes, filtered by person first; each item has a cursor for the next page (REST also sets X-Next-Cursor). |
— |
| Job and session archive (#108) | → finished jobs and settled sessions untouched for 30 days move to jobs_archive and sessions_archive: still found, listed and in the usage report, never deleted. Kept out: open or approved PRs, merged or closed ones whose session hasn't ended, auto-fix triggers waiting, a schedule's last job, sessions with checkpoints, a session a job points at. /metrics counts them off the event loop. |
HANGAR_JOB_RETENTION_DAYS=0, HANGAR_SESSION_RETENTION_DAYS=0 |
| The vault key ring (#109) | one key → HANGAR_SECRET_KEY (or the key file) holds keys newest first; a single key is a ring of one, so nothing changes until you rotate (step 12c). The first start records which key sealed the store (a fingerprint, never the key). |
— |
Repo builds' .git (#110) |
hooks and git config that install left went into every sandbox from the build → hooks (but the samples) and every config key but remotes, branches, user, core, sparse-checkout and git-lfs's own filters are removed before the checkout is captured; hangar's git commands run with hooks off. A repo that relied on install setting up hooks (husky's prepare) sets them up in start. |
— |
| Kubernetes templates per build and per set of services (#110) | services' templates were never deleted; a build's was deleted under a session still running from it → the reaper deletes either, with its warm pool, once no live sandbox uses it (every 10 minutes at most, never in its first 10). The first sweep deletes the services' templates that piled up. | — |
| Credential health (#104) | → every agent credential is checked without a model (by age, format, codex login status, gcloud) at capture, nightly and from hangar status; one that fails or expires within 14 days shows in hangar doctor, hangar status and its renewer's inbox. It never rolls a deploy back (deploy_gate: false). |
— |
| Flight records and the nightly pass (#104, #106) | → a record for each failed, timed-out or slow job and turn (its log end masked and sealed, kept 14 days); at 00:30 the nightly pass checks credentials and runs the Cloud Doctor, which only reports until it has a team key (step 12a). | HANGAR_DOCTOR_SCHEDULE=off |
| Model spend (#105) | → hangar doctor warns (model spend) and hangar status's models line says NO job budget or daily cap while neither HANGAR_JOB_BUDGET_MAX_COST_USD nor HANGAR_PERSON_MAX_USD_PER_DAY is set, whatever the default model (balanced; frontier is a job away) (step 8). |
set a budget |
| Jobs only fetch; hangar pushes (least privilege) | a job's sandbox could push hangar/** branches itself through the gateway (the Cloud Doctor's root-cause job included), and a credential_mode: direct environment put its git token in a job's sandbox → with HANGAR_GATEWAY_URL set, every job's sandbox gets a fetch-only grant whatever its environment's credential_mode or HANGAR_GATEWAY_FOR_JOBS (which now governs sessions only); a git credential its secrets: map is left out, with a warning. hangar pushes the commit it checked with a grant pinned to that branch and commit, without the sandbox's git hooks (a husky pre-push no longer runs on delivery: put the check in verify). Sessions keep their branch-scoped grant. A job's agent that runs git push gets hangar: this sandbox may fetch but not push. Job sandboxes alive across the upgrade (kept for follow-ups) are narrowed to fetch-only at the first start (gateway.narrow in the audit log); a job with no repo, or on a host the gateway doesn't serve, gets no git credential at all. The gateway's upstream credential now matters for jobs on direct-mode environments: their git_token, else HANGAR_GATEWAY_CREDENTIAL or the App |
— (fork the job's session to push by hand) |
| Builds and validation hold nothing (least privilege) | a repo build's sandbox got the environment's non-agent credentials, secrets: and cloud tokens while it ran install → it gets a fetch-only grant for its clone, its mapped build secrets (in install's environment only) and egress per policy, nothing else; env setup's validation the same, without build secrets. Without the gateway neither gets a git credential, so a private repo's build and validation fail with a message naming HANGAR_GATEWAY_URL; set it (Docker on a laptop: http://host.docker.internal:8788, security/git-gateway.md) |
— |
| Env setup (#102) | → hangar env setup <repo>, the hangar.kind: env-setup job label and the hangar-env-setup issue label start an agent-led setup that opens a PR with a validated .hangar/environment.yaml. Its PR opens only if the branch is still at the commit hangar validated; .hangar/** is high risk in every PR's packet. |
don't use the label |
| Build caches (#115) | every sandbox installed dependencies and compiled from nothing → on by default wherever the gateway is configured: after the clone, hangar-cache restores the repo's dependency caches in the background (the sandbox is ready at once), and sandboxes get the variables for sccache, Bazel's HTTP cache and Turborepo's remote cache on the gateway's /cache. Only repo builds write the shared layer; sessions and jobs write a private layer of their own. Objects go to the session bucket's caches/ prefix (up to 100 GB, LRU) unless step 10g names another. |
HANGAR_BUILD_CACHES=false (restart); HANGAR_BUILD_CACHE_PRIVATE_WRITES=false |
| Test reports (#116) | a check was its exit code and the end of its output → every check hangar runs sets the reporter variables (PYTEST_ADDOPTS gains --junitxml=$HANGAR_TESTS_DIR/pytest.xml, GOTESTSUM_JUNITFILE, JEST_JUNIT_OUTPUT_DIR) and reads the JUnit, Cobertura and LCOV files it wrote. Nothing is injected where the command or the repo's pytest config already names junitxml, and the check's own output and exit code are unchanged. A red base is then subtracted by test id, and the repo's flaky list starts filling. |
HANGAR_TEST_REPORTS=collect (read only) or off |
| Turn manifests (#114) | → each turn records the configuration it ran with (one command in the sandbox as it starts), stored under the data dir's new manifests/ (and experiments/ for experiment jobs). Nothing to migrate. |
— |
| Warm pools refill on create (Wave 4 review of #114) | since repo builds (#93) no create refilled its environment's warm pool, so a claimed member was replaced only at the next image build or scheduled sync → a create that started from the environment's own image refills it (never from a repo build's image or for an experiment trial), so warm pools stay at their size through the day, as they did before #93. | the environment's warm_pool: 0, HANGAR_POWER_WARM_POOLS=off |
| Sandbox admission (#117) | → kubectl apply -k deploy/k8s also applies heavy-policy.yaml: in hangar-sandboxes, no pod gets host namespaces, hostPath or hostPort, and a privileged container or added capability needs a Kata runtime class (step 10f). hangar's own pods use none of these. |
— |
| Harness versions and packs (#83, #87) | Claude Code and Antigravity followed "latest" → src/hangar/harnesses.lock.yaml pins every harness (Claude Code 2.1.296, Codex 0.162.1, Antigravity 1.3.3, pi 1.1.0, Gemini 0.64.0); a pack build fails unless the CLI reports the locked version. Packs hold only the vendor's files and are mounted by digest. Every pack tag changes once, and so does every image that bakes a harness in (step 3). hangar pack publish no longer overwrites an existing tag without --force. |
— |