Skip to content

When an acceptance check fails

This page is part of Acceptance: proving a deploy on the cluster.

Each line is the most likely cause first. Where to look is always hangar itself: hangar doctor, hangar status (where your hangar has it), hangar power status, hangar job get|events, hangar env builds|build-show, hangar requests ls, hangar inbox, or the REST views named. --keep leaves what the run made, so you can look at the exact sandbox, session or job.

Check What it proves A failure most likely means Where to look
doctor hangar's own checks pass whatever the failing doctor check names; a deploy that broke it hangar doctor (--fix repairs what it can)
sandbox a create works the pools are off or full, an image is missing, a quota hangar power status; the create's error; hangar images plan
exec, stream hangar-envd carries exec, unbuffered envd not reachable (NetworkPolicy allow-hangar-envd), a proxy buffering SSE --no-expect-envd to compare; GET /metrics hangar_sandbox_exec_seconds by transport
files, mcp MCP works within claude.ai's 230 s; a dropped client's create still comes up HANGAR_MCP_TOOL_WAIT_SECONDS too high; a regression of #86 the sandbox's stage in hangar sb ls; the MCP version line
isolation sandboxes can't reach each other, the metadata server or hangar a NetworkPolicy not applied hangar doctor; security/README.md
session checkpoint, suspend and resume restore the work the session store (bucket) unreachable hangar session get <name> (its reason); hangar doctor
backup, metrics backups, /metrics and its scope the backup bucket; HANGAR_METRICS_ENABLED the check's line; hangar admin backups
computer Chromium runs (under gVisor on GKE), a page screenshot is a whole PNG of the display's size and not blank, the display is captured, 3 s of video record, the screen opens through a one-time ticket, a view-only viewer's pointer and keys never arrive and a control viewer's do the image build failed (no Chromium), gVisor missing on the pool (runtime runc), /dev/shm or the display didn't start, the websocket blocked by the load balancer hangar env list (accept-computer); hangar-agent computer status and env doctor in a --keep sandbox; GET /api/audit for screen.*
egress an allowlist sandbox reaches the allowed host through the proxy, gets 403 for another, has no direct route and no DNS; with --with-model, the agent's env request egress reaches its owner as pending HANGAR_EGRESS_URL unset (no proxy in the sandbox); the sandbox-egress-proxied NetworkPolicy missing (direct route); a resolver left in the pod (#97: dnsPolicy: None); the proxy refusing the allowed host (its address resolves privately) GET /api/egress/sandboxes/<id> (mode, enforced, warning); GET /api/audit for egress.deny; hangar requests ls
services a Postgres sidecar answers at postgres:5432 the image refused by HANGAR_ENV_SERVICE_IMAGES, over the services budget, or not pulled the sandbox's info.setup_failures (hangar sb ls --json); hangar-agent svc status in it
heavy a heavy sandbox (accept-heavy) starts Docker; env doctor says heavy; a Compose stack with Postgres comes up under hangar-agent svc and answers on its published port; a Testcontainers test (testcontainers==4.15.0) gets a Postgres container that answers. Skips without a heavy tier in HANGAR_TIERS the GKE Agent Sandbox add-on refusing the privileged Kata pod (it never starts: use the OSS controller or microVMs); kata-deploy missing on sandboxes-heavy; Docker's start failing (storage, cgroups, iptables: the sandbox's info.heavy.status.error); pulls refused by egress hangar sb ls --json (info.heavy); sudo tail /var/log/hangar-dockerd.log and hangar-agent svc logs pg in a --keep sandbox
envbuild the fixture's config builds; a failing install leaves the good build active; a new sandbox starts from it with the dev server up; the preview link opens once the build sandbox can't clone (a private fixture without credentials), install's network blocked by egress, HANGAR_ENV_BUILDS=false; a preview domain without DNS or certificate hangar env builds you/accept-fixture, hangar env build-show <id> (the install's log)
secrets a job whose change holds a synthetic GitHub token isn't pushed: secret_found, its events name the file and line, never the value HANGAR_SECRET_SCAN set to warn or off; the scan couldn't read the change hangar job events <id>; GET /api/audit for secret.found
landing the job's PR comes from the GitHub App; the approver's approval arms auto-merge for that commit, a push disarms it the App not installed on the test repo (the PR comes from the control token: the job's events say so); the webhook lacks pull_request or review events (hangar doctor --fix); a required check never ran; "merged at once" means nothing kept the PR open (add the code-owner rule above) hangar job get <id> (landing.note); hangar doctor (landing … checks); hangar inbox
intake a signed synthetic Jira delivery makes a job for the person its account is linked to; a bad signature gets 401 and a replay is dropped the webhook secret differs from webhook_secret_credential's value; the project isn't mapped, or its repo isn't in HANGAR_GITHUB_REPOS hangar doctor (Jira checks); hangar job get <id>
scale-smoke 20 sessions at once over MCP across 3 test people: the cap refuses one past HANGAR_PERSON_MAX_SESSIONS, a member can't read another's session, p95 create, nothing left after; with --with-model, nobody runs more than their job cap and nobody waits behind another's batch nodes not scaling (p95 high: sandbox_max_nodes, quotas), a leak in teardown; a skip says the fleet was too full to leave the team room hangar power status; GET /api/overview; hangar sb ls for accept-*
power hangar reports the power state; with --power-cycle, the pools go down (a create refused at once with 503) and come back up, timed power.sh couldn't reach the cluster (gcloud, terraform outputs); HANGAR_POWER_UP_JOB without roles/run.invoker hangar power status; bash deploy/gke/power.sh status
prepull after power-up, the first create of each prepulled environment pulls no image an environment changed since the last power-up (its new image isn't on the floor node until the next prepull); the prepull DaemonSet timed out (prepull.ready: false) hangar power status (last_up.prepull); GET /api/admin/power/prepull
warmstart a warm pool claim against a cold start warm_pool not set on --warm-env hangar power status (warm_pools)
status hangar status answers an admin with power, loops, harnesses, builds, failures, egress denies, requests, SLOs and the models line a hangar from before the console (it skips), an admin token missing the admin scope hangar status; its log lines at the end
clouddoctor a Cloud Doctor dry run answers, and says what each new kind of failure would lead to a hangar from before the doctor (it skips); blocked lines name what a real run would miss (a team credential, HANGAR_DOCTOR_MAX_JOBS) hangar admin doctor run --dry-run; hangar admin doctor ls; doctor.md

A check that crashed (the check crashed (exit N)) is a bug in the suite or a Python missing its packages: run it again with HANGAR_ACCEPT_PYTHON pointing at the repo's environment (uv run python).

If the cleanup was incomplete, the run says so, keeps its state file and prints the command that retries it. Leftovers are all named accept-<run>; test people are accept-<run>-p1 and the like (hangar person ls).