Deploying from CI/CD¶

hangar's control plane deploys from GitHub Actions, not from a laptop. .github/workflows/images.yml builds the environment images and harness packs on every push to main; .github/workflows/deploy.yml builds the control-plane image, waits for your approval, rolls it out, proves it with the acceptance checks and rolls back on failure. A merge to main starts a deploy while the infra is up, and skips it while the infra is off, held or powered down (Infra states). The nightly harness canary needs no cloud access. This page takes you through the first deploy, end to end.

1. Set up the identities, once¶

Neither workflow holds a key. They sign in to Google Cloud with Workload Identity Federation and to hangar with a token kept in a GitHub environment. deploy/gke/setup-cicd.sh creates all of it from your machine, where gcloud (as the project's owner), kubectl and gh (as the repository's admin) are logged in, and hangar login has been run with an admin token. hangar must already run on the cluster (gke.md: deploy/gke/up.sh makes the first install; deploy.yml updates its image, and changes to deploy/k8s, such as the config map, still go through up.sh):

bash deploy/gke/setup-cicd.sh --dry-run   # read-only lookups, and every change it would make, printed
bash deploy/gke/setup-cicd.sh             # make them, then check them (read-only) and say what's left

It reads the project, cluster, zone and registry from deploy/terraform/gke's outputs (else gcloud config), the repository from gh repo view, and hangar's URL from hangar login; --help lists the variables that override each. It is safe to re-run: each step looks first and changes only what is missing or different.

HANGAR_DEPLOY_TOKEN is an admin token named github-deploy, minted through hangar's REST API and piped straight into gh secret set. hangar has no deploy-only scope: the deploy drains hangar, reads its power state, runs the acceptance checks and records the deploy, all admin endpoints, and the jobs, mcp and metrics scopes reach none of them. Only the approved deploy job can read it. --rotate-deploy-token mints a new one, stores it, then revokes the old.

The script also does the upgrade steps a script can do safely: it copies the vault key from the hangar Secret into Secret Manager (hangar-vault-key, piped, never printed), and creates the PyPI and npm remote repositories for the package mirrors. It prints, without doing them, what needs you: the quota requests (Google reviews each, so they can't be automated reliably), the terraform apply for the node pool in the power-down window, branch protection, the GitHub Apps, Jira, and the optional acceptance fixtures.

Re-running it narrows an earlier, wider setup: it removes any other project role from both accounts (an earlier deploy.yml setup gave roles/container.developer to one of them), any other federated identity bound to them, a deployment policy for another branch, repository copies of the deploy secrets, and a looser provider condition.

2. Start the deploy, then approve it¶

Deploy while the sandbox pools are on (10:00 to 01:00 ET): the acceptance checks need sandboxes. That is the window for routine deploy.yml deploys. Image or node-pool upgrades, which replace the sandbox pool's nodes or every environment's image, go by hand with deploy/gke/up.sh in the power-down window, 01:00 to 09:50 ET, as each upgrade guide says (upgrading.md, when to deploy).

gh workflow run deploy.yml --ref main
gh run watch "$(gh run list --workflow deploy.yml --limit 1 --json databaseId --jq '.[0].databaseId')"

A v* tag push starts one too. The run builds first: images (only what Artifact Registry lacks) and the control-plane image, tagged with the commit and pinned by digest. Then the deploy job waits for you: open the run and choose Review deployments, production, Approve and deploy (reviewing deployments). Nothing touches the cluster, and the environment's secrets don't exist for the run, until you approve. Rejecting it ends the run with nothing changed.

The deploy refuses while the pools are powered down. gh workflow run deploy.yml --ref main -f force=true deploys anyway and runs only the checks that need no sandbox (doctor, power, metrics). With hangar off (power.sh off) it refuses even forced: there is nothing running to deploy to; power.sh up deploys main for you.

Merges to main deploy while the infra is up¶

A push to main starts deploy.yml too. Its first job, state, runs bash deploy/gke/power.sh state as hangar-state@, which reads the cluster's power labels and nothing else: no hangar token, no kubectl, no approval needed because it can't change anything.

• Up: the run builds and then waits for your approval on production, like any deploy. A merge never deploys without your click: the deploy job, the only one with the deploy account, kubectl and hangar's admin token, is the production environment's.
• Off, held or down: the run ends green with a notice, infra is down; not deploying <sha>; it will deploy on the next power-up, and builds nothing. No failure, no rollback. images.yml still builds the images on every merge, whatever the power state.
• A burst of merges: the deploy-production concurrency group keeps one run going and at most one waiting; GitHub cancels the waiting run when a newer one queues (concurrency: "any existing pending job or workflow in the same concurrency group will be canceled"). A merge's run whose commit is no longer main's head skips with a notice, both before its builds and again right after your approval (a re-run of an old one too), so you approve the newest commit, not each one. Two things follow from GitHub's rules: a v* tag run waiting behind a deploy is replaced by a merge queued after it (push the tag again, or run it once the queue is clear), and a merge whose push started no run ([skip ci]) leaves the one before it skipped too, until the next merge or power-up.
• The infra went down while a run waited for you: the deploy job checks hangar's power state again after the approval, and skips green if the pools are down or held.

Before setup-cicd.sh has made hangar-state@ (an older setup), a merge skips with a warning instead: re-run the script, or deploy by hand.

Catch-up on power-up. bash deploy/gke/power.sh up (by hand, from off, down or a hold) ends by comparing main's head with the commit hangar runs (its newest deploy record, else the running image's tag) through GitHub's compare API. When main is ahead, and no deploy.yml run for main is already queued or waiting, it starts one with gh workflow run deploy.yml --ref main, as you; it waits for your approval. Same commit, behind, or diverged (a tag deploy): nothing. POWER_CATCH_UP=0 turns it off.

The scheduled power-up job doesn't start deploys. It holds no GitHub credential, and gets none: a token that can start workflows (actions: write) can also re-run, cancel and delete runs and their logs, and stored in Secret Manager it would be one more long-lived secret, readable by an account that already administers the cluster, to protect and rotate, for the sake of saving one command. Instead, with power_deploy_repo set in terraform (a public repository: it compares anonymously), it tells hangar that main is ahead. hangar power, hangar status and hangar doctor then show deploy pending with the command to run, gh workflow run deploy.yml --repo OWNER/REPO --ref main, until the next deploy is recorded. Nothing is recorded at merge time: what's pending is worked out at power-up, from GitHub and hangar, so it can't go stale.

3. Read the result in the run summary¶

Once approved, the job drains hangar (creates and destroys in flight finish), runs kubectl set image, waits for the rollout and for /healthz, then runs the acceptance checks over hangar's REST, MCP and E2B endpoints. The run summary shows:

• the server version before and after (<version>+tools.<hash>);
• the MCP tools added and removed (claude.ai users re-add the connector when the list changed: clients.md);
• one row per check, with core marking the core checks, the only ones that decide a rollback; the team checks after them are reported and never fail the deploy;
• the warm and cold create times.

hangar records the deploy too: hangar power and hangar doctor show the last one and its result.

4. When it fails, it rolls back¶

Any failure after the image changed, a cancelled or timed-out run included, sets the image back to the one it replaced (recorded before the rollout; not rollout undo, which can go one revision too far), waits for that rollout, adds Rolled back to to the summary, fails the run and records rolled_back in hangar. A doctor check that was already failing before the deploy is reported, not rolled back for: a rollback can't fix it. A failure before the rollout (a build, a refusal, your rejection) changes nothing on the cluster.

To go back to an earlier release later, push or re-run a v* tag that points at it: deploys run only from main and v* tags. Rolling back a whole release has steps beyond the image: releases/team-release-operations.md.

When signing in fails¶

• images: skipped on a branch: expected. Google Cloud trusts only runs of main and v* tags, so images.yml run by hand on another branch skips with a notice, and deploy.yml refuses before it builds anything.
• Permission 'iam.serviceAccounts.getAccessToken' denied in the deploy job: the job must run in the production environment with GitHub's default subject claim (either format). Re-run the script; it warns if the repository customizes the claim.

• deployments.apps "hangar" is forbidden from kubectl: the Role is missing or out of date (it needs list for rollout status); re-run the script, which applies it and checks it with kubectl auth can-i.

• deploy: skipped with a warning on a merge, "Couldn't read the infra state": GCP_STATE_SERVICE_ACCOUNT or HANGAR_CLUSTER is missing, or hangar-state@ can't sign in or read the cluster. Re-run the script (it creates and checks them); meanwhile deploy by hand.

Moved from this page¶