Runbook: governance break-glass¶

What to do when Tower (the Govern plane) is unreachable, or denies what it shouldn't. Also how to put a decision kind back from enforce to shadow or off. The design is spec 0015. hangar enforces, Tower decides whether a person or a policy allows something, and nothing that goes wrong on the way is ever an allow.

Break-glass is hangar governance local (the procedure). Read what is built today first. With every kind off (the code's default), hangar never asks Tower. GKE's deploy/k8s/config.yaml sets shadow: hangar's rules decide and Tower answers a dry run. The procedure covers a kind in enforce.

Today: hangar decides alone¶

What is built (spike #171, PR #208):

• src/hangar/governance.py, the client side of the hangar.decision/v1 contract. It has the request and decision models, LocalDecisionPoint (hangar's own rules) and HttpDecisionPoint (Tower over HTTP).
• HttpDecisionPoint allows 2 s to connect and 5 s in all, with one retry on a 401 with a fresh token. Its circuit breaker opens after 5 failures in 30 s, for 30 s, and it carries the fail-closed table.
• GovernanceRouter sets off, shadow or enforce per kind. The BreakGlass record needs two different admins, for at most 24 hours.
• tests/test_governance.py, the contract tests against a fake Tower: every failure fails closed for every kind, and shadow never changes an outcome. Break-glass needs two admins and lasts at most 24 hours.

Every approval path calls it through HANGAR_GOVERNANCE and HANGAR_GOVERNANCE_MODES:

• govrelease.py: job release and landing.
• governed.py: governed approvals of flagged tool calls, egress and secret requests.
• govevents.py: Tower's signed events, which suspend and resume turns and stop runs.

• govhook.py: the managed Tower hook for Claude Code turns.

• hangar governance local (governance_cli.py, govrelease.py): break-glass, with a doctor warning while it is on.

What isn't built: replaying break-glass decisions to Tower as records.

With every kind off, every approval is decided by hangar's own rules, as before. That is an admin or a linked repo approver, never on their own job (approvals.decide, #121), the taint gate (#130) and the self-approval rule (#160).

If every kind is off and Tower is down, or denies, there is nothing to do in hangar: nothing waits on Tower. Tower's side (its outage behaviour, its decisions API) is in Tower's own docs.

Moved from this page¶