Observability: traces, metrics, cost¶

Operators of a software factory need to see where time and money go, and to notice when something breaks. hangar gives you three things:

• OpenTelemetry traces from the control plane: one trace per job run, session turn, MCP call or REST request, down to each sandbox start, checkpoint and agent turn. Claude Code's and Codex's own spans join the same trace.
• Prometheus metrics at GET /metrics: fleet, sessions and jobs by state, latencies, tokens and dollars, budget stops, gateway pushes, MCP calls and control-plane health.
• A cost and usage view: GET /api/usage and the dashboard's Usage page.
• An operator console: hangar status, GET /api/status and the MCP resource hangar://status, for admins: power, loops, versions, builds, the day's failures by cause, egress denies, waiting requests, SLOs, credential health and recent warnings (the operator console).
• Flight records of every failed, timed-out or slow job and turn, with the end of its run log, sealed (flight records). The Cloud Doctor reads them every night.

This builds on the events, usage and budgets in activity.md, and covers the observability parts of spec 0005 and spec 0008 §5.7. It does not include 0005's in-process OTLP collector, traces.db or decision model.

Metrics¶

GET /metrics returns the Prometheus text format.

• Who can read it. An admin token, or a token that can read metrics and nothing else: hangar token create prometheus --scope metrics.
• Turning it off. HANGAR_METRICS_ENABLED=false turns /metrics off (404). hangar then never imports prometheus_client.

Labels never carry a job, session, sandbox or run id.

• Capped labels. Requesters, models, environments, tools, routes and reasons are capped at 200 distinct values each; anything past the cap is other.
• Counters reset on restart. Use rate() and increase().
• When usage is counted. Tokens and cost are added when a turn ends, so a running turn shows up in /api/usage (live) before it does here.
• Gauges are read live. hangar_sandboxes, hangar_sessions and hangar_jobs are computed from the store at scrape time.

Some useful queries:

sum by (requester) (increase(hangar_cost_usd_total[1d]))             # spend per requester, last day
histogram_quantile(0.95, sum by (le, provider) (rate(hangar_sandbox_create_seconds_bucket[1h])))
sum(rate(hangar_warm_pool_claims_total{result="hit"}[1h])) / sum(rate(hangar_warm_pool_claims_total[1h]))
increase(hangar_budget_stops_total[1d]) > 0
histogram_quantile(0.99, rate(hangar_event_loop_lag_seconds_bucket[5m])) > 0.25   # the control plane is stalling
hangar_io_pool_queue_depth > 0                                                     # provider calls are queueing
# prompt-cache hit ratio per harness: the share of prompt tokens read from the cache
sum by (harness) (rate(hangar_tokens_total{kind="cache_read"}[1d]))
  / sum by (harness) (rate(hangar_tokens_total{kind=~"input|cache_read|cache_write"}[1d]))
sum(increase(hangar_cost_uncached_usd_total[1d])) - sum(increase(hangar_cost_usd_total[1d]))   # saved by caching

Cost and usage¶

Weekly adoption numbers (yield, agent share, review wait, cost per merged PR, ...) and the morning standup line: metrics.md, hangar metrics weekly.

GET /api/usage?group_by=day|model|environment|requester|repo|harness|session|job|cache_ttl&since=7d&limit=50  (admin)

hangar usage --by session --cache prints it as a table; --cache adds the cache columns (activity.md).

since takes:

• a relative span: 7d, 12h or 30m;
• an ISO 8601 date or time, such as 2026-10-01;
• epoch seconds.

The default is 30 days ago. Days are UTC. limit caps the rows; for day it keeps the most recent rows.

{"group_by": "repo", "since": 1789313019.0,
 "rows": [{"key": "acme/widgets", "turns": 32, "input_tokens": 1732139, "output_tokens": 347633,
           "cache_read_tokens": 6228892, "cache_write_tokens": 0, "tokens": 2079772, "cost_usd": 67.74,
           "unpriced_turns": 0, "budget_stops": 4, "cache_write_1h_tokens": 0, "cache_hit_ratio": 0.7824,
           "saved_usd": 22.43}],
 "total": {"key": "total", ...},
 "budget_stops": [{"ts": 1791866712.4, "session": "job-3fa91c20", "job": "job-3fa91c20", "repo": "acme/widgets",
                   "requester": "github:sean", "reason": "spent $5.12; the budget is $5.00", "cost_usd": 5.12}]}

The numbers come from agent turns, each counted once:

• Session turns. A job's turns are the turns of the sessions that carry it, so best-of-N attempts, repair runs and the judge are all included. A fork's copy of its parent's history is not counted again.
• Older jobs. Jobs from before sessions carried them use their own turn records.
• Deleted sessions. A deleted session that carried a job keeps its total on the job, so its spend stays in the view. It counts on the day the job was created, with no turns.
• Running turns. A running turn counts with its live usage, on the day it started.
• Costs. cost_usd adds the known costs. unpriced_turns counts turns whose model has no price (activity.md).
• Repos. Repos are shown as owner/name for GitHub, and never with credentials from a URL.

The dashboard's Usage page (#usage) shows, for the last 7, 30 or 90 days:

• spend, tokens, turns and budget stops;
• spend per day, with each day's numbers on hover;
• the top repos and requesters;
• the latest budget stops.

The nightly pass¶

The ops loop runs once a night at HANGAR_DOCTOR_SCHEDULE (cron, default 30 0 * * *) in HANGAR_DOCTOR_TIME_ZONE (default: the power schedule's zone, else UTC): half an hour before the sandbox pools power down. It checks every credential, then runs the Cloud Doctor. The first look after a start only remembers the latest slot, so a deploy never runs it on the spot; a slot missed while hangar was down runs once when it is back. hangar status shows the last pass under nightly.

Deploying¶

Prometheus¶

Both read the token from a Secret:

hangar token create prometheus --scope metrics          # prints hgr_…
kubectl -n hangar create secret generic hangar-metrics-token --from-literal=token=hgr_...
kubectl apply -f deploy/k8s/observability/gke-managed-prometheus.yaml   # or prometheus-operator.yaml

Plain prometheus.io/scrape annotations can't send a token, so there is no annotation-only variant. Managed Prometheus's PodMonitoring fields were checked against the prometheus-engine API reference. Its collector service account (collector in gmp-system) was checked against the project's manifests/operator.yaml.

Traces in Google Cloud Trace¶

Cloud Trace takes OTLP at the Telemetry API, https://telemetry.googleapis.com (/v1/traces). That endpoint needs Google OAuth with the trace.append or cloud-platform scope, which a plain OTLP exporter can't add. So route traces through Google's collector:

1. Install the Google-Built OpenTelemetry Collector:

export GOOGLE_CLOUD_PROJECT=<project> PROJECT_NUMBER=<number>
kubectl kustomize https://github.com/GoogleCloudPlatform/otlp-k8s-ingest.git/k8s/base | envsubst | kubectl apply -f -

Give its Kubernetes service account (opentelemetry/opentelemetry-collector, through Workload Identity) the roles Google's page lists: roles/cloudtrace.agent, roles/monitoring.metricWriter and roles/logging.logWriter. The collector exports traces with otlphttp to https://telemetry.googleapis.com, with the googleclientauth extension.

1. Point hangar at it. Add these to the hangar ConfigMap and restart:

OTEL_EXPORTER_OTLP_ENDPOINT: http://opentelemetry-collector.opentelemetry.svc.cluster.local:4318
OTEL_SERVICE_NAME: hangar
HANGAR_OTEL_SANDBOX_ENDPOINT: http://opentelemetry-collector.opentelemetry.svc.cluster.local:4318   # optional

1. Optional: let sandboxes reach the collector. If you set HANGAR_OTEL_SANDBOX_ENDPOINT, apply deploy/k8s/observability/otel.yaml. It lets sandboxes reach the collector's port 4318 and nothing else in the cluster. The collector also forwards the harnesses' metrics and logs to Cloud Monitoring and Cloud Logging.

The endpoint, scopes, collector install and roles were checked against Google's docs on 2026-10-13: the Telemetry API v1.traces and the collector page above. hangar's spans can also go to any other OTLP backend (Tempo, Jaeger, Honeycomb, Grafana Cloud…) through OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS. None of those has been tested.

Settings¶

Moved from this page¶