Google AX and Agent Substrate: an assessment for hangar¶

Date: 2026-10-13 · Workstream G of spec 0011 · Sources: the main branches of google/ax and agent-substrate/substrate as of this date (AX pins Substrate v0.0.0-20260918… in its go.mod), plus the launch post. Every claim below links to its source; nothing was run against a live cluster.

Recommendation¶

1. Don't build on AX now. Spec 0011 §4 got this right. AX is a preview whose README promises "major breaking changes prior to a stable release". It duplicates hangar's control-plane role (its own Redis store, workspaces and models). It also hides the one Substrate feature hangar would want, memory snapshots: AX's suspend keeps /workspace only.
2. Interoperate through A2A, which hangar now serves (a2a.md). One correction to §4: AX itself has no A2A or MCP endpoint, only a gRPC API. What can delegate to hangar is an agent running in an AX Task (ADK, LangGraph, any A2A client), which Google's launch post lists as a supported workload.
3. If we ever add a provider here, target Agent Substrate directly, not AX. hangar and AX sit at the same layer; Substrate is the layer below both. A providers/substrate.py would get MEMORY-fidelity suspend and resume, which is hot resume for 0010 M4, without a second orchestrator in the path. The providers/ax.py design below shows what the AX route would take and why it is the weaker one.
4. Revisit when any trigger in §8 fires. The likeliest are Substrate actor forking landing, or Substrate exposing exec, files and memory snapshots through a stable API.

1. What they are¶

AX (Agent Executor) is "a high-throughput, declarative orchestrator to run billions of autonomous agent workloads in a cluster". It is written in Go, Apache-2.0, and was announced on 2026-05-20 as "available now in preview" (README, blog). It has three binaries (DESIGN.md):

• ax: a kubectl-shaped CLI;
• ax-server: a gRPC API that keeps state in Redis rather than CRDs ("storing millions of short-lived tasks as Kubernetes CRDs pushes etcd past its comfort zone");
• ax-task-runner: PID 1 inside every task.

Agent Substrate is the runtime under AX: "a secure-by-default agent execution runtime engineered to run millions of sandboxes with 10x higher density than standard container runtimes", claiming "sub-500ms resume operations at over 500 suspend/resume activations per second" (README). It maps many Actors onto fewer warm Workers, the pods of a WorkerPool. It runs gVisor or microVMs (cloud-hypervisor), keeps snapshots in GCS or S3, and routes traffic through the atenet router, which "resumes suspended Actors on demand" (glossary). It is "pre-1.0" with no backward-compatibility guarantees.

2. Primitives side by side¶

The mapping isn't one to one. An AX Task covers hangar's sandbox plus the command running in it. A Workspace is most of an environment. A session and a job have no AX counterpart: they are the parts of hangar above the sandbox, which spec 0011 §4 calls hangar's value. That is why AX is a peer of hangar's sandbox-and-environment layer, and Substrate is the layer under it.

3. Checkpoints and suspend, compared with 0010¶

hangar's checkpoints and Substrate snapshots complement each other. hangar's are semantic and portable: they survive a provider change, a Substrate upgrade and a lost cluster. Substrate's are fast and complete, including memory. A Substrate-backed provider would use MEMORY suspend as the hot path and keep 0010 checkpoints as the durable fallback. That is the shape 0010 M4 already plans for GKE Pod Snapshots.

The launch post's "durable execution … via an event log and snapshotting", "session consistency" and "trajectory branching" (blog) are not in the open-source repo yet. The repo's docs describe Redis-backed resources, phases and conditions, and list branching as roadmap. Spec 0011 §4 repeats the blog's claims. They should be read as direction, not as what ships.

4. Harness support (bring your own harness)¶

§4 says bring-your-own-harness "is still on its roadmap". That's only half right:

• What runs today: any image and command. A Task's spec.image can be any image that has an executable at /usr/local/bin/ax-task-runner, and the runner starts spec.command (runner.md). Substrate ships a Claude Code multiplex demo, with three Claude Code agents on two pods, although it needed runtime workarounds for two open atelet bugs (#189, #197). So Claude Code, Codex, pi or Antigravity can run in an AX Task now.
• What's on the roadmap: decoupling AX's own built-in harness. Workspace goal setup always uses Antigravity with GEMINI_API_KEY (sandbox.md). The roadmap item is "Allow Customization: decouple the built-in workspace bootstrap and coding agent harness".
• What hangar would miss: a way to drive a harness turn by turn. AX has no exec or turn API. Process and file access exist only as Substrate guest services, and only when the Task sets debug: true, which "allow[s] arbitrary process execution and file access inside the sandbox" (sandbox.md). hangar's whole model (inject credentials, write the prompt, run claude -p … --resume, read RESULT.md, verify, push) runs over exec and files. So a hangar provider on AX would have to run every Task in debug mode. AX's docs frame that mode as a debugging aid, not as the control path.

5. MCP and A2A¶

• MCP: AX Workspaces pre-wire MCP servers (static endpoints or a Google registry query) for the agent in the Task (manifests). hangar already has the equivalent in environment mcp: (0006). AX doesn't serve MCP: there's no MCP endpoint for clients to drive AX. Substrate's roadmap has "Native MCP Server Hosting" (MCP servers as Actors).
• A2A: the AX repo has no A2A endpoint. The launch post lists "any agent using the Agent2Agent Protocol (A2A)" among the workloads you can run, and Substrate's roadmap has an "Actor-to-Actor (A2A) Calling Model". So the interop path is agent-to-agent: an ADK or LangGraph agent in an AX Task calls hangar's /a2a with a jobs token. This needs nothing from AX beyond egress to hangar. Substrate's egress credential injection (docs) could even keep the hangar token out of the Task.

6. Kubernetes and operations¶

• Substrate: GKE 1.36 with Kubernetes beta APIs enabled at cluster creation (PodCertificateRequest, ClusterTrustBundle; "GKE only honors enableK8sBetaApis at creation time"), or GKE 1.37+. "Versions below 1.36 are not supported." (setup-gcp) It also needs:
• a node label ate.dev/substrate-version on every pool, or new nodes run no workers;
• a GCS (or S3) snapshot bucket with Workload Identity grants;
• a PostgreSQL state store (Cloud SQL supported);
• its own system namespace ate-system.
• AX on top: Redis, images built and pushed with ko, the ax-system namespace, and the Go toolchain for the CLI (README).
• hangar's cluster today: GKE Standard on the RAPID channel at 1.35.2+, for Agent Sandbox (gke.md). Adopting Substrate means a new cluster (or recreating ours with beta APIs on), plus Postgres, plus a second sandbox runtime next to Agent Sandbox. Its gvisor sandbox class overlaps with ours; its microvm class is an option hangar has only locally (Kata, local-microvms.md).

7. Maturity¶

8. Provider design: providers/ax.py¶

hangar's Provider interface (providers/base.py) is small on purpose: "everything above this layer … is built on exec, read_file and write_file". Here is how each method would map onto AX:

What's missing or awkward:

• A control path for exec and files. The only one is debug-mode guest services. We'd be building on an interface AX describes as for debugging, with arbitrary process execution open to anyone who can reach the router with the actor header. The router must be locked to the hangar control plane by NetworkPolicy, and Substrate's own "user authorization" is still a roadmap item.
• Immutable Tasks. hangar's extend and TTL changes are control-plane-only anyway, but nothing about a running Task can change.
• No memory snapshots through AX. AX suspends with /workspace fidelity, so it buys nothing over hangar's own checkpoints for resume quality.
• Two orchestrators. AX's Redis state, workspace bootstrap, Models and goal-driven Antigravity setup are all unused, but deployed and upgraded, and they break with every v1alpha1 change.
• Cluster prerequisites (§6) and no GKE Agent Sandbox features (warm pools as SandboxWarmPool, Pod Snapshots).

What it would buy:

• Density: Substrate's multiplexing of idle actors onto shared workers ("30x+ oversubscription" in the demo). Coding sessions are idle most of the time, waiting on the model or on a human, which is exactly the pattern it targets. hangar today pins a pod per live sandbox and releases it only on suspend.
• Start time: golden-snapshot restores instead of a pod start. hangar's cold path is node scale-up (2–4 min on GKE) plus an image pull (config); warm pools hide it only for prebuilt environments.
• Hot suspend: only via Substrate MEMORY fidelity, which AX doesn't expose today.

Why Substrate directly is the better target. providers/substrate.py would use the same guest services for exec and files. It would create Actors from an ActorTemplate per environment image, with preferredFidelity: MEMORY. It would suspend and resume through ateapi, and so get hot resume. The router already "resumes suspended Actors on demand", which fits sessions. There would be no AX server, no Redis and no workspace bootstrap to carry. hangar stays the orchestrator, and AX remains what it is: a peer we talk to over A2A.

9. Triggers for revisiting¶

Revisit this when any of these happens:

1. Substrate actor forking ships (roadmap "Coming Soon"). Memory-level fork_session and best-of-N (workstream D) would then be a few hundred milliseconds instead of N cold sandboxes.
2. Substrate gets a supported, authorized exec and file API, not debug-mode guest services, and a Python client, or "user authorization" lands on the control plane.
3. AX exposes MEMORY fidelity (its "Migration to the New Actor" item) and v1 of its API. Only then does providers/ax.py beat providers/substrate.py.
4. hangar's own numbers say so:
5. more than about 30% of session resumes or job starts pay a cold pod start;
6. or idle sandboxes held for keep_minutes cost more than the sessions bucket and a Postgres would;
7. or a 0010 M4 Pod Snapshots trial shows we need hot resume and GKE Pod Snapshots can't deliver it.
8. GKE ships Substrate as a managed add-on, or Agent Sandbox and Substrate converge, which would remove the cluster-recreate cost in §6.
9. A customer or partner runs AX and wants their agents to use hangar. That needs nothing new: point them at a2a.md.

Ideas worth borrowing now¶

These are already in §4 and still hold, with one addition:

• Declarative workspaces that pre-wire MCP servers and skills: 0006 mcp: plus a skills registry.
• A single-writer event log for run state: workstream C's events.
• Branching from checkpoints: workstream D's fork_session.
• New: goal-driven environment setup. AX hands a plain-language goal ("install dependencies and run the test suite") to an agent on first boot and records a marker so it never reruns. hangar could do the same for environments without a build recipe: run a setup turn, then snapshot the result as the environment's image.