The microVM tier: E2B and CubeSandbox as hangar providers¶

hangar can run sandboxes as microVMs that start by restoring a memory snapshot, in tens of milliseconds rather than the second or more a container or a GKE gVisor Pod takes. It doesn't run the microVMs itself. It drives a platform that does, through its e2b provider, and stays the control plane: environments, jobs, sessions, the git gateway, budgets and the MCP API all work the same on top.

This is spec 0013's phase P3 and the deployment design for its P4 (§6, §8.2, §9). Nothing here was run on KVM: this machine has none. The tests run the provider against a fake control plane, with real envd daemons behind it (see Testing).

Tiers¶

An environment (or a single create) asks for a tier, not a provider:

# envs/rollouts.yaml
name: rollouts
tier: fast          # fast | isolated | default | heavy

• How a provider is picked. A create that names no provider takes the first provider in its tier's list that is loaded and not cooling down. A create that names a provider gets that provider, whatever the tier, with one exception: an isolated environment (or a create that asks for isolated) only ever gets a provider listed for that tier. Naming another provider, or asking for another tier, fails the create.
• Fallback. A provider that answers "full" or "unreachable" (HTTP 429, 502, 503, 504, or no connection) is skipped for HANGAR_TIER_COOLDOWN_SECONDS (60), and the create moves to the next provider. The audit log records sandbox.tier_fallback. Any other error, such as a missing template, fails the create as before.
• Jobs and sessions use their environment's tier, since they create sandboxes through the same fleet.create.
• Change the lists with HANGAR_TIERS='{"fast": ["e2b"], "isolated": ["kubernetes"]}'. An empty list is allowed, so fast can be just the default provider.

The heavy tier on microVMs¶

The heavy tier (environments.md: Docker inside the sandbox, and the large and xlarge profiles) can run here as well as on GKE's Kata pool, and on a cluster with GKE's Agent Sandbox add-on it has to, since the add-on refuses privileged Kata pods (gke.md):

HANGAR_TIERS='{"fast": ["e2b"], "heavy": ["e2b"]}'   # or ["kubernetes", "e2b"]: Kata first, microVMs when it's full

• The template is the heavy environment's image: hangar renders it with dockerd, containerd, runc and the docker CLI (with Compose and Buildx) copied from HANGAR_HEAVY_DOCKER_IMAGE, plus iptables. A template mapped out of band (HANGAR_E2B_ENVIRONMENT_TEMPLATES) must carry them itself.
• dockerd runs on the guest kernel, started at claim by hangar-docker as root through the template's sudo (heavy environments need sudo: true). The VM's root disk is a real block device, so /var/lib/docker is used as it is: no loop-mounted image as under Kata. The guest kernel needs overlayfs, bridges, veth and netfilter NAT. That wasn't verified on E2B's or CubeSandbox's kernels here (no KVM on this machine): hangar-docker reports what fails (get_sandbox's info.heavy.status, hangar-agent env doctor), and the sandbox runs on without Docker.
• Size: a template is sized when it is built, so a sandbox has its environment's profile (or resources); a job's hangar.profile label that asks for another size is refused here. disk_gb becomes Cube's writableLayerSize.
• Containers stay in the VM: they leave it NATed to the VM's own address, under the platform's network policy (CubeSandbox's eBPF policy denies private ranges by default).
• Hot resume pauses the VM with its memory, so a resumed session finds dockerd, its images and running stacks as they were.

Security¶

• CubeSandbox's envd has no token. CubeAPI returns envdAccessToken: null (spec 0013 §5.4, services/sandboxes.rs), so the envd inside a Cube sandbox answers anyone who can reach it. hangar therefore refuses such sandboxes and kills them, unless HANGAR_E2B_ALLOW_UNAUTHENTICATED_ENVD=true. Set that only when all of these hold:
• CubeProxy and the VMs' network are reachable only from hangar's control plane. On GKE that means a NetworkPolicy on the microVM pool that admits only the hangar namespace; on bare metal, a VPC firewall rule from the control plane's subnet alone.
• Every sandbox is created with allowPublicTraffic=false, which hangar always does. CubeProxy then checks the traffic token on every port, envd's included (CubeProxy/lua/sandbox_backend.lua), and only hangar has that token. hangar checks the part it can see: a sandbox that comes back with neither an envd token nor a traffic token (CubeMaster made it public) is refused and killed, flag or not.
• The VMs' envd port is reachable only through CubeProxy, not directly on the node network.
• Nothing else on that network runs untrusted code. The sandboxes themselves are isolated by Cube's eBPF policy, which denies private ranges by default.
• Upstream fix. CubeSandbox should return an envd token and serve the v2 routes; spec 0013 §8.2 proposes the PR. Once it does, the cube flavour needs neither the flag nor the v1 routes.
• API key. The key is sent only to HANGAR_E2B_API_URL, never logged, and missing from every error message. For cube, CubeAPI can check keys against hangar through its AUTH_CALLBACK_URL (spec 0013 §8.3), which isn't built yet.
• A sandbox's answers are untrusted. Output is cut to a tail, reads stop at their limit, frames are capped and compression is refused, as for hangar-envd. Sandbox ids from the API are checked before they go into a URL path.
• Not a hangar-envd provider. The provider deliberately has no envd attribute, so the preview proxy never mistakes it for one.

Expected latencies¶

These numbers come from the platforms' own published measurements, which spec 0013 §5.5 and §6.2 collected. Source tags ([C8], [G1] and so on) refer to spec 0013 §11. None of them were reproduced on GCP, and the GCP nested-virtualization numbers are not known yet: measuring them is P3's gate.

Plan on about 50–70 ms per create at low concurrency and 25–35 MB per idle VM on bare metal. Expect more on GCP nested virtualization: at least Google's 10% CPU penalty, and more for I/O, which is what restore is. Spec 0013's targets for the tier are create ≤ 100 / ≤ 250 ms at concurrency 10, and a memory fork of N = 10 in ≤ 1 s (§9, P4).

The provider adds little on top. One control-API round trip creates a sandbox, and the envd endpoint comes back in the same answer. Exec through E2B's envd measured 4.4 ms for a raw Start and 18 ms through the SDK's bash -l on loopback (spec 0013 §3.7). Dropping the second login shell removes one profile run from every call.

Testing without KVM¶

tests/test_e2b_provider.py runs everything the provider does, in each flavour, with no microVM:

• Against a fake control plane. FakeE2B is a small FastAPI app with each flavour's routes and quirks:
• CubeAPI's 405 on POST /v2/sandboxes and v2 connect, and its missing envdAccessToken;
• a CubeProxy-like edge that checks the traffic token and authenticates to the daemon itself;
• pagination, pause and connect with rotating tokens, snapshots, alias lookups and Cube's template build.

Every fake sandbox runs a real hangar-envd daemon. hangar-envd accepts E2B's X-Access-Token, so commands, files and terminals cross the real envd protocol. The tests cover create (the v1 or v2 route per flavour), exec with exactly one login shell, the output and read caps, files and modes, a PTY, pause and resume keeping a process alive, a control-plane restart re-fetching tokens, timeouts, list, snapshots, kill, and the refusal of an unauthenticated envd. - Tiers (fast picks e2b, falls back when full and cools down; isolated never falls back; other errors don't fall back) on the fake fleet. - Hot resume (tests/test_hot_resume.py) uses the session tests' scripted agent, real git and tmux: - suspend checkpoints, then pauses; the next turn resumes the same sandbox, with the ignored .venv still there; - a newer checkpoint, a reaped sandbox, or a failed resume each restore the checkpoint instead; - parked sandboxes survive a restart and go with an archive; - the cap evicts the longest-parked sandbox, and neither a rotating credential nor a nearly full fleet is parked.

Follow-ups¶

1. P3's live gate. Run CubeSandbox and E2B Embed on one c3-standard-8 with nested virtualization, then measure create, restore and the nested penalty against the table above.
2. Upstream CubeSandbox. Add the v2 create and connect routes and an envd token, then drop the cube special cases.
3. hangar's edge as CubeAPI's auth callback (spec 0013 §8.3).
4. Native fork. E2B's POST /sandboxes/{id}/fork and CubeSandbox's snapshot-and-clone for fork_session and best-of-N, instead of hangar's snapshot-then-create.
5. hangar-envd as the template's envd. A Cube or E2B template could carry hangar-envd, which is authenticated and smaller, so Cube templates wouldn't depend on cubesandbox-base.
6. Option B in Terraform (a bare-metal MIG), if A misses P4's gate.

Moved from this page¶