The Architecture in One Picture
Over the past four posts, we built up the pieces individually: a lifecycle state machine, a TCP proxy, networking fixes for checkpoint/restore, and a sandboxed runtime. Now let us put them together.
A user or API client triggers a workload. The orchestrator — think Nomad or Kubernetes — schedules it on a worker node with available resources. A custom driver plugin on that node launches the workload inside a sandboxed runtime that intercepts system calls, enforcing a security boundary between the workload and the host kernel.
A TCP proxy binds to the host-side ports and forwards traffic into the container. An idle detector monitors network activity through the proxy. When the workload goes silent for longer than a configurable threshold, the driver checkpoints the container to disk, uploads the snapshot to object storage, and the proxy switches to holding mode.
When traffic arrives at a sleeping container, the proxy completes the TCP handshake so the client sees no refusal. It signals the driver to restore. The driver downloads the checkpoint, reconstructs the container, repairs the network stack — flushing stale ARP entries, rebuilding conntrack state, re-establishing DNAT rules — and the proxy drains its held connections into the now-live container. The client experiences a brief delay, nothing more.
The lifecycle state machine coordinates all of this, and webhooks notify the backend of every transition so it can update billing, status indicators, and metrics.
The key insight: none of these components work in isolation. The proxy depends on the lifecycle state machine to know when to hold versus forward. The network repair depends on the checkpoint sequence to know which layer needs fixing. The idle detector depends on the proxy to signal traffic. The orchestrator depends on the driver to report health accurately. It is a tightly coordinated system where each piece has a specific responsibility and a clear interface to its neighbors.
The Orchestration Layer
The separation of concerns between the orchestrator and the driver is the decision that makes the rest tractable.
The orchestrator’s job: scheduling (place the workload on a node with available resources), health monitoring (is the workload alive?), scaling (add or remove instances), and job lifecycle (start, stop, update).
The driver’s job: everything runtime-specific. Pulling images. Creating the sandbox. Configuring the network namespace, veth pairs, and DNAT rules. Starting the process. Checkpointing. Restoring. Network repair. The orchestrator does not know about sandboxed runtimes, checkpoints, or ARP tables — it calls the driver’s standard interface (start, stop, health check) and the driver handles the rest.
This separation means you can swap orchestrators (from Nomad to Kubernetes) or swap runtimes (from gVisor to Firecracker) without rewriting the entire system. The interfaces between components are stable even if the implementations change.
Service mesh integration: a service discovery system registers each running container and its host-side ports. A reverse proxy reads the registry and routes incoming HTTPS traffic to the correct host-side port. The TCP proxy and DNAT rules then deliver traffic to the container. When a container hibernates, its service registration stays active — the reverse proxy still routes traffic to the host, where the TCP proxy catches it in holding mode. The hibernation is invisible at the routing layer.
Health Checking Across Hibernate Boundaries
A hibernated container is not unhealthy — it is intentionally stopped. But the orchestrator’s default behavior is to restart unhealthy workloads, creating a conflict: the orchestrator wants to replace the hibernated container, but the driver wants to keep it sleeping until traffic arrives.
The solution: the driver owns the health-check response. Standard probes — TCP connect, HTTP endpoint, process liveness — do not work for hibernated containers because there is no process to check. The driver intercepts health-check requests and responds based on the lifecycle state. Running containers report healthy. Hibernated containers report healthy with metadata indicating sleep. Errored containers report unhealthy. The orchestrator only reschedules on unhealthy, never on sleeping.
The driver must maintain its own state even when the container process does not exist. It is always running on the worker node, holding the lifecycle state, responding to health checks, and listening for restore triggers. The driver is the container’s representative when the container itself is gone.
Data Persistence and the Checkpoint Store
Checkpoints must survive node failures. We use a two-tier storage pattern with a fallback.
Tier 1: Local disk. Checkpoint files are written to the node’s local storage first. Restoring from local disk is the fastest path — no network transfer. For the common case where a container hibernates and restores on the same node, this is all you need.
Tier 2: Object storage. After the local write completes, the checkpoint is uploaded to an S3-compatible object store. This provides durability that survives node failures, disk corruption, and cluster replacements. Restoring from object storage is slower but always available.
Tier 3 fallback: Cold start. If both checkpoints are unavailable — corrupt data, storage outage, a checkpoint that was never written — the driver falls back to starting a fresh container. The workload loses in-memory state but the environment is recreated from the base image. The system degrades gracefully rather than breaking.
Garbage collection: checkpoints accumulate. Each hibernate cycle creates a new snapshot. Our strategy: keep exactly one checkpoint per container on local disk (the latest), keep the latest N in object storage for rollback, delete everything older. Garbage collection runs asynchronously and must be idempotent and safe to run concurrently with checkpoint/restore operations. We use a lease mechanism: a checkpoint is eligible for deletion only if no operation has touched it within a grace period.
Divergence handling: if the local and remote checkpoints are from different points in time (e.g., the local write succeeded but the upload failed), the driver prefers the newer one. Each checkpoint carries a monotonic sequence number. The driver compares versions and picks the higher sequence. If only one exists, it uses that. If neither exists, cold start.
AI Agents as Infrastructure Clients
This is the use case that makes hibernation not just cost-effective but essential.
AI agents interact with containerized development environments in bursts. The user sends a prompt, the agent works for a few minutes (writing files, running commands, modifying databases), then the interaction stops. The container sits idle until the next prompt — which might be minutes, hours, or days later.
Without hibernation, every container stays running and consuming resources for the entire duration between interactions. At scale, the economics are brutal. A container might be actively used for five minutes out of every hour — an idle-to-active ratio of 12:1. You are paying for twelve units of compute to get one unit of work. Multiply across hundreds or thousands of concurrent user environments and the waste is staggering.
With hibernation, idle containers checkpoint their state and release their compute resources. Memory is freed. CPU cycles are returned to the pool. The only cost is storage for the checkpoint file. When the user returns, the container restores in seconds with everything preserved: open files, running processes, environment variables, installed packages, database state, even shell history. The user picks up exactly where they left off, with no setup, no re-installation, no context loss.
The agent communicates with the container through a structured protocol — a well-defined set of operations like reading files, writing files, and running shell commands. This protocol runs over a network connection that passes through the TCP proxy. When the container is hibernated and the agent sends a request, the proxy holds the connection, triggers a restore, and forwards the request once the container is ready. The agent experiences a brief delay, not an error.
The Agent-Container Interaction Loop
- User sends a prompt to the API backend.
- The backend checks if the user’s container is running. If hibernated, it triggers a wake or lets the proxy handle it on the next network request.
- The AI agent connects to the container’s tool protocol endpoint.
- The agent reads the current state of the project (files, database, etc.).
- The agent makes changes: writes files, runs commands, installs dependencies.
- The agent finishes and disconnects.
- The container goes idle. After the idle timeout, it hibernates.
- The user returns. Repeat from step 1.
This loop demands every component working in concert. The lifecycle state machine manages transitions. The proxy provides transparent reconnection. The network repair ensures connectivity after restore. The sandboxed runtime ensures the AI-generated code cannot escape the container.
That last point deserves emphasis. AI agents generate and execute code that no human has reviewed. The agent might install a malicious package, run a shell command with unexpected side effects, or produce code that attempts unauthorized network access. Defense in depth — sandbox plus network isolation plus per-workload database separation plus resource limits — ensures that a compromised container is a contained incident, not a lateral movement opportunity.
Operational Lessons
Five hard-won lessons from operating this system in production.
1. Always test the second hibernate/restore cycle, not just the first. The first cycle starts from a clean slate. The container was freshly created, the network namespace is pristine, there are no stale ARP entries, no leftover conntrack entries, no accumulated kernel state from previous checkpoints. Everything works. The second cycle is where reality asserts itself. ARP entries from the first run are cached on the gateway and conflict with the restored container’s expectations. Conntrack entries from the first session confuse the kernel’s connection tracking on the second restore. File handles that were properly closed on the first checkpoint are in a different state on the second because the process had time to open new ones. Most of our networking bugs only manifested on the second or third cycle. Our CI pipeline now runs every workload through at least three full cycles: start, run traffic, hibernate, restore, run traffic, hibernate, restore, run traffic, verify. If it passes three cycles, it is reasonably likely to pass N cycles.
2. Instrument every layer of the network stack independently. When a connection fails after restore, the failure symptom is always the same from the user’s perspective: the URL does not work. But the root cause could be at any of half a dozen layers. Is the veth interface up? Does it have the correct IP address? Is the ARP entry for the gateway resolved? Is the gateway’s ARP entry for the container resolved? Are conntrack entries clean, or is the kernel matching new packets against stale entries? Are the DNAT rules in the host namespace correct? Is the sandboxed runtime’s userspace network stack responsive? We built a post-restore health check that verifies each layer in sequence and reports which specific check failed. The check runs automatically after every restore, before the proxy switches from holding to forwarding mode. If any layer fails, the restore is retried with additional repair steps targeted at the failing layer. This turned hours of debugging into seconds of diagnosis. Instead of staring at tcpdump output trying to figure out where packets are disappearing, we get a structured report: “ARP resolution for gateway failed, flushing neighbor cache and retrying.”
3. Treat checkpoint/restore as a completely different code path from start/stop. They share some infrastructure — both need an image, both need network configuration, both end with a running container. The temptation to unify them is strong, and we succumbed to it early on. It cost us weeks. The preconditions are different. A start assumes no prior state — it pulls an image, creates a filesystem, initializes a network namespace from scratch. A restore assumes prior state exists — a checkpoint file on disk, a previously configured network namespace that needs repair, kernel state that needs reconciliation. The failure modes are different. A start can fail because the image is missing or the registry is down. A restore can fail because the checkpoint is corrupt, the kernel version changed, or the memory layout is incompatible. We now maintain explicitly separate code paths. They share utility functions but the top-level orchestration logic is distinct, with its own precondition checks, error handling, and documented assumptions about expected entry state.
4. Defense-in-depth is not optional when running untrusted code. The sandboxed runtime is the primary boundary, but it is not the only one. Per-workload database isolation, encrypted secrets, network namespace separation, and resource limits each prevent a different category of attack. A sandbox escape alone should not give access to other workloads’ data. The threat model is not “a sophisticated attacker” — it is “arbitrary code execution at scale, every day, by design.”
5. The TCP proxy is the single most important component for user experience. Users do not know what checkpoints are. They do not know about ARP tables, conntrack entries, or DNAT rules. They know one thing: they have a URL, and they expect it to work. The proxy is what makes the URL work even when the container behind it does not exist.
If the proxy is down, every hibernated container is unreachable. Users see connection refused or 502 errors. It does not matter that the checkpoints are perfectly preserved, that the lifecycle state machine is in the correct state, that the restore logic is flawless. If the proxy is not there to catch the connection and trigger the restore, none of it matters.
We invest disproportionately in proxy reliability. It has its own health monitoring, independent of the containers it serves. It has graceful degradation: if it cannot reach the lifecycle manager to trigger a restore, it holds the connection and retries rather than dropping it. It has connection limits to prevent resource exhaustion under load. It logs every held connection, every restore trigger, every mode transition, because when something goes wrong at 3 AM, the proxy logs are always the first place we look.
What Comes Next
Live migration: currently, restoring on a different node requires downloading the checkpoint from object storage. Live migration would transfer container memory directly between nodes, enabling seamless rebalancing without the cold-cache penalty. The challenge is coordinating the migration with network stack repair — IP addresses, ARP entries, and conntrack state all need to follow the container to its new node.
GPU workload hibernation: checkpointing GPU state (VRAM contents, compute context) is an unsolved problem in the general case. As AI workloads increasingly require GPU access, extending hibernation to GPU containers would unlock significant cost savings. Constrained versions — checkpointing at well-defined quiescent points where GPU state is minimal — are within reach.
Tighter agent-container integration: the structured protocol between the AI agent and the container could include lifecycle awareness. The agent could signal “I am done, you can hibernate” or “I am about to start a long operation, do not hibernate.” This would make idle detection more precise and reduce false hibernations.
At Flowdia, this infrastructure powers the platform’s ability to let AI agents build complete applications in isolated, secure, cost-effective containers. The patterns we have described in this series — state machines, proxies, network repair, sandboxing — are not specific to our use case. Anywhere you need to run workloads that sleep, wake on demand, and execute untrusted code, these same building blocks apply.
