Why Running and Stopped Are Not Enough
If you run containers that cost money while idle, the instinct is obvious: shut them down when nobody is using them, bring them back when someone arrives. You build a toggle. Running or stopped. Ship it.
This breaks immediately.
Consider the moment a container is being checkpointed to disk. Memory is being serialized, file descriptors catalogued, the process tree frozen mid-execution. Is it running? The workload is suspended and cannot serve traffic. Is it stopped? The process still exists, the kernel has it pinned, a partial checkpoint is being written. You are in a third state your model has no name for, and every component that makes decisions based on state — proxy, scheduler, billing — is now guessing.
A request arrives during that checkpoint. The proxy sees “running” (what else?) and forwards it. The frozen process cannot respond. The request times out. A two-state model gave the proxy no vocabulary to distinguish “running and healthy” from “running but being serialized to disk, please hold.”
The failure that finally forces a redesign: ambiguity between intentional shutdown and crash. A container stopped on purpose and one that segfaulted during restore both land in “stopped.” The correct response is completely different — one needs nothing, the other needs retry or escalation. Same state, opposite semantics.
At Flowdia, this surfaced while building isolated compute environments for our AI-powered application builder, where hundreds of containers cycle between active use and long idle periods. What replaced the two-state toggle is a 7-state finite state machine.
Anatomy of a 7-State Lifecycle
Each state exists because something in the system needs to distinguish it from every other state. No state is decorative.
Starting — the container image is being pulled, filesystem prepared, network configured. The workload process has not started. The proxy should never route here; the idle timer should not count; billing should not charge. Entry: job scheduled or retry from Error. Exit: process begins (to Running) or setup fails (to Error).
Running — the happy path. Workload executing, serving traffic, consuming resources. Entry: successful start or successful restore. Exit: idle timeout fires (to Checkpointing), explicit stop signal (to Stopped), or crash (to Error). This is the only state where the proxy forwards unconditionally.
Checkpointing — memory, file descriptors, open sockets, and the process tree are being serialized to disk. The workload is frozen. The proxy must hold incoming connections, not reject or forward them. Entry: idle timeout or explicit hibernate command. Exit: checkpoint plus upload complete (to Hibernated) or serialization fails (to Error).
Hibernated — zero resource consumption. The container process is dead, all state on disk, ideally replicated to durable storage. This is the state that saves money. Entry: successful checkpoint only. Exit: traffic arrives or explicit wake (to Restoring), or explicit stop (to Stopped).
Restoring — checkpoint data loading, process reconstruction, network stack repair. Entry: wake signal. Exit: container reports healthy (to Running) or restore fails (to Error). The proxy holds connections in a queue during this state.
Error — failed start, corrupt checkpoint, crashed process, failed restore. The key decision: Error is a first-class state, not an exception you hope to catch. Any active state can transition here. The only exit is retry (to Starting), which means every recovery goes through the same initialization path.
Stopped — clean, intentional shutdown. Semantically distinct from Error: we meant to do this. No retry needed, no alert fires. Entry: explicit stop command. Exit: reschedule (to Starting).
Transition Guards and Invariants
A state machine is only as good as the discipline around its transitions. If any component can set the state to any value at any time, you have a string field, not a state machine.
The core invariant: transient states cannot be skipped. A container in Checkpointing cannot jump to Running. It must complete Checkpointing to Hibernated, then Restoring, then Running. Each transition triggers side effects downstream systems depend on. The proxy uses the Checkpointing transition to stop forwarding and start holding. The billing system uses the Hibernated transition to stop the meter. Skip a state and some system misses its cue.
Enforce this with a transition table: a map from each state to its allowed next states. A single transition(from, to) function validates before mutating. If the requested transition is not in the allowed set, reject and log the violation. This replaces scattered if state == X blocks across a dozen files with one auditable source of truth. During development, an illegal transition attempt is almost always a bug in the caller — you want to catch it loudly, not silently allow the system to enter an impossible configuration.
A related concern is concurrent transitions. Two threads might try to move the same container from Running to different states simultaneously — one responding to an idle timeout, the other to a crash. The transition function must be atomic: check the current state, validate the transition, and write the new state in a single operation. Without atomicity, you get split-brain scenarios where the idle detector thinks it sent the container to Checkpointing while the crash handler thinks it sent it to Error, and the actual state is whichever write landed last.
Every transition should emit an event. Not optionally, not in debug mode — every one. These events are the integration surface for webhooks, metrics, and structured logs. When a container follows Starting to Running to Checkpointing to Hibernated to Restoring to Running over a day, you have an ordered, timestamped record. Without transition events, you reconstruct this timeline from scattered logs after the fact, and you get it wrong.
Idle Detection Without Overhead
The trigger for hibernation sounds simple: “nobody is using this container.” Detecting that reliably at scale, without overhead, is harder.
The approach: periodically inspect the OS networking stack for active TCP connections on the container’s published ports. The kernel maintains connection state — ESTABLISHED, SYN_RECV, TIME_WAIT, FIN_WAIT, CLOSE_WAIT — and exposes it through standard interfaces. You read this without intercepting traffic, without a sidecar, without workload cooperation. No active connections for a configurable duration means the container is idle.
The tuning trade-offs are real. Poll too frequently and you burn CPU across thousands of containers. Poll too infrequently and containers stay awake longer than necessary, costing money. The right interval depends on your cost model and latency tolerance; in our experience the sweet spot is a modest frequency that avoids both extremes.
A subtlety: TIME_WAIT connections. When the remote side closes, the local socket enters TIME_WAIT for a kernel-defined period. Technically tracked, but no client is on the other end. Depending on your hibernation aggressiveness, filter these out. Otherwise a burst of short-lived requests keeps the container “active” long after the last client disconnects.
The idle timer must also reset on proxy-level activity, not just socket state inside the container. If the proxy received a connection attempt that has not yet been forwarded, that is demand. Without this reset, a race condition emerges: the idle detector fires, the container hibernates, but the proxy was holding a connection waiting for it to become ready. The client sees an error.
Resist the temptation to rely on application-level idle signals. Having the workload report “I am idle” creates a dependency on workload cooperation. If the workload hangs without crashing, if it enters an infinite loop that consumes CPU but accepts no connections, or if it was generated by an AI agent that never implemented idle reporting, the infrastructure has no fallback. Kernel-level connection state works regardless of what the workload does or fails to do. The infrastructure owns the idle decision, and the workload does not get a vote.
Event-Driven State Notifications
Every state transition is an event external systems care about. The backend updates the UI. Billing starts or stops the meter. Monitoring tracks state distribution. If these systems learn about changes by polling, you get a slow, wasteful, eventually inconsistent system.
Webhook delivery: on every transition, POST a JSON payload with container ID, previous state, new state, and timestamp. This is the primary integration point.
Async dispatch is mandatory. If webhook delivery blocks the transition, a slow receiver delays a restore or causes a checkpoint to time out. The lifecycle engine completes the transition, then enqueues the notification. Receiver latency is the receiver’s problem.
Retry with backoff and a cap. The first retry is fast; subsequent retries space out exponentially. Do not accumulate an unbounded queue for a downed receiver. Cap retries, dead-letter the rest. Make receivers idempotent — the same notification delivered twice should produce the same result. Include a unique transition ID so receivers can deduplicate.
Control API: expose a local interface (Unix socket or localhost HTTP) for programmatic lifecycle management. This lets the backend force-hibernate containers when credits run out, force-wake when the user returns, or force-stop containers that should not be running. Webhooks tell the world what happened; the control API lets the world tell the lifecycle engine what to do.
Failure Modes and the Error State
Design for failure, not around it.
The most interesting failure domain is restore. When a container moves from Hibernated to Restoring and the operation fails, we use a three-tier fallback that trades speed for reliability.
Tier 1 — local cache. If checkpoint files are still on the same node, restore from local disk. No network transfer, no decompression. In the common case — recent hibernation, same node — this is the fastest path by a wide margin.
Tier 2 — remote object store. If local files are gone (node replaced, disk cleaned), download the checkpoint from durable storage. Slower but resilient to node-level failures. Every successful checkpoint should upload to the object store before transitioning to Hibernated, precisely so this fallback exists.
Tier 3 — cold start. If both checkpoints are unavailable or corrupt, pull the image fresh and start a new container. The user loses in-memory state, but the workload comes back. Losing ephemeral state is almost always preferable to a permanently broken container needing manual intervention.
The Error state is where this decision tree executes. Tier-1 failure transitions to Error; the handler attempts tier 2. Tier-2 failure cycles back for tier 3. Only when all tiers are exhausted does the container stay in Error for human review.
Checkpoint failures deserve equal attention. If serialization fails mid-write — disk full, process crash, upload timeout — you have a partial checkpoint on disk. This partial file is dangerous: it may pass a naive size or header check during restore but produce a corrupt, unrecoverable container. The lifecycle engine must transition to Error (not Hibernated, which implies success), delete the partial checkpoint to prevent a future restore from trusting it, and either retry the checkpoint or fall back to keeping the container Running until the next idle window. A system that hibernates on partial data is a system that delivers corrupt state to users who thought their work was saved.
Takeaways
The 7-state lifecycle gives every component — proxy, idle detector, orchestrator, billing, monitoring — a shared, unambiguous vocabulary for a container’s current condition. The transition table enforces invariants that scattered conditionals cannot. The event system provides external systems a consistent, ordered view of every change.
This pattern generalizes beyond hibernatable containers. Serverless function lifecycles (cold, warming, hot, draining, error), database replica promotion (standby, promoting, primary, demoting, error), CI runner provisioning (provisioning, idle, busy, draining, terminated) — any system where multiple components coordinate around async lifecycle transitions benefits from explicit states, enforced transitions, and emitted events. Wherever you find yourself adding boolean flags like is_healthy or is_shutting_down to a two-state model, you are reinventing a state machine badly.
At Flowdia, this model underpins our isolated compute environments, and it has been one of those foundational abstractions that simplified everything built on top of it.
In the next post in this series, we will look at the TCP proxy that makes hibernation invisible to end users — holding connections while the container wakes behind the scenes, then forwarding once it is healthy, all without the client knowing anything happened.
