The Transparency Problem
You have a container workload that hibernates when idle. We covered the lifecycle states and orchestration model in our previous post — the container checkpoints its memory to disk, releases its compute resources, and waits. The user has a URL. They click it. They expect their application to respond.
Without a proxy: the user’s browser opens a TCP connection. The SYN packet arrives at the host machine. The container is dead — its process tree is gone, its network namespace is empty, nothing is listening on the port. The kernel has no socket to deliver the SYN to, so it responds with RST. The browser sees “connection refused.” If there is a reverse proxy upstream, the user sees a 502 Bad Gateway. Either way, the experience is broken.
The requirement is strict: something must accept the TCP connection, hold it open, trigger the container restore, and then forward the bytes — all without the client knowing anything happened beyond a brief delay.
A critical insight shapes the entire design: this must work at the TCP level, not at the HTTP level. The workload might serve WebSocket connections, gRPC streams, Server-Sent Events, or raw TCP protocols. The proxy cannot parse the application protocol. It operates on raw byte streams. It is protocol-agnostic by necessity, not by preference.
At Flowdia, we encountered this problem early in our infrastructure work. The solution — a hold-and-forward TCP proxy — is what makes hibernation invisible. The rest of this post describes the design in general terms, because the pattern is applicable far beyond our specific use case.
Two Modes — Holding and Forwarding
The proxy has exactly two operational modes, and they map directly to the container’s lifecycle states.
Forwarding mode is active when the container is running. The proxy accepts an incoming TCP connection on the host-side port, immediately establishes a corresponding connection to the container’s internal port, and splices the two together. Bytes flow bidirectionally. The proxy signals the idle detection system that traffic is present, resetting the hibernation timer. When either side closes or errors, the proxy drains and closes the other side. Textbook TCP proxying.
Holding mode is active when the container is hibernated. The proxy still accepts incoming TCP connections. The three-way handshake completes — the client gets a SYN-ACK — so from the client’s perspective, the connection is established. But the proxy does not forward any data. It queues the connection, triggers a container restore, waits for the restore and health check to pass, then switches to forwarding mode and processes every queued connection as if it had just arrived.
The transition between modes is driven by the container lifecycle state machine. The proxy does not make lifecycle decisions — it reacts to them.
Connection Queuing Under Holding Mode
When a connection arrives at a hibernated container, the proxy accepts the TCP connection immediately. This is not optional. If you delay the accept call, the kernel’s SYN backlog fills up and the kernel silently drops new SYN packets. The client retransmits, gets no response, and eventually times out. By accepting immediately, you complete the handshake and move the connection into userspace, where you control its fate.
The accepted connection goes into a queue. No data is read from it or written to it. The connection simply exists, established but idle. Meanwhile, the client has already sent its application data — an HTTP request, a WebSocket upgrade, a gRPC call setup. That data sits in the kernel’s TCP receive buffer on the proxy side. The kernel manages retransmissions and flow control automatically. The client’s TCP stack is in a perfectly normal “waiting for response” state. Most HTTP clients have generous response timeouts, so a few seconds of delay is invisible.
The queue must have a capacity limit. Each queued connection consumes a file descriptor and kernel buffer memory. Without a cap, a burst of traffic during a slow restore could exhaust resources. When the queue is full, close new connections immediately — the client will retry.
There must also be a hold timeout. If the container fails to restore within a reasonable window, the proxy cannot hold connections forever. After the timeout, close all queued connections cleanly and let the orchestrator attempt a fresh start.
The Restore Trigger
The first connection to arrive at a hibernated container triggers the restore. This is a once-only operation, and getting the synchronization right matters.
When the proxy detects it is in holding mode and a connection arrives, it signals the orchestration layer to begin a restore. What matters is the guarantee: exactly one restore is initiated regardless of how many connections arrive simultaneously. If ten requests hit the proxy in the same millisecond, you do not want ten restore attempts competing with each other. Use a synchronization primitive — a compare-and-swap on an atomic flag, a once-trigger construct — to ensure the first connection triggers the restore and all subsequent connections simply join the queue.
Subsequent connections during the restore window are queued silently. They do not trigger additional restore attempts.
The proxy must also handle externally triggered restores. The control plane might proactively wake a container — perhaps because a scheduled job is about to run, or because a user is navigating to their dashboard. In that case, the proxy simply observes the state transition from Hibernated to Running and switches to forwarding mode.
Forwarding Mode and Connection Splicing
Once the container is healthy and the proxy switches to forwarding mode, two things happen in quick succession.
First, every queued connection is forwarded. The proxy iterates over the hold queue, and for each connection, establishes a new TCP connection to the container’s internal port. The two connections are spliced together. The data sitting in the kernel’s receive buffer — the client’s original request — flows through to the container as if it had just arrived. The container responds, the response flows back to the client. From the client’s perspective, their request took a few extra seconds. No error, no redirect, no retry.
Second, new connections arriving after the mode switch go directly to the container without queuing.
The forwarding itself is bidirectional byte copying. Each connection pair requires two concurrent copy operations: one reading from the client and writing to the container, one reading from the container and writing to the client. When either direction encounters an EOF or error, it signals the other direction to drain and close. The coordination between the two directions is where bugs hide. A common mistake is closing both directions immediately when one side disconnects, truncating the final response bytes still in transit.
There is an important architectural subtlety in how the proxy connects to the container. Rather than connecting through the host’s network stack — which would involve NAT rules, conntrack entries, and iptables traversal — the proxy connects from within the container’s network namespace. It dials the container’s internal IP directly, bypassing an entire category of networking complexity. We will cover exactly why this matters in the next post in this series.
Edge Cases That Will Bite You
The happy path is straightforward. The edge cases are where real-world proxy implementations earn their scars. Here are the ones we have encountered that a first-pass design typically misses.
Client disconnects during holding. The client gives up — timeout expires, user navigates away, laptop closes. The proxy must detect the closed connection and remove it from the queue. Poll the connection for readability: if a read returns EOF, the client is gone. If this was the connection that triggered the restore, the restore still completes. Canceling a restore mid-flight is dangerous — the container is already being reconstructed, and the next request will benefit from it being alive.
Container fails to restore. The checkpoint might be corrupted. The process might crash immediately after reconstruction. The health check might never pass. The proxy needs a clean failure path: close every queued connection, transition back to holding mode (or signal the orchestrator to attempt a clean start instead of a restore), and be ready for the next wave of connections. The worst outcome is leaving connections hanging indefinitely — the client is stuck, the proxy’s resources are consumed, and nothing is making progress. In our experience, it is worth distinguishing between a restore failure (try a clean start) and a repeated clean-start failure (stop trying and surface the error). The proxy should communicate the failure reason upstream so monitoring can differentiate between infrastructure issues and application bugs.
Half-open connections after proxy restart. The proxy crashes and restarts. The new instance knows nothing about the old instance’s connections. The client’s kernel still thinks those connections are open. TCP keepalive is the solution: both sides should have keepalive enabled with reasonable intervals. When the probe goes unanswered, the connection is cleaned up. Without keepalive, half-open connections can persist for hours or days.
TCP keepalive during long holds. If the hold duration exceeds the client’s TCP keepalive interval, the client sends keepalive probes. The proxy’s kernel responds automatically since the connection is established at the OS level. But load balancers and intermediate network appliances have their own idle connection timeouts. If a cloud load balancer’s idle timeout is shorter than your restore duration, it closes the connection before the container is ready. You need to be aware of every hop in the network path and its timeout characteristics.
Rapid hibernate-restore cycles. A container hibernates, gets traffic, restores, handles one request, goes idle, and hibernates again. The proxy must handle the mode switch cleanly every time. The holding-to-forwarding transition must be atomic with respect to incoming connections — a connection arriving during the exact moment of the switch must not be lost. There cannot be a gap between modes where connections fall through.
Port mapping changes. In dynamically orchestrated environments, the host-side port assigned to a container may change between lifecycle transitions. The proxy cannot assume its listener port is stable across a hibernate-restore cycle. Its listener lifecycle must be wired to the container’s lifecycle state machine — when the container gets new port mappings after a restore, the proxy must bind to the new ports and release the old ones.
Concurrent lifecycle transitions. The container is restoring, and the orchestrator decides to hibernate it again (perhaps due to a policy change or resource pressure). The proxy must handle state transitions that interrupt other state transitions. The simplest approach is to let the current transition complete before processing the next one — a serialized state machine with a queue of pending transitions. Trying to support concurrent or preemptive transitions introduces complexity that is rarely worth it.
Performance Characteristics
The proxy adds latency in two distinct profiles, and understanding both is essential for setting expectations.
The cold path is the first request after hibernation. The client waits for the full restore cycle: checkpoint retrieval, container process reconstruction, network namespace repair, and health check passage. The total delay depends on checkpoint size and storage locality. A locally cached checkpoint restores in seconds. A remote checkpoint adds network transfer time. This is the price of hibernation, paid once per wake cycle. The alternative — keeping the container running continuously — costs far more in compute resources than a few seconds of latency on an infrequent first request.
The hot path is steady-state forwarding. The proxy adds negligible latency here. It is performing a TCP byte splice — two file descriptor copies with no protocol parsing, no header inspection, no body buffering. The overhead per forwarded packet is measured in microseconds, well below what any application logic or network round-trip contributes. For all practical purposes, the proxy is invisible on the hot path.
Memory usage is bounded by design. The proxy uses pooled buffers for byte copying — each connection pair draws a fixed-size buffer from a pool and returns it when the connection closes. This avoids allocation pressure under high throughput and keeps memory consumption proportional to the number of concurrent connections rather than the volume of data transferred.
Observability is worth calling out separately. The proxy is an ideal instrumentation point for the entire hibernation system. It knows exactly when the first connection arrives (restore trigger latency), how long the hold lasts (restore duration from the user’s perspective), how many connections were queued (demand during restore), and how many were dropped (capacity issues). Emitting these as metrics gives you a clear picture of hibernation health without instrumenting the container itself. In a system like this, the proxy’s metrics are often more valuable than the orchestrator’s logs.
The fundamental trade-off is clear: the cold path is the cost of hibernation, and the hot path is essentially free. For workloads with bursty access patterns — development environments that sit idle overnight, AI agent workspaces that activate in response to user prompts, internal tools that see traffic a few times per day — this trade-off is overwhelmingly positive. You pay seconds of latency on the first request in exchange for hours of saved compute.
Takeaways
The hold-and-forward TCP proxy is what makes container hibernation invisible to users. Without it, hibernation is a cost optimization that degrades user experience. With it, hibernation is transparent — a brief delay on the first request, and nothing unusual after that.
The pattern extends well beyond container hibernation:
- Serverless cold starts: hold the incoming request while the function instance initializes, forward when ready
- Database connection poolers: hold the application’s connection request while a backend database connection is established or recovered
- Edge compute: hold the request at the edge while the nearest node provisions the workload, forward when the workload is healthy
- Graceful deployments: hold connections during a rolling update window, forward to the new version once its health check passes
The core abstraction is the same in every case: accept the connection so the client does not see an error, queue it, make the backend available, and forward. The details change — the trigger mechanism, the health check, the timeout policy — but the architecture holds.
In the next post, we will dig into what happens to Linux networking state when you checkpoint and restore a container — conntrack entries, veth pairs, iptables rules, and the four production bugs we discovered that made the proxy’s job considerably harder than it should have been. If you have ever wondered why a container’s network “just breaks” after a restore, that post is for you.
