Checkpoint/Restore Sounds Simple Until You Add Networking
Process-level checkpoint/restore captures memory pages, file descriptors, register state, and signal masks — everything the kernel knows about a process — serializes it to disk, and later reconstructs the process from that snapshot. On paper, you freeze a container, store the checkpoint, and bring it back minutes or hours later.
In practice, the process’s view of the network and the kernel’s view diverge during hibernation. The process thinks it has connections, routes, and interfaces. The kernel has moved on. ARP caches have expired. Connection tracking entries have gone stale or been replaced. The veth interface that used to carry traffic is in a state the process never anticipated. The process wakes up believing it is still connected to the world. It is not.
In Part 1 we designed the lifecycle states, and in Part 2 we built the proxy that makes transitions invisible. Now we go deeper: what actually breaks in the Linux networking stack across a checkpoint/restore cycle?
At Flowdia, these bugs surfaced while operating hibernatable sandbox environments in production. Each one manifested as a generic connectivity failure — the kind where logs say nothing useful and monitoring shows “container running, health check passing” while real traffic silently dies. The fixes required understanding how the Linux networking stack works at each layer, and how checkpoint/restore violates assumptions the stack makes about its own continuity.
The Setup — Bridge Mode Networking With a Userspace Stack
Each container gets its own Linux network namespace — a fully isolated copy of the network stack with its own interfaces, routing table, ARP cache, iptables rules, and conntrack table.
A veth (virtual ethernet) pair connects the container’s namespace to the host. One end lives inside the container namespace, the other is attached to a bridge on the host that provides a path to the outside world through NAT.
A userspace network stack runs inside the container sandbox. It captures all ethernet frames via a raw socket bound to the container-side veth interface and owns all IP-level processing: TCP state machines, UDP handling, ICMP, ARP responses. The kernel-level veth interface inside the container has no IP address at steady state. It operates as a dumb pipe, shuffling raw frames between the physical veth and the userspace stack.
On the host side, iptables DNAT rules translate the host’s dynamically assigned ports to the container’s internal ports. The kernel’s conntrack system tracks these translations so response packets can be correctly un-NATed on the way back.
This architecture provides strong isolation but creates unique challenges for checkpoint/restore. Network state is distributed across three layers: the kernel (interfaces, routes, ARP, conntrack), the veth hardware layer (MAC addresses, link state), and the userspace stack (TCP connections, socket buffers, protocol timers). A correct restore must reconcile all three.
Bug 1 — The Vanishing IP Address
Symptom: After restoring a checkpointed container, the container has no network connectivity at all. It cannot reach the internet, cannot receive inbound traffic. Completely dark. The container process is running and reports healthy at the application level, but every network operation fails.
What happened: During initial container start, the runtime’s network setup code discovers the veth interface, reads its IP address, and hands that information to the userspace stack. As part of this handoff, the setup code strips the IP from the kernel-level interface. This is intentional: the userspace stack now owns that IP. If the kernel also had it, you would get duplicate ARP responses and conflicting TCP handling.
This works on first boot. The orchestrator’s network plugin configures the veth with an IP and default route before the sandbox starts. The runtime finds the IP, configures the userspace stack, strips the IP. Clean handoff.
When the container is checkpointed, the sandbox process is killed. The veth survives — it is owned by the network namespace, not the process — but the IP is gone. The setup code stripped it during first boot, and nothing re-added it.
When the container is restored, the setup code runs again. It scans for veth interfaces with IP addresses. It finds the interface. No IP. Skips it. The userspace stack receives no network configuration.
The fix: Before invoking the restore, re-add the original IP address and default route to the veth inside the container’s namespace. The setup code finds them, configures the userspace stack, strips the IP again — just like first boot. The key insight: restore-time preconditions are not the same as first-boot preconditions. First boot starts from a state the orchestrator prepared. Restore starts from a state the previous boot’s setup code left behind.
The general lesson: Any initialization code that modifies its own preconditions — reads a value, then deletes it — will fail on the second invocation unless those preconditions are explicitly re-established. This pattern appears everywhere in checkpoint/restore, container migration, and any system where setup code runs more than once against the same environment. If your setup code is destructive (it consumes or modifies the inputs it reads), you must either make it idempotent (detect that it already ran and skip the destructive step) or ensure an external system restores the inputs before the second run. We chose the latter because modifying the runtime’s setup code was not practical — it is upstream code we do not control.
Bug 2 — Stale ARP Cache on the Host
Symptom: Container is restored and reports healthy. The userspace network stack is running with correct IP configuration. But external traffic never reaches the container. Packet captures on the host bridge show packets being sent but never arriving at the container’s veth. The host side shows no packet delivery to the container’s IP.
What happened: When the container was running before hibernation, the host kernel had an ARP cache entry mapping the container’s IP to its MAC address in the REACHABLE state. When the container hibernated and the userspace stack (which handles ARP responses) stopped running, the entry began to age.
The kernel’s ARP state machine transitions entries through REACHABLE, STALE, DELAY, PROBE, and finally FAILED. A FAILED entry means the kernel tried to reach this IP, got no response, and gave up. After restore, the userspace stack is ready to respond to ARP queries. But the kernel does not send new ARP queries for a FAILED entry — it assumes the destination is unreachable and drops packets silently.
The fix: After restore, delete the stale ARP entry for the container’s IP from the host’s ARP table, then actively probe the container with ARP requests until the userspace stack responds. The deletion forces the kernel to treat the next packet as a fresh resolution. As a fallback, if probing fails after a short timeout (sometimes the userspace stack takes a moment to initialize its ARP handler), read the container’s veth MAC address directly and install a static ARP entry on the host bridge.
The general lesson: Any system that caches layer-2 or layer-3 address mappings — ARP tables, IPv6 neighbor discovery caches, DNS resolver caches, service mesh endpoint tables, load balancer health records — will hold stale entries across lifecycle boundaries. The cache does not know that your container hibernated and restored. It only knows that the IP stopped responding and then started again. Depending on timing and cache TTLs, the cache may be in a state where it refuses to re-probe. After any transition that changes a network endpoint’s availability, you must actively invalidate these caches. Do not wait for natural expiry. The expiry timers are designed for normal network conditions, not for orchestrated lifecycle events.
Bug 3 — Userspace Network Stack Resume vs. Restore
Symptom: Container restores from checkpoint. Status shows running. ARP works, raw frames flow. But every TCP connection attempt fails silently — no RST, no ICMP error. Connections are accepted at the IP level but never processed by the TCP state machine. The entire transport layer is frozen.
What happened: The userspace network stack has two distinct code paths for recovering state:
The restore path is designed for checkpoint/restore. It initializes fresh dispatcher threads for each transport protocol (TCP, UDP, ICMP). These dispatchers pull packets off the network layer, demultiplex them by protocol and port, and deliver them to the correct socket. The restore path creates them from scratch.
The resume path is designed for live pause/unpause where the process never dies. It assumes dispatcher threads were merely paused — still alive in memory, blocked on a synchronization primitive. The resume path signals that primitive to unblock the threads.
After checkpoint/restore, the process was killed and reconstructed from serialized state. The dispatcher threads are gone — they were runtime constructs (goroutines, OS threads) that cannot be serialized. The synchronization primitives they were blocked on are nil, marked as non-serializable.
When the resume path is mistakenly invoked after a restore, it tries to signal a nil primitive. In our case, this blocked forever. The transport dispatchers never started. Packets arrived at the network layer, were classified as TCP, and vanished into a dead channel.
A configuration flag controls which path is taken. The default value can cause the wrong path to be selected. This is a one-line change, but finding it required tracing through the runtime’s restore sequence to understand why the TCP dispatcher was deadlocked while the IP layer appeared healthy.
The fix: Ensure the configuration flag is explicitly set so that the restore path is always used after checkpoint/restore.
The general lesson: Pause/unpause and checkpoint/restore are fundamentally different operations, even though both look like “stop the thing, then start it again.” Pause preserves in-memory state: threads, channels, locks, condition variables. The process never dies, it just stops executing. Checkpoint destroys in-memory state: the process is killed, its memory serialized, and a new process created from that serialization. Runtime constructs (threads, goroutines, event loops) are not part of the serialized state and must be recreated.
Any subsystem with both a resume path and a restore path must use the correct one. The symptoms of using the wrong path are insidious: no crash, no error log, just a silent deadlock or a protocol handler that processes zero packets. The system appears healthy at every layer except the one that is stuck. This class of bug is especially dangerous because monitoring systems typically check “is the process alive?” and “is the interface up?” — both of which report true while the transport layer is completely dead.
Bug 4 — Conntrack Confusion on the Second Hibernate Cycle
Symptom: The first hibernate/restore cycle works perfectly. The second cycle: ARP resolves, DNAT rules are in place, the proxy can connect, but traffic through the reverse proxy returns 504 Gateway Timeout. The URL never recovers until the connection tracking table is manually flushed.
What happened: This is the subtlest bug in the series. It took the longest to diagnose because the first cycle always worked.
During the first restore, DNAT rules route external traffic from the host’s published port to the container’s internal port. Connections create conntrack entries that record the NAT translations. Everything works. When the container hibernates again, the DNAT rules are removed (traffic goes to the proxy instead). But conntrack entries from the first session persist — TCP ESTABLISHED entries have five-day timeouts by default.
On the second restore, DNAT rules are re-added. New connections arrive. The kernel’s conntrack subsystem matches incoming packets against existing entries. A new connection from the same client can match a stale entry from the first session. The stale entry contains dead NAT translations. The kernel applies them, the translated packets go nowhere valid, and the client sees a 504.
We initially flushed conntrack entries by filtering on the container’s IP as source or destination. It did not work. The stale DNAT entries survived.
The reason is how conntrack represents DNAT entries. Each entry has an original tuple and a reply tuple:
Original: src=<client_ip> dst=<host_ip> dport=<host_port>
Reply: src=<container_ip> dst=<client_ip> sport=<container_port>The container’s IP does not appear in the original tuple. The original destination is the host’s IP (the client connected to the host; DNAT had not yet rewritten the destination). The container’s IP only appears in the reply tuple’s source field. Our flush searched the original tuple and missed every DNAT entry.
The fix: Flush conntrack entries using multiple criteria: by container IP as direct destination (catches non-DNAT connections), by container IP as direct source (catches outbound connections), by container IP as reply-tuple source (catches DNAT entries — the critical piece we were missing), and by published port number as a catch-all. All four flushes run before DNAT rules are re-added on each restore.
The general lesson: Stateful NAT and connection tracking are the hardest networking components to manage across lifecycle boundaries. Conntrack entries are invisible in normal operation — they do not appear in interface configuration, routing tables, or iptables listings. They silently affect packet routing deep in the netfilter stack. When debugging 504s or failed connections after a lifecycle transition, always check conntrack.
More importantly, when flushing conntrack entries, remember that DNAT creates a fundamental asymmetry between original and reply tuples. The destination IP after DNAT rewriting does not appear where you might expect it. You must query both tuples, and you must understand which fields contain which addresses for each type of NAT rule (SNAT, DNAT, masquerade). Getting this wrong does not produce an error — it produces a flush that silently misses the entries you needed to remove, and the bug returns on the next cycle.
A Pattern for Debugging Networking Across Lifecycle Boundaries
These four bugs share a common shape: a networking layer designed for continuous operation that maintains state across a lifecycle boundary it was never meant to survive. The debugging approach we developed is now standard in our restore procedure.
Instrument each layer independently. Capture the state of interfaces, ARP tables, conntrack entries, iptables rules, and the userspace stack before and after every lifecycle transition. Do not assume that one healthy layer means the others are too. Bug 1 had healthy interfaces but no IP. Bug 2 had healthy IPs but stale ARP. Bug 3 had healthy ARP but dead dispatchers. Bug 4 had healthy everything except invisible conntrack entries.
Compare pre-checkpoint and post-restore snapshots. Diff the captured state. Any discrepancy is a candidate for investigation. The missing IP address and the FAILED ARP entry would both have been immediately visible in a diff.
Test the second cycle. The first cycle often works because the system starts clean. The second cycle is where accumulated state — stale conntrack, cached ARP — causes failures. Always test at least two full cycles. Three is better.
Automate the verification as a health check. After each restore, verify every networking layer before declaring the container ready. Check the veth IP, the host ARP entry, conntrack state, and transport dispatcher status. If any check fails, trigger the repair before allowing the proxy to forward connections.
Takeaways
Network state in a containerized environment is distributed across at least four layers: interface configuration, address resolution caches, connection tracking tables, and the userspace network stack. Checkpoint/restore must address every one of them. Missing any single layer produces failures that manifest as generic “connection refused” or “504 timeout” errors with no obvious cause.
The unifying theme is that Linux networking subsystems are designed for continuous operation. They maintain caches and assumptions that are valid when an endpoint runs without interruption. Checkpoint/restore violates that assumption fundamentally. The endpoint disappears and reappears, but the surrounding infrastructure — ARP caches, conntrack entries, veth configuration — does not reset itself. Every layer that caches state about the endpoint must be explicitly reconciled after restore.
So far in this series, we have treated the container as a standard Linux process. In the next post, we will examine why we run a userspace kernel inside the sandbox in the first place — what a sandboxed container runtime buys you for security, what it costs in operational complexity, and why the networking bugs in this post exist partly because of that architectural choice.
