When Zombies Refuse to Die: A Deployment Debugging Lesson

"Port 9820 and 9821 are still bound by the zombie process."

This single line, from message [msg 2701] in an opencode coding session, captures a moment of frustration familiar to every systems engineer: the moment when a process that should be dead refuses to release its grip on system resources. The assistant, deep in a deployment workflow for the CuZK zero-knowledge proving system, had just spent several rounds wrestling with a stubborn zombie process on a remote test machine (141.0.85.211). This message represents the pivot point where the assistant shifts from gentle process termination to aggressive, port-level force.

The Broader Mission

To understand why this message matters, we must first understand what was being deployed. The assistant was in the final stages of segment 20 of a long-running development session, focused on deploying and refining a real-time monitoring panel for the CuZK proving pipeline. Two critical fixes were being shipped:

  1. A GPU worker race condition fix in status.rs: The partition_gpu_end function unconditionally cleared a worker's busy state, even when the worker had already picked up a new job. This caused GPU workers to appear permanently idle in the monitoring UI, because a stale finalizer task from a completed job would reset the worker's state after it had already started new work. The fix added a guard to only clear the worker if it was still assigned to the same job and partition.
  2. A job ID truncation fix in ui.html: The monitoring panel truncated job IDs to 8 characters, which for IDs starting with ps-snap- resulted in the unhelpful display ps-snap- — cutting off right at the hyphen, making the label appear broken. The fix increased the substring to 16 characters. These fixes had been built successfully using Docker (for the Go-based vast-manager) and a Docker rebuild pipeline (for the Rust-based cuzk daemon). The vast-manager had been deployed and restarted without incident. But the cuzk binary deployment to the remote test machine had hit a wall.

The Zombie That Wouldn't Die

The sequence of events leading to message [msg 2701] reveals a classic systems debugging spiral. The assistant first attempted a simple pkill -f cuzk-daemon to stop the old daemon, but this failed because the actual process name was cuzk, not cuzk-daemon ([msg 2693]). A direct kill 40513 turned the process into a zombie — marked <defunct> in the process table ([msg 2696]). Even kill -9 couldn't eliminate it; zombies are already dead, they simply await reaping by their parent process ([msg 2697]).

The zombie's continued existence blocked the binary replacement: the kernel refused to overwrite /usr/local/bin/cuzk because the zombie still had the file mapped, returning the infamous "Text file busy" error. The assistant cleverly worked around this with a rename trick — deleting the old binary and copying the new one in its place ([msg 2698]). This succeeded because unlinking a file doesn't require the file to be unmapped; the zombie retained its mapping to the now-deleted inode, while the new binary occupied a fresh inode at the same path.

But the victory was short-lived. When the assistant started the new daemon and checked its status, the HTTP endpoint returned an empty response ([msg 2699]). The daemon had failed to bind to its ports because the zombie process, though technically dead, still held port 9820 and 9821 open. This is the critical insight that message [msg 2701] addresses.

The Message Itself: Reasoning and Decisions

The subject message reads:

Port 9820 and 9821 are still bound by the zombie process. Let me kill all cuzk processes more aggressively:

>

``bash ssh -p 40612 root@141.0.85.211 'kill -9 40513 2>/dev/null; sleep 1; fuser -k 9820/tcp 2>/dev/null; fuser -k 9821/tcp 2>/dev/null; sleep 2; ss -tlnp | grep 982' 2>&1 ``

>

`` bash: line 1: ss: command not found ``

The assistant's reasoning is visible in the opening sentence: it has correctly diagnosed that the zombie process is still holding the ports. This is a subtle point — many developers assume that a zombie process is completely inert, but a zombie retains its open file descriptors (including network sockets) until its parent reaps it via wait(). The ports remain bound, preventing any new process from listening on them.

The decision to use fuser -k is a tactical shift. Previous attempts targeted the process by PID (kill, kill -9) or by process name (pkill). Neither approach worked because the zombie was immune to signals. fuser -k takes a different approach: it identifies which processes have the specified port open and sends them a signal. This is more robust because it targets the resource (the port) rather than the process identity. Even if the zombie's PID had been reaped and recycled, fuser would find whatever process currently held the port.

The command also includes a final ss -tlnp | grep 982 to verify the ports are free — a sensible verification step. But this reveals an incorrect assumption: the assistant assumed that ss (a modern replacement for netstat) would be available on the remote machine. It was not, resulting in the anticlimactic "command not found" error.

The Assumptions at Play

Several assumptions underpin this message, and examining them reveals the assistant's mental model:

Assumption 1: The zombie is the problem. The assistant assumes that the zombie process (PID 40513, marked <defunct>) is what's holding the ports. This is likely correct — the zombie was the original cuzk daemon that had bound ports 9820 and 9821. However, there's a subtle possibility: the failed startup attempt from message [msg 2699] might have left a half-initialized process that also tried to bind. The assistant doesn't consider this alternative.

Assumption 2: fuser will be available. The assistant assumes that the remote machine has fuser installed (part of the psmisc package on Linux). This assumption proves incorrect, though it's a reasonable one for a server environment.

Assumption 3: ss will be available. Similarly, the assistant assumes ss (from the iproute2 package) is present. It is not, leaving the assistant without a way to verify port availability.

Assumption 4: Aggressive killing will work. The assistant assumes that fuser -k can kill the zombie by targeting its ports. In practice, fuser -k sends SIGKILL to the process holding the port — but zombies cannot be killed, so this would fail silently. The assistant doesn't account for this edge case.

Input Knowledge Required

To fully understand this message, a reader needs knowledge of:

Output Knowledge Created

This message, despite its brevity and apparent failure, creates several pieces of valuable knowledge:

  1. Confirmation that the zombie holds the ports: The assistant's diagnosis that the zombie is blocking ports 9820/9821 is confirmed by the failed startup. This is a useful data point for debugging the deployment.
  2. Tool availability on the remote machine: The "command not found" errors for both fuser and ss reveal that the remote machine has a minimal toolset. This is actionable knowledge for future debugging sessions — the assistant should use netstat or /proc/net/tcp instead.
  3. The limits of aggressive killing: The message demonstrates that zombie processes require parent reaping, not signal delivery. This is a lesson that informs future debugging strategies.
  4. A failed hypothesis: The message represents a hypothesis ("killing by port will work") that is tested and found to be unverifiable due to tooling gaps. This negative result is still valuable — it narrows the search space.

The Thinking Process

The assistant's thinking process, visible in the transition from message [msg 2699] to [msg 2701], follows a clear diagnostic chain:

  1. Observe failure: The new daemon started but the status endpoint returned empty. The assistant checks the log ([msg 2700]) but the log is truncated — we can't see the bind error directly.
  2. Form hypothesis: The assistant infers that the zombie process is still holding the ports. This is a sophisticated inference — it requires understanding that zombies retain file descriptors and that the failed bind() would cause the daemon to either crash or fail to register its HTTP listener.
  3. Design experiment: The assistant constructs a command that will (a) attempt to kill the zombie by PID one more time, (b) use fuser to kill by port as a backup, and (c) verify with ss that the ports are free.
  4. Execute and interpret: The command fails because ss is not available. But the assistant doesn't stop here — in the next message ([msg 2702]), it tries netstat instead, and when that also fails, it checks /proc/net/tcp or simply retries the startup. The thinking is methodical and hypothesis-driven, characteristic of experienced systems debugging. Each failure narrows the possibilities and informs the next attempt.

The Resolution

While message [msg 2701] itself ends in failure, the subsequent messages show that the assistant successfully resolved the issue. By message [msg 2703], the ports are free and the daemon starts successfully, reporting "OK uptime=5s workers=2 w0=idle." The zombie was presumably reaped by its parent process (the shell or init system that spawned it) after some delay, or the assistant found another way to clear the ports.

The successful startup then leads to a proof benchmark ([msg 2704]) and a status check 60 seconds later ([msg 2705]) showing active synthesis across 10+ partitions. The GPU worker fix is working — workers show as "synthesizing" rather than permanently "idle."

Broader Lessons

This message, though small, encapsulates several important lessons about systems engineering:

Zombie processes are not harmless. A common misconception is that zombie processes are inert entries in the process table that consume negligible resources. While they don't use CPU or memory, they retain file descriptors, including network sockets. A zombie can block port binding, file deletion, and other operations that depend on file descriptor closure.

Signal delivery has limits. Signals cannot wake a zombie because zombies are not schedulable — they have already exited and are waiting only for their parent to call wait(). The only way to eliminate a zombie is to either kill its parent (making it an orphan, which is then reaped by init) or wait for the parent to reap it.

Tool availability is not guaranteed. In production debugging, the tools you expect may not be installed. The assistant's assumption that fuser and ss would be available on the remote machine proved incorrect. A more robust approach would check for tool availability first, or use universally available interfaces like /proc/net/tcp.

Deployment pipelines must handle process lifecycle. The assistant's deployment workflow lacked a robust mechanism for stopping the old daemon before replacing the binary. A production deployment system would use proper service management (systemd, supervisor) with guaranteed stop and restart semantics, avoiding the manual kill-and-copy dance that led to this situation.

Conclusion

Message [msg 2701] is a small but revealing window into the reality of systems debugging. It shows a skilled practitioner working through a thorny deployment problem, forming hypotheses, testing them, and adapting when tools fail. The zombie process that "wouldn't die" is a classic systems adversary — technically dead but still causing trouble, immune to the usual tools of process termination. The message captures the moment of pivot, when the assistant realizes that a different approach is needed and reaches for a more aggressive tool. The "command not found" error that follows is not a failure but a data point — one more piece of information that narrows the search and brings the assistant closer to a working deployment.

In the end, the fix was deployed, the GPU worker status display worked correctly, and the monitoring panel showed live proof data. But the journey to get there, captured in this single message about a zombie and its ports, is where the real engineering happens.