The do_wait Trap: Debugging a Bash Supervisor Failure in Production GPU Infrastructure

Introduction

In the high-stakes world of distributed GPU proving infrastructure, where machines cost hundreds of dollars per day and proofs must be generated continuously to maintain blockchain consensus, process supervision is not a theoretical concern—it is a critical operational requirement. When a daemon crashes, the supervisor must detect the failure and restart it within seconds. If it does not, proving capacity degrades, deadlines are missed, and real money is lost.

This article examines a single message in a coding session that diagnosed and resolved exactly such a failure. The message, indexed as <msg id=4349> in the conversation, represents the culmination of a deep systems debugging exercise spanning multiple production nodes. The assistant had been investigating why the cuzk proving daemon was crashing silently across a fleet of GPU instances, and—more critically—why the bash-based supervisor loop in the container entrypoint was failing to restart it. The answer lay not in application-level error handling, but in the subtle and often misunderstood behavior of bash's wait -n builtin, the kernel's do_wait syscall, and the interaction between process reaping and shell job control.

This message is the moment of diagnosis: the assistant identifies the root cause, articulates it with precision, and proposes a concrete fix. It is a masterclass in systems-level debugging, combining process inspection, kernel state analysis, shell behavior reasoning, and production fleet observability into a coherent narrative.

Context: The Production Crash

The conversation leading up to <msg id=4349> had been focused on deploying a budget-integrated pinned memory pool—a critical optimization for GPU proving workloads. The user had directed the assistant to test this pool on a constrained memory machine before committing the code. A new Docker image was built and pushed, and test instances were provisioned on vast.ai, a GPU cloud marketplace.

But the focus shifted dramatically when multiple production nodes began reporting failures. The cuzk daemon—the core proving engine—was crashing, and it was not coming back. Across a fleet of six nodes, four were effectively dead: the cuzk process had exited, but the supervisor loop in the entrypoint script had not restarted it. Only curio, the Filecoin storage mining companion process, remained alive on these nodes. One node was healthy, and one was inaccessible.

The assistant's investigation in <msg id=4346> and <msg id=4347> had already established the key facts. The entrypoint script used a supervisor loop structured like this:

while true; do
    run.sh &
    CUZK_PID=$!
    # ... wait for cuzk to be ready ...
    curio run &
    CURIO_PID=$!
    wait -n "$CUZK_PID" "$CURIO_PID" 2>/dev/null
    # ... restart logic ...
done

The wait -n call was supposed to block until either cuzk or curio exited, then return so the loop could restart the dead process. But on the crashed nodes, the entrypoint process was still alive—stuck in a do_wait kernel state—while cuzk was long gone. The restart logic was never executing.

The Diagnostic Evidence

Message <msg id=4349> opens with the assistant summarizing the evidence gathered from a direct SSH inspection of one of the crashed nodes (instance 32915747, an RTX PRO 4000 machine):

So: - Entrypoint PID 504 is in do_wait state — it's stuck in a wait system call - PID 6304 (cuzk) is completely gone (not even a zombie) - No zombie processes - Bash 5.2.21

This is a remarkable set of facts. The entrypoint process (PID 504) is alive and in a do_wait kernel sleep, meaning it has called a wait family syscall and is blocked waiting for a child process to change state. But the child it was waiting for—cuzk at PID 6304—is not just dead; it is gone. Not even a zombie entry remains in the process table. The process has been fully reaped.

In normal operation, when a child process exits, it becomes a zombie briefly until the parent calls wait() to collect its exit status, at which point the zombie is reaped and the PID is freed. If the parent is already in a wait() call, the kernel should wake it immediately when the child dies. The fact that PID 504 is still sleeping in do_wait despite PID 6304 being fully reaped is the central mystery.

The assistant immediately recognizes the implications:

The entrypoint is stuck in wait. Since PID 6304 is gone and there are no zombies, bash should have gotten the SIGCHLD notification and wait -n should return. But it's stuck.

This is the crux of the bug. The kernel delivered SIGCHLD, bash's signal handler collected the exit status, the zombie was reaped—but wait -n did not return. The supervisor loop is frozen, and the proving daemon will never be restarted.

The Root Cause: Bash's wait -n Reliability Bug

The assistant's reasoning in <msg id=4349> walks through several hypotheses before arriving at the correct diagnosis. Let me quote the key reasoning section in full:

Wait, I think I found it. Look at wait -n "$CUZK_PID" "$CURIO_PID" 2>/dev/null:

>

The issue is that CUZK_PID (6304) died and was reaped, while CURIO_PID (6565) is still running. In bash 5.2, wait -n with specific PIDs should return when any of them terminates, but there's a catch — bash's built-in wait can only track processes it started. Since run.sh used exec cuzk, the PID should be the same one the entrypoint launched, so that's not the problem. But there's something else going on when wait -n is called with multiple PIDs where one has already been reaped.

>

The real issue might be that bash reaped PID 6304 asynchronously through its SIGCHLD handler before wait -n was even called. By the time wait -n checks the status, the process is already gone and fully reaped, so bash might not have the exit status readily available anymore.

This is a subtle point about bash's internal architecture. Bash installs a SIGCHLD handler that reaps child processes asynchronously. When a child dies, the kernel sends SIGCHLD to bash, the handler runs, collects the exit status via waitpid(), and marks the job as completed in bash's internal job table. The zombie is reaped immediately.

The problem arises when wait -n is called after the SIGCHLD handler has already processed the child's exit. In some versions of bash, wait -n with explicit PID arguments checks its internal job table for the specified PIDs. If a PID's job has already been marked completed by the SIGCHLD handler, wait -n may not recognize it as a pending child and instead blocks waiting for the remaining PIDs. This is a race condition between the asynchronous SIGCHLD handler and the synchronous wait -n call.

The assistant considers and rejects several alternative explanations:

The real issue is that the entrypoint is stuck in do_wait blocking on a wait -n call, probably because bash's job table lost track of one of the PIDs after it exited.

The Fleet-Wide Impact

After diagnosing the root cause, the assistant presents a summary table of all six nodes in the fleet:

| Instance | GPU | cuzk | curio | Status | |---|---|---|---|---| | 32790145 | RTX 5090 test | DEAD (since Mar 13) | alive | Supervisor stuck | | 32874928 | RTX 4090 | DEAD (since Mar 15 20:19) | alive | Supervisor stuck | | 32854594 | A40 | can't SSH | can't SSH | Unknown | | 32915747 | RTX PRO 4000 | DEAD (since Mar 15 21:03) | alive | Supervisor stuck | | 32919934 | RTX 5090 new | alive | alive | HEALTHY | | 32919936 | RTX 5090 new | DEAD (since Mar 15 20:37) | alive | Supervisor stuck |

Four of six nodes are dead. The cuzk daemon crashed at various times over the preceding days, and in every case, the supervisor loop failed to restart it. Only one node remained healthy—possibly because cuzk had not yet crashed on it, or because a different bash version or timing condition allowed wait -n to work correctly.

The assistant summarizes the root cause in two crisp bullet points:

Root cause of no-restart: wait -n "$CUZK_PID" "$CURIO_PID" (entrypoint.sh:366) blocks forever. Bash 5.2's wait -n fails to notice the reaped child. The supervisor loop never executes its restart logic.

>

Root cause of cuzk dying: No OOM in dmesg. The daemon logs end abruptly mid-synthesis — likely a panic or CUDA fatal error during GPU operations. Need to check if there's a signal or core dump.

The second point acknowledges that there are actually two problems: why cuzk crashes, and why the supervisor doesn't restart it. The assistant correctly prioritizes the supervisor fix, because even if the crash cause is identified and fixed, the supervisor must be reliable enough to handle unexpected failures. In production systems, processes will crash—the question is whether the system recovers.

The Fix: Replacing wait -n with a Polling Loop

The assistant's proposed fix is pragmatic and robust:

I'll replace wait -n with a polling loop that periodically checks if each process is still alive using kill signals, which is more reliable even if less elegant.

The kill -0 approach is a well-known pattern in robust shell scripting. The kill command with signal 0 does not send a signal; it only checks whether the process exists and is accessible. By polling both PIDs in a loop with a short sleep interval, the supervisor can reliably detect when either process exits, regardless of bash's internal job tracking behavior.

This fix trades elegance for reliability. The wait -n approach is cleaner—it blocks efficiently without consuming CPU, and it returns immediately when a process exits. But it depends on bash's internal job management, which has proven unreliable in this scenario. The polling loop is slightly less efficient (it wakes up periodically to check) but is immune to the SIGCHLD race condition that broke wait -n.

Assumptions and Knowledge Required

To fully understand <msg id=4349>, the reader needs knowledge in several domains:

Linux process lifecycle: Understanding the distinction between a running process, a zombie (exited but not yet reaped by parent), and a fully reaped process (exit status collected, PID freed). The assistant's observation that PID 6304 is "completely gone (not even a zombie)" is a key diagnostic clue.

Kernel wait syscalls: The do_wait state visible in /proc/PID/wchan indicates that the process is blocked in a kernel wait family syscall. This is the kernel-level manifestation of bash's wait builtin.

Bash job control: Bash maintains an internal job table tracking background processes. The wait builtin interacts with this table, and its behavior depends on bash version and compilation options. The SIGCHLD handler asynchronously reaps children and updates the job table.

Shell scripting with set -e: The set -e option causes the shell to exit on any command failure, but with numerous exceptions that vary across shells and versions. Understanding when set -e does and does not apply is crucial for writing robust supervisor scripts.

GPU proving infrastructure: The context of Filecoin proof generation, cuzk daemon, curio, and vast.ai GPU cloud instances provides the operational backdrop. While not strictly necessary for understanding the bug, it explains why this failure is costly.

The Thinking Process

The reasoning in <msg id=4349> is notable for its systematic elimination of hypotheses. The assistant walks through:

  1. Observation: Entrypoint in do_wait, cuzk PID gone, no zombies.
  2. Hypothesis 1: set -e caused the script to exit when wait -n returned non-zero. Rejected because the entrypoint is still alive.
  3. Hypothesis 2: exec in run.sh confused bash's PID tracking. Rejected because exec preserves the PID.
  4. Hypothesis 3: Zombie process not being reaped. Rejected because no zombies found.
  5. Hypothesis 4: SIGCHLD handler reaped the child before wait -n could process it. Accepted as the most likely explanation. This is textbook debugging methodology: generate hypotheses, test each against the evidence, and converge on the explanation that fits all observed facts. The assistant also demonstrates excellent use of available diagnostic tools. The /proc/PID/wchan file reveals the kernel wait state. The /proc/PID/stack file (though empty in this case) could show the kernel call stack. The process file descriptor list confirms that stdout/stderr are piped to the log shipper. These are standard Linux process introspection tools that every systems engineer should know.

Output Knowledge Created

This message creates several pieces of valuable output knowledge:

  1. A confirmed bug pattern: Bash 5.2.21's wait -n with explicit PIDs can hang indefinitely when one of the specified processes has already been reaped by the SIGCHLD handler. This is a specific, reproducible failure mode that can inform similar debugging efforts elsewhere.
  2. A diagnostic procedure: The sequence of checks—verify process existence, check for zombies, inspect kernel wait state, examine bash version—forms a repeatable procedure for debugging supervisor failures.
  3. A concrete fix: Replacing wait -n with a kill -0 polling loop is a robust alternative that avoids the race condition entirely.
  4. Fleet status awareness: The summary table provides immediate operational awareness of which nodes are healthy and which are dead, enabling rapid manual recovery.

Mistakes and Incorrect Assumptions

The assistant makes one notable assumption that may be incorrect: that cuzk is crashing due to "a panic or CUDA fatal error during GPU operations." While this is plausible—GPU workloads are notoriously prone to driver crashes, memory errors, and CUDA runtime failures—it is an assumption without direct evidence. The daemon logs end abruptly, but no crash dump, core file, or signal information has been collected. The assistant acknowledges this gap ("Need to check if there's a signal or core dump") but does not pursue it in this message.

This is a reasonable prioritization. Fixing the supervisor loop restores automatic recovery regardless of why cuzk crashes. The crash cause can be investigated separately. However, if cuzk is crashing due to a bug in the newly deployed budget-integrated pinned memory pool, fixing the supervisor alone might mask a deeper problem. The assistant implicitly recognizes this by noting both root causes separately.

Broader Implications

The wait -n reliability bug has implications beyond this specific deployment. Any bash-based process supervisor that uses wait -n with explicit PIDs could be vulnerable to the same failure mode. This includes:

Conclusion

Message <msg id=4349> is a remarkable piece of systems debugging. In a single message, the assistant: