The Moment of the Fix: Replacing wait -n with a Polling Loop in Production

"Now I'll replace wait -n with a reliable polling loop:" [edit] /tmp/czk/docker/cuzk/entrypoint.sh Edit applied successfully.

These three lines — a statement of intent, a file path, and a confirmation — represent the culmination of one of the most consequential debugging sessions in the entire opencode conversation. The message at <msg id=4351> is deceptively brief. On its surface, it is a simple edit operation: the assistant modifies a shell script. But this single edit was the resolution of a critical production outage that had silently crippled four out of six GPU proving nodes, leaving them with dead daemons and a supervisor loop that was supposed to restart them but instead sat frozen in kernel space, waiting for a signal that would never arrive.

To understand why this message was written, one must first understand the scale of the failure it was meant to address.

The Production Crisis

The cuzk proving infrastructure consisted of six vast.ai instances running a Docker image that launched two core processes: cuzk (the GPU proving daemon) and curio (the Filecoin storage/retrieval daemon). These were supervised by an entrypoint.sh script that used a while true loop to monitor both processes and restart them if they crashed. The supervisor loop relied on bash's wait -n builtin — a command that blocks until any one of a specified set of background processes exits, returning that process's PID and exit code.

By the time the assistant began investigating, the situation was dire. Four of the six nodes had dead cuzk processes. The RTX 5090 test node's cuzk had been dead since March 13. The RTX 4090 node had lost its cuzk on March 15 at 20:19. The RTX PRO 4000 node lost its cuzk at 21:03 the same day. The RTX 5090 "new2" node lost its cuzk at 20:37. In every case, curio was still running, the entrypoint script was still alive — but cuzk was gone, and the supervisor had not restarted it. The fleet was operating at one-third capacity, and the restart mechanism was silently failing.

The Diagnostic Journey

The assistant's reasoning in the messages leading up to <msg id=4351> reveals a meticulous, hypothesis-driven debugging process. The investigation began with a straightforward question: why isn't the supervisor restarting cuzk?

The first hypothesis was set -e. The entrypoint script had set -euo pipefail at its top, which means any command returning a non-zero exit code would cause the entire script to exit. If cuzk crashed with a signal (SIGSEGV returning 139, SIGABRT returning 134), wait -n would return that non-zero code, and set -e would terminate the script. But this hypothesis collapsed under scrutiny: the entrypoint PID was still alive on all crashed nodes. If set -e had triggered, the script would have exited entirely.

The second hypothesis involved bash's SIGCHLD handling. Perhaps bash had reaped the dead cuzk process asynchronously through its SIGCHLD handler before wait -n was called, and by the time wait -n checked, the exit status was no longer available. This is a known edge case in bash's job control: when a background process exits, bash's signal handler collects the exit status immediately, and wait -n may not find it if it wasn't the one to reap the child.

The third hypothesis — the one that proved correct — emerged from examining the actual process state. The assistant SSHed into a crashed node and checked /proc/504/wchan, which revealed that the entrypoint process was stuck in do_wait — the kernel's wait syscall. This meant the entrypoint had called wait -n and was blocking indefinitely, even though cuzk was fully dead (PID gone, no zombie process). The wait -n builtin was not returning at all.

This is a known reliability issue in bash's wait -n implementation. When wait -n is called with specific PIDs, bash internally calls waitpid() to check for exited children. But there is a race condition: if the child process exited and was reaped by bash's SIGCHLD handler before wait -n was invoked, the kernel's waitpid() call may not find the exited child, and wait -n will block waiting for the remaining children — which are still running. The process that already died is simply forgotten.

The Decision to Replace with a Polling Loop

The assistant's decision to replace wait -n with a polling loop using kill -0 was a pragmatic engineering choice. The kill -0 approach sends signal 0 to a process — it doesn't actually deliver a signal, but returns zero if the process exists and is accessible, and non-zero if the process is gone. By wrapping this in a loop with a short sleep interval, the supervisor can reliably detect when either process has died, without relying on bash's fragile job tracking.

The trade-off is clear: a polling loop is less elegant than wait -n. It consumes a small amount of CPU periodically, and it introduces a latency between process death and detection equal to the sleep interval. But in a production system where reliability is paramount, correctness trumps elegance. A supervisor that misses a crash entirely is worse than one that detects it a few seconds late.

Assumptions and Their Validity

The fix rested on several assumptions. First, that the cuzk crashes themselves were not caused by the supervisor loop — they were independent failures (likely GPU driver faults or CUDA errors, as the daemon logs ended abruptly mid-synthesis with no panic message or OOM kill). Second, that the polling loop would not introduce new failure modes — for instance, that kill -0 itself would not hang on a zombie process or a process in an unrecoverable state. Third, that the fix could be deployed by rebuilding the Docker image and restarting the nodes, without requiring changes to the process architecture itself.

These assumptions were reasonable given the evidence. The daemon logs showed no panics, no CUDA errors, and no OOM kills — the processes simply vanished. The kill -0 approach is a well-established pattern in shell scripting for process health checks. And the Docker-based deployment meant the fix could be rolled out to all nodes simultaneously.

The Broader Significance

This message at <msg id=4351> is a case study in the fragility of shell-based process supervision in production GPU workloads. The wait -n builtin is a standard tool for coordinating background processes, but its reliability depends on subtle details of bash's internal job tracking — details that can fail in edge cases like a process dying between SIGCHLD handling and the wait call. In a system managing expensive GPU hardware across multiple cloud instances, such edge cases are not theoretical: they are the difference between a fleet running at full capacity and one silently losing 66% of its proving power.

The fix itself — a polling loop — is not glamorous. It is the kind of change that a developer makes reluctantly, knowing it adds a small but constant overhead. But it is also the kind of change that demonstrates a deep understanding of the system's failure modes. The assistant did not just patch the symptom; it diagnosed the exact mechanism by which the supervisor failed, traced it to a specific bash behavior, and implemented a replacement that eliminated the root cause.

Conclusion

The message at <msg id=4351> is a turning point in the conversation. It marks the moment when a critical production outage was definitively resolved, not through a complex architectural overhaul but through a targeted replacement of an unreliable primitive. The assistant's reasoning in the preceding messages — the SSH investigations, the /proc inspections, the careful elimination of false hypotheses — demonstrates the kind of systems-level debugging that is essential for maintaining distributed GPU infrastructure. And the fix itself, for all its simplicity, restored the fleet's ability to self-heal, turning four dead nodes into four nodes that would automatically recover from future crashes.