The Moment the Second GPU Went Dark: Diagnosing a Silent Hardware Underutilization Bug

Introduction

In a complex distributed proving system built for Filecoin's proof-of-replication (PoRep) and other zero-knowledge proof types, few moments are as revealing as when the monitoring tools lay bare a fundamental architectural flaw. Message 428 in this opencode coding session captures precisely such a moment. The assistant, having just deployed a mutex fix to resolve GPU data races on a multi-GPU proving host, runs nvidia-smi to check utilization and discovers that the second RTX A6000 GPU — half the available computational horsepower — is sitting at 0% utilization while the first GPU is pegged at 100%. This single diagnostic command transforms the conversation's understanding of the system's behavior and exposes that the previous "fix" was treating a symptom rather than the root cause.

The Context: A Chain of Fixes Leading to This Discovery

To understand why message 428 was written, one must trace the preceding conversation. The session had been wrestling with partitioned PoRep proofs failing on a multi-GPU remote host (cs-calib). The initial diagnosis identified that the C++ GPU proving code (groth16_cuda.cu) always routed single-circuit proofs to GPU 0, regardless of which Rust worker submitted them. On a system with two GPUs and four workers (two per device), this meant that workers assigned to GPU 1 would still execute their GPU work on GPU 0, creating data races when multiple workers hit the same device simultaneously.

The initial "fix" was a shared mutex that serialized all partition proofs onto GPU 0. This prevented data corruption but effectively wasted the second GPU entirely — all four workers would queue up on GPU 0's single execution path. The assistant had deployed this fix, verified that PoRep proofs passed (20 proofs completed in 30 minutes), and reported success. The user then asked about throughput and whether the second GPU was seeing load, suspecting that "not enough synthesis" might be the bottleneck.

Message 428: The Diagnostic That Changed Everything

The assistant's reasoning in message 428 begins with a careful analysis of worker distribution. The agent checks the journal logs and finds that all four workers are picking up jobs equally — 43 jobs each. This is the first clue that something is wrong with the mental model. If workers 2 and 3 are assigned to GPU 1, and partitioned proofs always run on GPU 0 in the C++ code, then these workers should be competing for the shared mutex and running their GPU work on GPU 0. But the equal distribution suggests the workers are all busy — yet the question remains: where is the actual GPU computation happening?

The agent then articulates the key insight: "When they grab the shared mutex (for partitioned proofs), they still run on GPU 0 because C++ forces it. But this means GPU 1 is effectively idle — the workers assigned to it are still doing work, but the actual GPU computation happens on GPU 0."

This is a critical moment of reasoning. The assistant realizes that the mutex fix only prevents data corruption; it does nothing to address the fundamental misrouting of GPU work. Workers 2 and 3 are consuming CPU cycles for synthesis and witness generation, but when they reach the GPU proving step, they are forced onto GPU 0, where they must wait for the shared mutex alongside workers 0 and 1. The second GPU sits idle because no work is ever dispatched to it.

To confirm this hypothesis, the assistant runs the decisive command:

ssh 10.1.16.218 "nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used --format=csv,noheader,nounits"

The result is stark:

0, NVIDIA RTX A6000, 100, 12986
1, NVIDIA RTX A6000, 0, 698

GPU 0 is at 100% utilization with 12,986 MB of memory used. GPU 1 is at 0% utilization with only 698 MB used — essentially baseline memory for the CUDA runtime with no meaningful computation. The second GPU is completely dark.

Input Knowledge Required to Interpret This Message

Understanding the full significance of message 428 requires several layers of context. First, one must understand the architecture of the CuZK proving system: it uses a pipeline where Rust workers handle circuit synthesis and witness generation, then delegate GPU proving to C++ code via an FFI boundary. The C++ code in groth16_cuda.cu contains the core GPU kernel execution logic.

Second, one must understand the multi-GPU configuration: the system has two NVIDIA RTX A6000 GPUs, with gpu_workers_per_device = 2 meaning four workers total, two assigned to each GPU. The Rust engine assigns workers to GPUs via a gpu_index parameter, but this parameter was not being propagated through the C++ call chain for single-circuit (partitioned) proofs.

Third, one must understand the distinction between partitioned and batched proofs. Partitioned proofs involve a single circuit per proof and are used for PoRep and other proof types. Batched proofs aggregate multiple circuits and can utilize multiple GPUs. The C++ code had a select_gpu() function that was only called for batched proofs; partitioned proofs always defaulted to GPU 0.

Fourth, one must understand the shared mutex fix that was just deployed. This fix added a mutex that serialized access to the GPU proving code, preventing data races when multiple workers hit GPU 0 simultaneously. But it did nothing to route work to GPU 1.

The Assumptions That Were Challenged

Message 428 exposes several assumptions that had been operating beneath the surface of the conversation. The first assumption was that the mutex fix was sufficient — that serializing access to GPU 0 would solve the problem. While it did prevent proof corruption, it left half the computational resources untapped.

The second assumption was that workers assigned to GPU 1 would naturally drive work to GPU 1. The equal distribution of job pickups (43 each) suggested the system was load-balanced at the worker level, but this masked the fact that the GPU computation was not following the worker assignment.

The third assumption, implicit in the user's question about "not enough synthesis," was that the bottleneck might be CPU-side synthesis rather than GPU routing. The user suspected that synthesis capacity was insufficient to keep both GPUs busy. The diagnostic revealed the opposite: synthesis was keeping all four workers busy, but the GPU work was being funneled to a single device.

The Mistake That Became Visible

The shared mutex fix, while well-intentioned, was a mistake in the sense that it treated the wrong layer of the problem. The root cause was not that multiple workers were accessing the GPU without synchronization — it was that work was being routed to the wrong GPU entirely. The mutex fix was a band-aid that prevented crashes but preserved the fundamental misrouting.

The assistant's earlier reasoning had identified that "C++ code always runs single-circuit proofs on GPU 0 regardless of which Rust worker submits them," but the fix addressed only the synchronization aspect, not the routing aspect. Message 428 represents the moment this inadequacy becomes undeniable, because the utilization data provides an objective measure of the problem's severity.

Output Knowledge Created

Message 428 produces several concrete pieces of knowledge. First, it establishes a clear, quantitative baseline: GPU 0 at 100% utilization, GPU 1 at 0%. This is an unambiguous measurement that any future fix must improve upon.

Second, it confirms that the worker distribution is balanced at the job-pickup level (43 jobs per worker), which means the scheduling logic in the Rust engine is functioning correctly. The bottleneck is downstream of the scheduler, in the C++ GPU dispatch layer.

Third, it narrows the search space for the fix. The problem is specifically in the C++ groth16_cuda.cu code path for single-circuit proofs, which does not respect the gpu_index parameter. The fix must involve threading gpu_index through the call chain so that the C++ code calls select_gpu(gpu_index) for partitioned proofs, just as it does for batched proofs.

Fourth, it provides a clear success criterion for the eventual fix: both GPUs should show non-zero utilization under load, and the memory usage on GPU 1 should rise from 698 MB to a level comparable to GPU 1's active state.

The Thinking Process: From Observation to Insight

The reasoning in message 428 follows a clear diagnostic pattern. The agent begins with an observation from the logs: all four workers are picking up jobs equally (43 each). This is the initial data point.

The agent then connects this observation to the known architecture: workers 2 and 3 are assigned to GPU 1, but partitioned proofs always run on GPU 0. This creates a contradiction: if workers 2 and 3 are running their GPU work on GPU 0, then GPU 1 should be idle. But the equal job distribution suggests all workers are busy — so where is the GPU work happening?

The agent then articulates the resolution of this apparent contradiction: the workers on GPU 1 are doing CPU work (synthesis, witness generation) but their GPU proving step is routed to GPU 0, where they must wait for the shared mutex. This means GPU 1 is idle despite its workers being busy.

The final step is to confirm this hypothesis with direct measurement. The nvidia-smi command provides the definitive evidence. The agent doesn't stop at the utilization numbers — it also notes the memory usage (698 MB on GPU 1 vs 12,986 MB on GPU 0), which provides additional confirmation that no meaningful computation is happening on GPU 1.

The Broader Significance

Message 428 is a textbook example of why monitoring and observability are critical in distributed systems debugging. The proof results were passing (20/20 successful), the logs showed balanced worker activity, and the system appeared to be functioning correctly. Only the GPU utilization data revealed the silent underutilization.

This message also illustrates the danger of fixing symptoms rather than root causes. The shared mutex fix was a reasonable response to the immediate problem (proof corruption), but it created a new problem (GPU underutilization) that was invisible until the right metric was checked. The user's intuition — asking whether the second GPU was seeing load — was what prompted the diagnostic that revealed the deeper issue.

For the assistant, message 428 represents a pivot point. The conversation moves from "the fix is working, proofs pass" to "the fix is incomplete, we're wasting half our hardware." The subsequent chunks in this segment describe the proper fix: threading gpu_index through the entire call chain from Rust engine to C++ GPU code, so that partitioned proofs respect the worker's GPU assignment. This is the architectural change that the diagnostic in message 428 demands.

Conclusion

Message 428 captures a moment of diagnostic clarity in a complex debugging session. By combining log analysis with direct hardware measurement, the assistant identifies that a recently deployed fix, while preventing proof corruption, has left a second GPU completely idle. The message demonstrates the importance of measuring the right metrics, the danger of treating symptoms, and the value of user intuition in guiding investigation. The stark numbers — 100% vs 0% utilization — provide an unambiguous target for the next phase of work, transforming the conversation from validation to fundamental architectural repair.