The Phantom GPU: Debugging a Race Condition That Only Exists on Two GPUs
Introduction
In the landscape of high-performance zero-knowledge proof systems, few debugging challenges are as disorienting as a bug that disappears on your development machine but strikes with 100% certainty in production. This is the story of such a bug — a GPU race condition in the CuZK proving engine that caused every single PoRep partitioned proof to fail on a multi-GPU remote host, while the exact same binary ran flawlessly on a single-GPU development machine.
This article synthesizes the second segment of an opencode coding session, spanning messages 274 through 376. The investigation began as a routine deployment of Pre-Compiled Constraint Evaluator (PCE) extraction for all proof types — WinningPoSt, WindowPoSt, SnapDeals, and PoRep. After fixing a WindowPoSt crash caused by a constraint system type mismatch (covered in [segment 0]), the team deployed the updated build to a remote test host at 10.1.16.218 — a machine equipped with two NVIDIA RTX 4000 Ada GPUs. What followed was a methodical hunt for a race condition that would ultimately reveal a fundamental misunderstanding about how CUDA_VISIBLE_DEVICES interacts with static initialization in mixed-language systems.
The narrative arc moves through four distinct phases: (1) initial suspicion of the recently modified PCE code, (2) a decisive experiment that exonerated PCE and redirected the investigation, (3) local validation confirming the code was not fundamentally broken, and (4) the discovery of the root cause — a mutex mismatch caused by ineffective CUDA_VISIBLE_DEVICES manipulation — culminating in a clean, elegant fix.
Phase 1: The PCE Suspect — A Reasonable but Wrong Hypothesis
Every debugging journey begins with a hypothesis, and the most natural one is often the most misleading. The team had just modified WitnessCS::new() and RecordingCS::new() as part of the WindowPoSt fix — changing both constructors from pre-allocating one input to starting empty, with explicit alloc_input("one") calls added to both extract_precompiled_circuit() and synthesize_with_pce(). This was a delicate change: if the PCE file cached on disk had been extracted with the old initialization (1 input), but the new witness generation code used the new initialization (0 inputs + explicit alloc_input), the input counts would be off by one, producing garbage proofs.
The remote host logs told a devastating story. Every single PoRep partitioned proof was failing verification — a 100% failure rate, with 0 out of 10 partitions valid in every proof attempt. The pattern was unmistakably non-deterministic: different runs produced different sets of valid and invalid partitions, sometimes 0/10, sometimes 2/10, sometimes 8/10. This random variation was the classic hallmark of a GPU-level data race.
The assistant's initial suspicion fell on the recently modified PCE code path. After all, the WitnessCS::new() and RecordingCS::new() constructors had been changed as part of the WindowPoSt fix, and the failures appeared immediately after deployment. The assistant spent significant effort investigating this hypothesis — checking PCE file timestamps on the remote host, examining journal logs to determine when PCE extraction occurred, tracing through the evaluate_pce function to understand where the assert_eq! on input counts would catch a mismatch, and even considering whether the PCE was loaded from memory from an even earlier daemon instance.
The investigation was thorough but time-consuming. Each new log entry or file timestamp required interpretation, and the chain of reasoning grew increasingly tangled as the assistant tried to reconstruct the sequence of daemon restarts, PCE extractions, and proof submissions. The assistant examined the PCE code path in detail: the evaluate_pce function that computes a = A*w, b = B*w, c = C*w via CSR sparse MatVec; the RecordingCS::enforce method that captures constraint structure; the into_precompiled function that builds the CSR matrices; and the density bitmap computation. Each component was scrutinized for correctness. The analysis suggested the code was logically sound, but the assistant recognized that theoretical correctness does not guarantee practical correctness — especially in a system as complex as a GPU-accelerated zk-SNARK prover.
The logs revealed a critical clue: the PCE file on disk (/data/zk/params/pce-porep-32g.bin, 27.6 GB) had been written at 10:54 — after the service restarted with the new code at 10:51. This meant the PCE was extracted with the new RecordingCS (0 inputs + explicit alloc_input), and the witness generation used the new WitnessCS (0 inputs + explicit alloc_input). They should match. Yet proofs were still failing.
Phase 2: The Decisive Experiment — Disabling PCE
At a critical juncture, the assistant made a decision that would reshape the entire investigation. Instead of continuing to trace through logs and source code, it designed a simple, binary experiment: disable PCE entirely and see if proofs still fail. The CUZK_DISABLE_PCE=1 environment variable was a kill switch designed exactly for this kind of debugging — it would force the service to bypass the PCE path entirely and fall back to standard bellperson synthesis.
The experiment was executed in [msg 335], where the assistant added CUZK_DISABLE_PCE=1 to the systemd service file and restarted the daemon. The logic was impeccable: if the PCE changes were the root cause, then disabling PCE entirely should make the proofs pass, since the standard bellperson path uses ProvingAssignment directly and never touches WitnessCS::new(). The service restarted successfully, and the logs confirmed with the message "PCE disabled via CUZK_DISABLE_PCE=1 — using standard synthesis."
Then the assistant waited for a proof to complete. The proof with PCE disabled took significantly longer — the logs showed synth_ms=871453 (871 seconds of synthesis time) compared to the PCE-accelerated path which would have been much faster. But the result was unambiguous: still 0 out of 10 partitions valid.
The message at [msg 347] captures the moment of realization: "Still 0/10 valid even with PCE disabled! This is NOT a PCE bug. The problem is in the partitioned pipeline itself or the GPU proving path. PCE is innocent."
This is a textbook example of scientific debugging. The assistant formulated a hypothesis, designed an experiment to test it, executed the experiment cleanly, and — most crucially — accepted the results even when they contradicted its expectations. The willingness to abandon a favored hypothesis after investing hours in it demonstrates intellectual honesty and methodological rigor.
The assistant then walked through the logic explicitly: WitnessCS::new() is only used by the PCE path. With PCE disabled, the standard bellperson path uses ProvingAssignment directly — it never calls WitnessCS::new(). So if PCE is disabled and proofs still fail, the issue must be something else entirely. This reasoning, visible in [msg 352], represents the paradigm shift that redirected the investigation from the PCE subsystem to the GPU proving pipeline.
Phase 3: Local Validation — Ruling Out a Code Regression
With PCE exonerated, the assistant needed to determine whether the code itself was fundamentally broken or whether the issue was environmental. The user had confirmed that the partitioned pipeline worked on the local development machine, but the assistant needed to verify this independently. The local daemon might have been running with a cached old PCE from before the WindowPoSt fix, masking a regression. The only way to know for sure was to run the exact same proof type through the local daemon and observe the result.
However, this plan hit an unexpected roadblock. The cuzk-bench benchmarking tool — the natural choice for running a local test — was not present in the build output directory. A find command in [msg 358] revealed only cuzk-daemon and two hashed dependency variants, but no cuzk-bench. This forced the assistant to build the tool from source, which in turn required a Rust toolchain that supported the edition2024 Cargo feature — a feature not available in the stable Rust 1.82.0 installed on the system.
The assistant explored the available toolchains via ls /home/theuser/.rustup/toolchains/, discovering a rich collection of Rust versions from 1.64.0 through 1.83.0, plus a truncated 1.86.x entry. Using rustup run 1.86.0 cargo build --release -p cuzk-bench, the assistant successfully built the benchmark tool and ran the local test.
The result was clear and unambiguous:
=== Proof Result ===
status: COMPLETED
job_id: e27983c5-31ff-4f23-8c39-3c8f7a2b833e
timings: total=55571 ms (queue=156 ms, srs=0 ms, synth=258801 ms, gpu=61591 ms)
wall time: 55966 ms
proof: 1920 bytes (hex: a11266aa6...)The proof completed successfully with status COMPLETED, producing a valid 1920-byte proof. The timing breakdown showed that the partitioned pipeline was working correctly — synthesis and GPU proving overlapped (the wall time of 56 seconds was less than the sum of synth and GPU times), which is the key feature of the partitioned pipeline design.
This local validation was the critical pivot point. It confirmed that the code was not fundamentally broken — the same binary, the same pipeline, the same proof type all worked correctly on the local machine. The only meaningful difference between the two environments was the hardware configuration: the local machine had a single NVIDIA RTX 5070 Ti GPU, while the remote host had two NVIDIA RTX 4000 Ada GPUs.
Phase 4: Tracing the Root Cause — The CUDA_VISIBLE_DEVICES Trap
With the search space narrowed to environmental differences, the assistant began a deep investigation of the GPU selection mechanism. The CuZK engine uses a per-GPU mutex pattern: each GPU gets its own C++ std::mutex, shared by all workers assigned to that GPU. Workers acquire the mutex only during CUDA kernel execution, allowing CPU preprocessing to overlap. The Rust code attempts to select which GPU a worker should use by setting CUDA_VISIBLE_DEVICES via std::env::set_var().
The assistant's initial hypothesis was that the set_var calls were causing a race condition — multiple threads racing to set the same environment variable, with the CUDA runtime seeing inconsistent values. This was a plausible theory, but reading the C++ source code in all_gpus.cpp ([msg 315]) revealed a deeper truth.
The gpus_t class is a singleton that, in its constructor, calls cudaGetDeviceCount() to enumerate available GPUs. This singleton is constructed at static initialization time — before any Rust code runs. The CUDA_VISIBLE_DEVICES environment variable is read by the CUDA runtime during this initialization, and subsequent set_var calls from Rust have absolutely no effect on the already-initialized CUDA runtime.
The consequences of this discovery were profound. Inside generate_groth16_proofs_start_c, with num_circuits=1 (the partitioned proof case, where each partition is proved individually), the code computes n_gpus = min(ngpus(), 1) = 1, meaning it always selects GPU 0 via select_gpu(0). Every partition proof targets GPU 0, regardless of which Rust worker picks up the job.
But the Rust engine creates separate mutexes per GPU. Workers assigned to "GPU 1" use gpu_mutexes[1], while workers on "GPU 0" use gpu_mutexes[0]. Since all C++ prove calls actually target the same physical GPU 0, workers from different Rust-side GPU assignments can run CUDA kernels simultaneously on GPU 0 without mutual exclusion. This is the race condition: two concurrent GPU kernel regions on the same physical device, unprotected by the same mutex, causing data races on device memory (g_d_a_cache, CUDA streams, etc.).
The 100% failure rate makes perfect sense in this light. Every proof attempt involves concurrent GPU kernel execution on the same device without proper mutual exclusion. The corruption is deterministic in its occurrence — it happens on every run because the race window is always present when multiple workers are active. The random-looking partition validity pattern is the predictable result of concurrent GPU kernel launches without proper serialization.
The Fix: A Single Shared Mutex
The fix, applied in [msg 375] to engine.rs, was elegantly simple: use a single shared mutex for all workers when num_circuits=1 (the partitioned proof case), since the C++ code internally serializes all GPU work to the same physical GPU. This eliminates the mutex mismatch entirely — all workers contend for the same lock, ensuring that only one CUDA kernel region executes at a time on the shared GPU.
The assistant considered two approaches. The "simplest and most correct fix" was a single shared mutex for all workers. The "more principled fix" would be to pass the mutex for GPU 0 to all workers, acknowledging that the C++ code always picks select_gpu(0) for num_circuits=1. The assistant settled on the simplest approach, noting that the real issue is that the CUDA_VISIBLE_DEVICES approach is fundamentally broken — the C++ code uses its own GPU selection internally, and the Rust side's attempts to influence it via environment variables are ineffective.
This fix acknowledges the reality of the C++ code's behavior rather than trying to fight it. If the C++ code always uses GPU 0 for single-circuit proofs, then the Rust synchronization layer should match that behavior with a single mutex. The alternative — fixing the CUDA_VISIBLE_DEVICES approach or modifying the C++ code — would be far more invasive and error-prone.
The Architecture of the Bug
What makes this bug particularly insidious is that it only manifests on multi-GPU systems. On a single-GPU machine, gpu_mutexes has only one entry, so all workers share the same mutex regardless of their assigned GPU index. The race condition is invisible. On a multi-GPU machine, the extra mutexes create a false sense of isolation, allowing concurrent access to the same physical device.
The bug also highlights a class of problems that plague heterogeneous distributed systems: the gap between logical resource management and physical resource behavior. The Rust code operated on a logical model of "GPU 0" and "GPU 1" with corresponding mutexes, but the underlying CUDA runtime operated on a physical model where only GPU 0 existed. The mismatch between these models produced a failure mode that was invisible on single-GPU systems and only manifested under specific multi-GPU configurations.
Let us trace the exact chain of events that produces the race condition:
- Static initialization: The C++
gpus_tsingleton is constructed before any Rust code runs. It callscudaGetDeviceCount(), which readsCUDA_VISIBLE_DEVICES(or its absence) and discovers 2 physical GPUs. The singleton stores pointers to both GPUs. - Rust engine startup: The Rust code creates
gpu_mutexes— a vector of 2 mutexes, one per GPU. It spawns 4 GPU workers: workers 0-1 assigned to GPU 0 (mutex[0]), workers 2-3 assigned to GPU 1 (mutex[1]). - Proof submission: A PoRep partitioned proof arrives. The engine splits it into 10 partitions, each synthesized independently and submitted to the GPU pipeline.
- GPU prove (worker 0, partition 3): Worker 0 calls
std::env::set_var("CUDA_VISIBLE_DEVICES", "0")— this has no effect because the CUDA runtime already initialized. Worker 0 acquiresgpu_mutexes[0]and enters the C++generate_groth16_proofs_start_cwithnum_circuits=1. Inside,n_gpus = min(2, 1) = 1, soselect_gpu(0)is called. The CUDA kernel runs on GPU 0. - GPU prove (worker 2, partition 5): Worker 2 calls
std::env::set_var("CUDA_VISIBLE_DEVICES", "1")— also a no-op. Worker 2 acquiresgpu_mutexes[1](a different mutex!) and enters the same C++ function. Sincenum_circuits=1,n_gpus = 1, andselect_gpu(0)is called again. The CUDA kernel runs on GPU 0 simultaneously with worker 0's kernel, because they hold different mutexes. - Data race: Two CUDA kernels execute concurrently on GPU 0, both reading and writing to
g_d_a_cache(the device-side cache for intermediate MSM results). The cache is corrupted. The proof is garbage. This sequence repeats for every pair of workers that happen to be active simultaneously. The random partition validity pattern is simply a function of which workers overlap in time.
Lessons for Debugging Complex Systems
This debugging session offers several valuable lessons for engineers working on complex, multi-language, GPU-accelerated systems:
1. Design experiments that can falsify your hypothesis. The CUZK_DISABLE_PCE=1 test was precisely such an experiment. It could have confirmed the hypothesis (proofs pass → PCE was the cause) or falsified it (proofs still fail → PCE is not the cause). The assistant accepted the falsification and adjusted its investigation accordingly. This is the essence of the scientific method applied to debugging.
2. Environment variables are not a reliable communication channel across language boundaries. The Rust code's attempt to influence C++ GPU selection via CUDA_VISIBLE_DEVICES failed because the CUDA runtime reads this variable at a different point in the initialization sequence than the Rust code expected. This is a well-known but often misunderstood aspect of CUDA programming. The CUDA_VISIBLE_DEVICES environment variable is consumed by the CUDA driver at the first CUDA API call — typically during cudaGetDeviceCount() or cudaSetDevice(). Once the CUDA runtime has initialized its device list, changing the environment variable has no effect.
3. Race conditions can be environmentally contingent. The bug only manifested on multi-GPU systems, making it impossible to reproduce on the developer's single-GPU machine. This is a common pattern in GPU programming, where hardware configuration differences can hide or expose concurrency bugs. The corollary is that testing on a single-GPU machine provides no assurance about behavior on multi-GPU machines.
4. The simplest fix is often the most robust. The assistant considered a "more principled fix" but chose the simplest approach: a single shared mutex. This fix is less elegant but more resilient to future changes in the C++ GPU selection logic. It acknowledges the reality of the system's behavior rather than trying to impose an idealized model.
5. Verify assumptions against actual source code. The assistant's initial hypothesis about the CUDA_VISIBLE_DEVICES race condition was plausible but wrong. Reading the C++ source code in all_gpus.cpp revealed that the environment variable was read only once at static initialization time, forcing a complete revision of the causal model. The lesson: when debugging across language boundaries, always verify your assumptions about the other language's runtime behavior by reading the actual source code.
6. The gap between logical and physical resource models is a fertile source of bugs. The Rust code modeled the system as having two independent GPUs with independent mutexes. The C++ code, however, collapsed all work to a single physical GPU for single-circuit proofs. The mismatch between these models was invisible until the system was exercised with multiple GPUs. This pattern — logical resource management diverging from physical resource behavior — is a recurring source of concurrency bugs in distributed and heterogeneous systems.
The Broader Context: Multi-GPU Programming in Mixed-Language Systems
The CuZK proving engine is a remarkable piece of software engineering. It combines Rust for safe concurrency management with C++ CUDA code for maximum GPU performance. The Rust side manages worker threads, mutexes, and job scheduling, while the C++ side handles the actual GPU kernel launches and memory management. This division of labor is natural — Rust provides memory safety and concurrency primitives, while CUDA C++ provides direct access to GPU hardware.
However, this architecture creates a seam where the two languages' assumptions about resource ownership can conflict. The Rust code assumes it controls which GPU a worker uses (via CUDA_VISIBLE_DEVICES), while the C++ code has its own internal GPU selection logic that completely bypasses the Rust-side control. The seam is invisible in single-GPU configurations because there's only one mutex and one GPU — the Rust and C++ models happen to agree. But with multiple GPUs, the seam tears open.
This is not a criticism of the architecture — the per-GPU mutex pattern is a sound design for multi-GPU workloads where each GPU is truly independent. The problem was that the partitioned proof pipeline (with num_circuits=1) doesn't use multiple GPUs at all, yet the Rust code still created multiple mutexes as if it did. The fix — a single shared mutex for the single-circuit case — restores the alignment between the Rust and C++ resource models.
Conclusion
The investigation into the 100% PoRep proof failure rate on the multi-GPU remote host is a masterclass in systematic debugging. From the initial suspicion of the PCE changes, through the decisive experiment that exonerated PCE, to the local validation that confirmed the code was not fundamentally broken, and finally to the deep dive into the C++ GPU selection code that revealed the CUDA_VISIBLE_DEVICES trap — each step was guided by evidence rather than intuition.
The root cause was a mismatch between the Rust engine's assumption (that different GPUs have different mutexes) and the C++ backend's behavior (that all partitioned proofs target GPU 0). The fix — a single shared mutex for all workers when num_circuits=1 — was simple in concept but required deep understanding of the interaction between Rust's concurrency management and C++'s GPU runtime initialization.
This debugging session stands as a testament to the value of the scientific method in software engineering. Formulate hypotheses, design experiments, run them cleanly, accept the results, and iterate. The bug that seemed like a PCE regression was, in the end, a GPU race condition that had nothing to do with constraint synthesis. Only by systematically eliminating possibilities could the true root cause be found.
The fix still needed to be built, deployed, and tested on the remote host — but the hard part was done. The race condition had been identified, understood, and neutralized with a clean, minimal change. The phantom GPU — the one that existed only in the Rust code's logical model but not in the C++ runtime's physical reality — had been laid to rest.