Tracing the Ghost GPU: How a Single `read` Call Unraveled a CUDA Race Condition

The Message

The subject of this article is message index 296 in an opencode debugging session — a single read tool call issued by the AI assistant:

[assistant] [read] /tmp/czk/extern/supraseal-c2/src/lib.rs
<path>/tmp/czk/extern/supraseal-c2/src/lib.rs</path>
<type>file</type>
<content>60:         extern "C" {
61:             fn clone_SRS(by_ref: &SRS) -> SRS;
62:         }
63:         unsafe { clone_SRS(self) }
64:     }
65: }
66: 
67: unsafe impl Sync for SRS {}
68: unsafe impl Send for SRS {}
69: 
70: pub fn generate_groth16_proof<S, D, PR>(
71:     ntt_a_scalars: &[*const S],
72:     ntt_b_scalars: &[*const S],
73:     ntt_c_scalars: &[*const S],
74:     ntt_scalars_actual_size: usize,
75:     ...

At first glance, this appears unremarkable — the assistant is reading a Rust wrapper file for a CUDA library. But this message sits at a critical inflection point in a multi-hour debugging session. It represents the moment when the assistant pivoted from a mistaken hypothesis about the root cause of a catastrophic proof failure toward the correct diagnosis. Understanding why this specific file was read, and what the assistant hoped to find there, reveals the full arc of a complex systems debugging journey.

Context: The 100% Failure Rate

The session leading up to this message is a story of escalating alarm. The team had recently deployed a suite of changes to a remote proving host (10.1.16.218) — Pre-Compiled Constraint Evaluator (PCE) extraction for all proof types, a partitioned pipeline for SnapDeals, and a fix for a WindowPoSt crash caused by a mismatch in constraint system initialization. After deployment, the user reported that "proofs still failing / not fixed" ([msg 279]). The assistant investigated and discovered something far worse than expected: every single PoRep proof was invalid. Not a random fluke, not an intermittent glitch — a 100% failure rate across all 10 partitions of every proof ([msg 286]).

The logs showed a damning pattern: valid_partitions=[0, 1, 3, 4, 5, 6, 7, 8] invalid_partitions=[2, 9] for one proof, then valid_partitions=[] invalid_partitions=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] for the next. The number of valid partitions varied randomly from proof to proof, but the overall proof always failed verification. This was the hallmark of a race condition — non-deterministic corruption that sometimes produced correct-looking intermediate results but never a valid final proof.

The First Hypothesis: PCE Corruption

The assistant's initial suspicion fell on the PCE (Pre-Compiled Constraint Evaluator) fast path. The team had just modified WitnessCS::new() and RecordingCS::new() as part of the WindowPoSt fix, changing them from starting with 1 pre-allocated input (the constant ONE) to starting with 0 inputs. The user had noted that the old code worked locally with the partitioned pipeline, and the key difference was that the local machine still had the old PCE cache on disk ([msg 351]). This suggested a mismatch: perhaps the remote host had a stale PCE file extracted with the old constraint system, while the new witness generation code expected a different number of inputs.

The assistant tested this hypothesis by setting CUZK_DISABLE_PCE=1 on the remote host and restarting the service ([msg 335]). If the PCE path were the culprit, disabling it should make proofs pass. The result was definitive — and devastating: even with PCE disabled, every proof still failed with 0/10 valid partitions ([msg 347]). The PCE was innocent. The bug lay deeper, in the partitioned pipeline itself or the GPU proving path.

The User's Critical Clue

At this point, the user interjected with an observation that would redirect the entire investigation: "Note only one GPU is seeing actual activity" ([msg 292]). This was a crucial piece of information. The remote host had two NVIDIA RTX 4000 Ada GPUs, and the configuration specified gpu_workers_per_device = 2, creating four GPU worker threads. Yet only one GPU was doing real work. This hinted at a GPU selection problem — perhaps all workers were inadvertently targeting the same device.

The assistant had already noticed something suspicious in the logs: workers on different GPUs were interleaving their CUDA operations, with GPU_START for one partition occurring before GPU_END for another partition on the same worker ([msg 291]). This violated the intended serialization provided by the per-GPU mutex.

The `read` Call: What the Assistant Was Looking For

This brings us to message 296. The assistant had just finished reading the C++ CUDA source file groth16_cuda.cu and the sppark utility gpu_t.cuh in the preceding messages (<msg id=299-315>). Those reads had revealed the critical mechanism: the C++ code's select_gpu() function, which chooses which physical GPU to use, is called from generate_groth16_proofs_start_c. But the Rust-side code sets CUDA_VISIBLE_DEVICES via std::env::set_var() before calling into the C++ prover — a process-wide environment variable that is notoriously non-thread-safe.

The assistant's theory was evolving. The CUDA_VISIBLE_DEVICES race was real: four concurrent worker threads were all calling set_var() with different GPU indices, overwriting each other's values. But the assistant needed to understand exactly how the C++ side consumed this variable. Did select_gpu() read CUDA_VISIBLE_DEVICES at call time, or was it read once during CUDA initialization? The answer would determine whether the race condition was the root cause or merely a red herring.

Message 296 is the assistant reading supraseal-c2/src/lib.rs — the Rust FFI wrapper that bridges the bellperson prover to the C++ CUDA library. The assistant was looking for the function signatures of start_groth16_proof and generate_groth16_proof to see if there was a way to pass the GPU ordinal directly as a parameter, bypassing the fragile CUDA_VISIBLE_DEVICES mechanism. If the C++ API accepted a gpu_id argument, the fix would be straightforward: pass the GPU index directly instead of relying on environment variables.

What the Message Revealed — and What It Didn't

The file content shown in message 296 is frustratingly truncated. It shows only the clone_SRS extern block and the beginning of generate_groth16_proof's signature — the scalar array parameters for NTT. The critical function signatures for start_groth16_proof and generate_groth16_proofs_c (which accept the gpu_mtx mutex pointer) are not visible in this excerpt. The assistant would have seen the full file, but the conversation data only captures the first few lines.

Nevertheless, the act of reading this file was productive. It confirmed the architecture: the Rust wrapper calls into C++ functions that internally manage GPU selection. The assistant had already found in the preceding messages that select_gpu(0) is called with a hardcoded 0 when num_circuits=1 ([msg 316]), meaning the C++ code always picks GPU 0 for single-circuit proofs regardless of CUDA_VISIBLE_DEVICES. This was the key insight: the set_var calls were completely ineffective because the C++ code doesn't read CUDA_VISIBLE_DEVICES at call time — it reads it once during static initialization of the gpus_t singleton, which calls cudaGetDeviceCount() at construction time.

The Deeper Understanding

The knowledge created by message 296, combined with the preceding reads of groth16_cuda.cu and gpu_t.cuh, formed a complete picture of the bug:

CUDA_VISIBLE_DEVICES is a no-op: The C++ code reads GPU visibility once during static initialization via cudaGetDeviceCount(). By the time Rust calls set_var(), the CUDA runtime has already enumerated all devices. The environment variable has no effect.
select_gpu(0) always picks GPU 0: With num_circuits=1 (the partitioned proof case), the C++ code computes n_gpus = min(ngpus(), 1) = 1, spawning only one GPU thread with tid=0, which calls select_gpu(0). Every partition proof targets physical GPU 0.
The per-GPU mutex is useless: The Rust engine creates separate std::mutex instances per GPU — one for GPU 0 and one for GPU 1. But since all proofs actually run on GPU 0, workers assigned to "GPU 1" use a different mutex than workers on "GPU 0," yet both sets of workers are actually executing CUDA kernels on the same physical GPU 0. This allows concurrent kernel execution without mutual exclusion, causing data races on device memory. The input knowledge required to understand this message includes: familiarity with the CuZK proving engine architecture (the partitioned pipeline, PCE extraction, the bellperson prover), understanding of CUDA GPU selection mechanisms (CUDA_VISIBLE_DEVICES, cudaGetDeviceCount, cudaSetDevice), knowledge of Rust's std::env::set_var thread-safety limitations, and the specific context of the WindowPoSt fix that changed WitnessCS::new() and RecordingCS::new().

The Assumption That Nearly Led Astray

The assistant made a significant incorrect assumption earlier in the session: that the PCE path was the likely culprit. This was reasonable given the recent changes to constraint system initialization, but it turned out to be wrong. The 100% failure rate with PCE disabled conclusively disproved this hypothesis. The assistant's willingness to test this assumption empirically — by deploying a configuration change to the remote host and waiting for a proof to complete — was the right debugging discipline.

A subtler assumption was that the CUDA_VISIBLE_DEVICES race was the primary mechanism of failure. While the race is real and problematic, the deeper issue turned out to be that select_gpu(0) always picks GPU 0 regardless of the environment variable, making the mutex-per-GPU design fundamentally broken for the num_circuits=1 case. The race condition is a symptom, not the root cause.

The Thinking Process

The assistant's reasoning in this portion of the session follows a clear arc: observe the failure (100% invalid proofs), form a hypothesis (PCE corruption), test it (disable PCE), discard it when falsified, then follow the user's clue (only one GPU active) into the GPU selection code. Each read call peels back another layer: first the Rust engine code that calls set_var, then the C++ CUDA kernel that calls select_gpu, then the sppark utility that initializes the GPU list, and finally the FFI wrapper that bridges the two worlds. Message 296 is the last of these reads — the assistant is verifying that the C++ API doesn't accept a GPU ordinal parameter, confirming that the fix must be on the Rust side.

The Fix That Followed

The correct fix, which the assistant implemented in the subsequent chunk, was to use a single shared mutex for all workers when num_circuits=1 (the partitioned proof case). Since the C++ code internally serializes all GPU work to the same physical GPU 0, a single mutex provides the necessary mutual exclusion. This was applied via an edit to engine.rs and deployed to the remote host.

Message 296, in isolation, is just a file read. But in context, it is the moment when the assistant confirmed the architecture of the GPU selection mechanism and solidified the understanding that led to the correct fix. It is a testament to the value of reading the source code — not just the code you wrote, but the code you depend on.

Tracing the Ghost GPU: How a Single read Call Unraveled a CUDA Race Condition