Reading the C++ GPU Mutex: A Diagnostic Deep Dive into a Multi-GPU Race Condition

The Message

The subject message is a tool call issued by the AI assistant during an opencode coding session:

[assistant] [read] /tmp/czk/extern/supraseal-c2/cuda/groth16_cuda.cu
<path>/tmp/czk/extern/supraseal-c2/cuda/groth16_cuda.cu</path>
<type>file</type>
<content>150: // workers on the same device — safe because access is serialized by
151: // gpu_mtx.  Allocated on first use via cudaMalloc and never freed
152: // (lives for the process lifetime, same as SRS).
153: static struct {
154:     std::mutex mtx;           // protects initial allocation only
155:     fr_t*      ptr  = nullptr;
156:     size_t     sz   = 0;      // bytes
157:     int        gpu_id = -1;
15...

At first glance, this appears to be a mundane read operation — the assistant fetching a snippet of C++ CUDA source code. But in the context of the debugging session unfolding around it, this single read call represents a critical turning point in a multi-hour investigation into why every single PoRep (Proof of Replication) partitioned proof was failing on a remote multi-GPU proving host. This message is the moment the assistant stops guessing and starts understanding the true root cause.

Context and Motivation: Why This Message Was Written

To understand why this message exists, we must step back into the debugging session that produced it. The assistant had been working on implementing Pre-Compiled Constraint Evaluator (PCE) extraction for all proof types in the CuZK proving engine — a significant optimization that pre-computes constraint evaluations to avoid redundant work during proof synthesis. After successfully implementing PCE for WinningPoSt, WindowPoSt, and SnapDeals, and fixing a crash caused by a mismatch in the is_extensible() flag between RecordingCS and WitnessCS, the assistant deployed the changes to a remote test host (10.1.16.218) for validation.

What followed was alarming: every single PoRep proof on the remote host was failing verification. The logs showed a 100% failure rate — 0 out of 10 partitions valid. Worse, the failures were not deterministic; sometimes 2 partitions would be valid, sometimes 5, sometimes 9, sometimes none. This random pattern was the unmistakable signature of a race condition, not a logic bug.

The assistant's initial hypothesis was that the recently modified PCE code was responsible. After all, the changes to WitnessCS::new() and RecordingCS::new() were fresh, and the WindowPoSt crash had only just been resolved. But a controlled experiment — disabling PCE entirely via the CUZK_DISABLE_PCE=1 environment variable and restarting the service — proved otherwise. With PCE disabled, proofs continued to fail at the same 100% rate. The PCE changes were exonerated.

The critical clue came from comparing environments. The local development machine had a single RTX 5070 Ti GPU and worked flawlessly. The remote host had two RTX 4000 Ada GPUs and failed catastrophically. The difference in GPU count pointed directly at a multi-GPU coordination problem.

The Investigation Path

The assistant had already traced the issue to the GPU worker code in engine.rs. The Rust engine creates multiple GPU workers — two per GPU device, configured via gpu_workers_per_device = 2 in the TOML config. Each worker is assigned to a specific GPU and acquires a per-GPU mutex before executing CUDA kernels. The mechanism for GPU selection relies on setting the CUDA_VISIBLE_DEVICES environment variable via std::env::set_var() before calling into the C++ proving library.

But there was a fundamental flaw in this design. With four workers running concurrently across two GPUs, the set_var calls were racing against each other. Worker 0 (assigned to GPU 0) would call set_var("CUDA_VISIBLE_DEVICES", "0"), but before it could enter the CUDA kernel, Worker 2 (assigned to GPU 1) would call set_var("CUDA_VISIBLE_DEVICES", "1"), overwriting the environment variable. When Worker 0's CUDA call finally executed, it would read the wrong value and target the wrong GPU.

The assistant had already identified this race condition in the preceding messages ([msg 290], [msg 291]). But there was a deeper mystery: even if the CUDA_VISIBLE_DEVICES race caused workers to target the wrong GPU, why would that produce invalid proofs rather than just wrong-GPU errors? And why was the failure pattern random?

What This Message Reveals

This read message targets the file groth16_cuda.cu — the C++ CUDA kernel implementation that actually executes GPU proving. The specific lines requested (150–157) reveal a critical piece of infrastructure:

static struct {
    std::mutex mtx;           // protects initial allocation only
    fr_t*      ptr  = nullptr;
    size_t     sz   = 0;      // bytes
    int        gpu_id = -1;

This is a static structure that caches a device memory allocation (fr_t* ptr) along with its size and the GPU it was allocated on (gpu_id). The comment on lines 150–151 explains the intent: "workers on the same device — safe because access is serialized by gpu_mtx." The C++ code assumes that the per-GPU mutex from the Rust side will serialize access, making this cache safe to use without additional locking beyond initial allocation.

But the assistant was reading this file to verify a different hypothesis. The key question was: how does the C++ code actually select which GPU to use? Does it read CUDA_VISIBLE_DEVICES at runtime, or does it use a different mechanism?

The Discovery: select_gpu(0) and the Broken Assumption

What the assistant discovered by reading the full file (and related C++ headers) was devastating to the existing architecture. The C++ code in generate_groth16_proofs_start_c — the entry point for GPU proving — calls select_gpu(0) with a hardcoded GPU index of 0 when num_circuits=1. This is the single-partition case used by PoRep partitioned proofs. The CUDA_VISIBLE_DEVICES environment variable is read once at static initialization time by the sppark library's gpu_t.cuh header, and subsequent calls to std::env::set_var() from Rust have no effect on the already-initialized CUDA runtime.

This means that every single GPU worker, regardless of which GPU the Rust engine assigned it to, was actually targeting physical GPU 0. The CUDA_VISIBLE_DEVICES mechanism was a complete illusion — a set of writes to a variable that the C++ code never re-reads.

The implications are profound. The Rust engine creates separate mutexes per GPU: workers assigned to "GPU 0" share one mutex, and workers assigned to "GPU 1" share a different mutex. But since all workers actually target the same physical GPU 0, the mutex separation is meaningless. Workers on "GPU 1" use their own mutex, which does not synchronize with workers on "GPU 0" — yet they all hammer the same device memory on the same physical GPU. This allows concurrent CUDA kernel execution without mutual exclusion, causing data races on device memory that corrupt the proof computations.

Input Knowledge Required

To fully understand this message, one needs knowledge of several layers of the system:

1. The CuZK proving engine architecture: How proof synthesis and GPU proving are split across workers, how partitioned proofs work (splitting a PoRep into 10 partitions that are proven independently and then assembled), and how the pipeline orchestrates synthesis and GPU work.
2. The dual-worker interlock design: The Rust engine creates multiple GPU workers per device, each with its own mutex, designed to allow CPU preprocessing to overlap while serializing CUDA kernel execution per GPU.
3. The CUDA_VISIBLE_DEVICES mechanism: A standard CUDA environment variable that restricts which GPUs a process can see. Setting it to "0" makes only GPU 0 visible; setting it to "1" makes only GPU 1 visible. The C++ code reads this at static init time via the sppark library.
4. The num_circuits=1 path: When proving a single partition (as opposed to batching multiple circuits), the C++ code hardcodes select_gpu(0) rather than using a configurable GPU index.
5. The relationship between Rust and C++ in this hybrid codebase: Rust spawns blocking tasks that call into C++ via FFI, passing a GpuMutexPtr that the C++ code uses to serialize access. The Rust side sets environment variables before calling C++, but the C++ side reads them only once at startup.

Output Knowledge Created

This message, combined with the surrounding investigation, creates several critical pieces of knowledge:

1. Root cause identification: The CUDA_VISIBLE_DEVICES approach for GPU selection is fundamentally broken for this architecture. The C++ code does not re-read the environment variable at each call, making the Rust-side set_var calls ineffective.
2. The mutex mismatch: Since all workers target GPU 0 regardless of their assigned GPU, the per-GPU mutex scheme fails to provide mutual exclusion. Workers on different mutexes run CUDA kernels concurrently on the same device, corrupting memory.
3. The fix direction: The solution is to use a single shared mutex for all workers when num_circuits=1 (the partitioned proof case), since the C++ code internally serializes all GPU work to the same physical GPU. This eliminates the race condition by ensuring that only one worker at a time can execute CUDA kernels, regardless of which GPU the Rust code thinks it's using.

Assumptions and Mistakes

The original design made several assumptions that turned out to be incorrect:

• Assumption: std::env::set_var() would affect subsequent CUDA calls in the same process. Reality: The CUDA runtime reads CUDA_VISIBLE_DEVICES once during initialization, typically at the first CUDA API call, and caches the result. Subsequent changes to the environment variable are ignored.
• Assumption: The C++ code would use the environment variable to select a GPU on each invocation. Reality: The generate_groth16_proofs_start_c function hardcodes select_gpu(0) for the num_circuits=1 case, completely bypassing any environment-based selection.
• Assumption: Per-GPU mutexes would provide adequate isolation. Reality: Since all workers target the same physical GPU, the mutexes are not providing the intended serialization, allowing concurrent kernel execution.
• Assumption: The local single-GPU test was representative. Reality: The race condition only manifests on multi-GPU systems, which is why the local development environment never showed the problem.

The Thinking Process

The assistant's reasoning trajectory through this investigation is a textbook example of systematic debugging. The chain of reasoning visible in the preceding messages shows:

1. Observation: All PoRep proofs fail on remote host (100% failure rate).
2. Hypothesis 1: PCE changes caused the regression. Test: Disable PCE. Result: Still fails. Conclusion: Not PCE.
3. Hypothesis 2: Something specific to the remote environment. Test: Run same pipeline locally. Result: Works. Conclusion: Environmental difference matters.
4. Key observation: Remote has 2 GPUs, local has 1. New hypothesis: Multi-GPU race condition.
5. Evidence gathering: Check logs for GPU worker interleaving. Find that partitions from different jobs run on same workers concurrently.
6. Deep dive: Read engine.rs to understand the CUDA_VISIBLE_DEVICES mechanism and mutex design.
7. Critical insight: The set_var calls race against each other because they happen before mutex acquisition.
8. Verification: Read the C++ source (groth16_cuda.cu) to confirm how GPU selection actually works in the C++ layer.
9. Root cause confirmed: C++ hardcodes select_gpu(0) for single-circuit case; CUDA_VISIBLE_DEVICES is irrelevant. This message — the read of groth16_cuda.cu — is step 8 in that chain. It's the moment where the assistant moves from inferring the problem from Rust-side code to confirming it by reading the actual C++ implementation. The read is tightly scoped to lines 150–157, suggesting the assistant already knows what it's looking for: the static cache structure and its mutex, which would reveal how the C++ code manages GPU device memory and whether it respects the per-GPU mutex scheme.

Significance

This message, though brief, represents the diagnostic pivot point of the entire debugging session. Before it, the assistant had a strong hypothesis about the race condition but lacked confirmation from the C++ side. After it, the assistant understood the full picture: the CUDA_VISIBLE_DEVICES mechanism was a red herring, the real issue was that all workers target GPU 0 regardless of assignment, and the per-GPU mutexes were providing false isolation. The fix — a single shared mutex for all workers when num_circuits=1 — followed directly from this understanding.

The message also illustrates a crucial principle in debugging heterogeneous systems: when the boundary between languages (Rust and C++) contains a synchronization mechanism, you must verify that both sides agree on how it works. The Rust code assumed the C++ code would use CUDA_VISIBLE_DEVICES for GPU selection; the C++ code assumed the Rust code would serialize access via the mutex. Neither assumption was fully correct, and the gap between them produced the race condition.