The Anatomy of a GPU Data Race: Debugging Multi-GPU Proof Failures in a Zero-Knowledge Proving Engine

Introduction

In the course of building a high-performance zero-knowledge proving system, engineers often encounter bugs that defy straightforward diagnosis. These bugs hide at the intersection of multiple programming languages, concurrency models, and hardware abstractions. Message [msg 377] captures one such moment: a comprehensive summary written by an AI assistant during a debugging session that had already consumed dozens of messages, multiple tool calls, and several false leads. The message is a state-of-the-investigation document—a structured knowledge dump that consolidates everything learned about a baffling 100% proof failure rate on a remote multi-GPU test host.

This article examines that single message in depth. We will explore why it was written, what reasoning it reveals, what assumptions it makes, what knowledge it consumes and produces, and what it tells us about the nature of debugging distributed GPU systems. The message is not a fix—it is a diagnosis, a plan, and a confession of partial completion. And in that, it is perhaps more interesting than the fix itself.

Context: The Debugging Arc

To understand message [msg 377], we must understand what preceded it. The assistant had been working on implementing Pre-Compiled Constraint Evaluator (PCE) extraction for all proof types in the CuZK proving engine—a system for generating zero-knowledge proofs for the Filecoin network. The PCE is an optimization that pre-compiles circuit constraints so that proving can skip expensive synthesis steps. The assistant had successfully implemented PCE for WinningPoSt, WindowPoSt, and SnapDeals proof types, and had added a partitioned pipeline for SnapDeals to overlap CPU synthesis with GPU proving.

However, enabling PCE for WindowPoSt caused a crash. The assistant traced that crash to a mismatch in the is_extensible() flag between two constraint system types (RecordingCS and WitnessCS), and fixed it by making RecordingCS fully extensible. That fix compiled cleanly and restored correct proving for WindowPoSt with PCE enabled.

But then a new problem emerged. When the assistant deployed the fixes to a remote test host (10.1.16.218) with two NVIDIA RTX 4000 Ada GPUs, partitioned PoRep proofs began failing with a 100% failure rate. The failure pattern was erratic: sometimes 0 out of 10 partitions were valid, sometimes 2, sometimes 5, sometimes 9. The assistant initially suspected the PCE changes, since those were the most recent modifications. But a critical experiment—disabling PCE entirely via CUZK_DISABLE_PCE=1—showed the same failure rate. The PCE was innocent.

This launched a deeper investigation. The assistant compared the local development machine (single RTX 5070 Ti, where proofs worked) with the remote host (dual RTX 4000 Ada, where proofs failed). The key difference was the number of GPUs. Through a series of code reads, git diffs, and logical deduction, the assistant identified the root cause: a GPU mutex mismatch on multi-GPU systems.

The Structure of Message 377

Message [msg 377] is structured as a formal document. It has clear sections: Goal, Instructions, Discoveries, Accomplished, Remaining work, and Relevant files / directories. This structure is notable because the assistant is not addressing the user directly—it is writing a reference document that could serve as a handoff note, a debugging journal entry, or a task specification for the next phase of work.

The message begins with a goal statement that frames both the immediate tactical objective ("Fix PoRep proof failures on the remote test host") and the broader strategic context ("enabling PCE extraction for all proof types and adding a partitioned pipeline for SnapDeals proofs"). This dual framing is important: it reminds the reader (and perhaps the assistant itself, in a future round) that the current bug, while urgent, is a stepping stone toward a larger architectural goal.

The Instructions section is remarkably detailed. It documents the remote host access method, the service configuration, the build command with its intricate environment variable setup (including paths to CUDA 13.0, gcc-13, and the NVCC compiler flags), the deployment procedure via rsync and systemctl, and the local test setup. This section reads like an operations manual—and that is precisely its purpose. The assistant is externalizing working memory, ensuring that if the session is interrupted or if the user needs to take over, the operational context is preserved.

The Discoveries: A Detective Story in Seven Points

The heart of the message is the Discoveries section, which lists seven findings. These are not presented in chronological order but in logical order, from root cause to consequences to confirmatory evidence. Let us examine each one.

Discovery 1: The GPU mutex mismatch. The C++ SupraSeal code (generate_groth16_proofs_start_c in groth16_cuda.cu) selects GPUs internally using the formula n_gpus = min(ngpus(), num_circuits). For partitioned proofs, num_circuits=1, so n_gpus = 1 regardless of how many physical GPUs are available. The code always calls select_gpu(0), targeting the first physical GPU. Meanwhile, the Rust engine attempts to assign workers to specific GPUs by calling std::env::set_var("CUDA_VISIBLE_DEVICES")—but this call is completely ineffective because the CUDA runtime reads that environment variable only once, at static initialization time, before any Rust code runs.

This is a classic systems debugging insight: an API that appears to control behavior (set_var) is actually a no-op because the underlying runtime has already committed to its configuration. The assistant traced this through the C++ codebase, identifying sppark/util/all_gpus.cpp as the location where the static gpus_t::all() singleton captures the GPU configuration at construction time.

Discovery 2: The mutex-per-GPU architecture. The Rust engine creates one C++ mutex per GPU. Workers 0 and 1 (assigned to GPU 0) share gpu_mutexes[0], while workers 2 and 3 (assigned to GPU 1) share gpu_mutexes[1]. Since all partition proofs actually execute on GPU 0 (due to Discovery 1), workers from different GPU assignments can run CUDA kernels simultaneously on the same physical device without mutual exclusion. This corrupts proof data.

The beauty of this diagnosis is that it explains the erratic failure pattern. When two workers happen to enter the GPU code region at the same time, they race on device memory (g_d_a_cache, CUDA streams, etc.). Sometimes they collide, sometimes they don't. Hence the varying number of valid partitions per proof (0/10, 2/10, 5/10, 9/10).

Discovery 3: Single-GPU systems are immune. The local dev machine has one GPU, so all workers share the same mutex. The bug only manifests on multi-GPU systems. This explains why the partitioned pipeline worked perfectly during local testing but failed catastrophically on the remote host.

Discovery 4: PCE is not the cause. This is a crucial methodological point. The assistant had spent significant effort debugging PCE-related issues (the WindowPoSt crash). When the new failures appeared, it was natural to suspect the PCE changes. But the assistant performed a controlled experiment: disable PCE and observe whether the failure persists. It did. This allowed the assistant to discard the PCE hypothesis and focus on the actual cause.

Discovery 5: Dead code. The set_var("CUDA_VISIBLE_DEVICES") calls in the pipeline GPU workers are dead code. They have no effect on GPU selection. This is a code quality issue that the assistant flags but deprioritizes.

Discovery 6: Local proofs pass. This confirmatory evidence reinforces the single-GPU vs multi-GPU hypothesis. The assistant ran cuzk-bench single --type porep --c1 /data/32gbench/c1.json against the local daemon and got a successful result (1920 bytes, status=COMPLETED).

Discovery 7: Remote host details. The assistant documents the remote host's software stack: Ubuntu 24.04, 2x NVIDIA RTX 4000 Ada, CUDA 13.0, gcc-13, Rust 1.93.1. This information is critical for reproducing the bug and for ensuring the fix targets the right environment.

The Reasoning Process: How the Assistant Thought

What is remarkable about message [msg 377] is not just the conclusions but the reasoning process that produced them. The message is the output of a chain of inference that spanned multiple rounds of tool use, code reading, and hypothesis testing.

The assistant's thinking followed a pattern that any experienced debugger would recognize:

  1. Observe the symptom: 100% proof failure on remote host, erratic per-partition validity.
  2. Form an initial hypothesis: The recent PCE changes must have broken something.
  3. Test the hypothesis: Disable PCE, observe the same failure. Hypothesis rejected.
  4. Broaden the search: If PCE is innocent, what else changed? Check git history, examine diffs.
  5. Find the differentiator: The local machine works; the remote machine fails. What's different? Number of GPUs.
  6. Trace the mechanism: How does the Rust engine assign workers to GPUs? How does the C++ code select GPUs? Where is the mutex acquired?
  7. Identify the contradiction: The Rust code thinks it's assigning workers to different GPUs with different mutexes, but the C++ code always uses GPU 0. The mutexes don't match the actual execution.
  8. Verify the explanation: Does this explain the erratic failure pattern? Yes—concurrent access to the same GPU without mutual exclusion causes data races, which manifest randomly.
  9. Design the fix: Use a single shared mutex for all workers when num_circuits=1. This is textbook root cause analysis. The assistant moved from symptom to mechanism to root cause, systematically eliminating alternative explanations along the way.

Assumptions and Potential Mistakes

The message contains several assumptions that deserve scrutiny.

Assumption 1: The shared mutex fix is correct. The assistant proposes using a single global mutex for all workers when num_circuits=1. This is a pragmatic fix, but it serializes all GPU work onto a single device, potentially wasting the second GPU. The assistant acknowledges this implicitly by noting that the fix is for partitioned (single-circuit) proofs, while batched (multi-circuit) proofs would use per-GPU mutexes. But the message does not discuss whether the fix could be made more elegant—for example, by threading a gpu_index parameter through the entire call chain so that the C++ code uses the correct GPU. That more principled fix would come later (in subsequent chunks), but at this moment the assistant is opting for the simplest possible correction.

Assumption 2: The CUDA_VISIBLE_DEVICES calls are harmless dead code. The assistant says removing them is "cosmetic, not urgent." This is probably correct, but dead code has a cost: it misleads future readers into thinking GPU assignment is controllable via environment variables. The assistant is prioritizing correctness over code hygiene, which is a reasonable triage decision.

Assumption 3: The partial edit compiles. The assistant notes that the edit in engine.rs has renamed gpu_mutex_addr to per_gpu_mutex_addr but has not updated downstream references. The code "won't compile until fixed." This is an acknowledged broken state—the assistant is documenting the incomplete work rather than presenting a finished product.

Potential mistake: Overlooking the monolithic worker path. The assistant lists three callsites that need updating, noting that the monolithic workers (line ~2511) "may also need update." This hedging language suggests uncertainty about whether the monolithic path is affected. If the monolithic path is used for non-partitioned proofs (e.g., the original non-partitioned pipeline), it might not need the shared mutex. But if it is also used for partitioned proofs in some configurations, failing to update it would leave the bug partially unfixed.

Input Knowledge Required

To understand message [msg 377], a reader needs substantial domain knowledge:

Output Knowledge Created

Message [msg 377] creates several forms of knowledge:

  1. A documented root cause analysis: The seven discoveries form a coherent explanation of the bug, from mechanism to manifestation. This is valuable even if the fix changes—the analysis captures the reasoning that led to the fix.
  2. An operational handbook: The Instructions section documents how to build, deploy, and test the system. This is knowledge that would otherwise be scattered across multiple previous messages and tool calls.
  3. A task specification: The Remaining work section lists five concrete tasks with enough detail for someone (or the assistant itself) to execute them without re-reading the entire conversation history.
  4. A code map: The Relevant files section maps the codebase, identifying which files are involved in which aspect of the bug. This is a form of architectural documentation.
  5. A partial implementation: The partially-edited engine.rs file represents work in progress. The message documents what was changed and what remains, preventing duplicate work or confusion.

The Incomplete Fix: A Deliberate Choice

One of the most interesting aspects of message [msg 377] is that it documents an incomplete fix. The assistant has started editing engine.rs to add a shared mutex, but has not updated the downstream callsites. The code will not compile. Why would the assistant stop mid-edit and write a summary instead?

The answer lies in the nature of the conversation. The assistant is an AI that operates in rounds: it issues tool calls, waits for results, and then produces the next message. The edit to engine.rs was made in a previous round ([msg 375]). When the assistant received the result ("Edit applied successfully"), it could have continued editing the downstream callsites. Instead, it produced message [msg 377]—a comprehensive summary.

This suggests a deliberate architectural choice. The assistant recognized that the fix was becoming complex, spanning multiple files and requiring careful coordination. Rather than rushing to complete the edit and potentially introducing new bugs, the assistant paused to document the current state, consolidate the discoveries, and plan the remaining work. This is a mature engineering practice: when a debugging session reveals unexpected complexity, stop and document before proceeding.

The message also serves as a handoff point. If the user wants to take over the fix, they have all the context they need. If the assistant continues, it has a clear roadmap. The message is a contract between the assistant and the user about what has been learned and what remains to be done.

The Broader Implications

Message [msg 377] illustrates several broader truths about debugging complex systems:

The red herring is the rule, not the exception. The assistant initially suspected PCE, spent significant effort investigating it, and only ruled it out through a controlled experiment. This is normal. Debugging is a process of elimination, and most hypotheses are wrong.

The environment matters as much as the code. The bug only manifested on multi-GPU systems. The local development environment (single GPU) was not just different—it was actively misleading, because it masked the bug entirely. This is a powerful argument for testing on production-like hardware.

Concurrency bugs are probabilistic. The erratic failure pattern (sometimes 0/10 valid, sometimes 9/10) is a hallmark of data races. Deterministic bugs produce deterministic failures. The assistant correctly inferred a race condition from the non-deterministic symptom pattern.

The boundary between languages is where bugs hide. The bug existed at the intersection of Rust (which managed worker assignment and mutex selection) and C++ (which managed GPU selection and kernel execution). Neither language's code was wrong in isolation—the bug was in the mismatch between their assumptions. This is a recurring theme in systems that span multiple languages.

Conclusion

Message [msg 377] is a snapshot of a mind at work—a mind that happens to be an AI assistant, but a mind that follows the same logical processes any skilled engineer would use. The message documents a root cause analysis that moved from symptom to mechanism to fix, systematically eliminating alternative explanations along the way. It captures the state of a partially-completed edit, acknowledging incompleteness rather than pretending to have finished. It externalizes operational knowledge that would otherwise be lost across dozens of previous messages.

The message is also a testament to the value of structured documentation during debugging. By writing down the discoveries, the assumptions, the remaining work, and the code map, the assistant creates a reference that can be used by the user, by itself in future rounds, or by anyone else who needs to understand or continue the work. In a complex debugging session, such documentation is not a luxury—it is a necessity.

The GPU mutex bug itself is a cautionary tale about the dangers of assuming that APIs work as advertised. The set_var("CUDA_VISIBLE_DEVICES") calls looked like they should control GPU assignment, but they were silently ineffective because the CUDA runtime had already committed to its configuration. The per-GPU mutexes looked like they should provide mutual exclusion per device, but they failed because the C++ code always targeted GPU 0 regardless of which mutex was held. The system appeared correct at every individual layer, but the layers did not agree on their shared assumptions.

This is the essence of systems debugging: finding the place where two correct pieces of code make incompatible assumptions about each other. Message [msg 377] shows us how that discovery happens, how it is documented, and how it is transformed into a plan for repair.