The Commit That Fixed Multi-GPU Proving: A Case Study in Systems Debugging

On March 6, 2026, an AI assistant working on the CuZK proving engine executed a single command that represented the culmination of hours of debugging, architectural reasoning, and cross-layer systems engineering. The message — message index 533 in the conversation — was a git commit command with a meticulously crafted commit message. On its surface, it looks mundane: five files changed, 75 insertions, 63 deletions. But this commit was the final act in a saga that had exposed a subtle GPU race condition, required two separate fix attempts, and ultimately demanded changes across C++, Rust FFI, and application-level code.

This article examines that single message in depth: the reasoning that led to it, the decisions encoded within it, and the knowledge it represents.

The Message

Here is the message in full:

Clean diff. All 5 files, focused changes.

```bash cd /tmp/czk && git add extern/bellperson/src/groth16/prover/supraseal.rs extern/cuzk/cuzk-core/src/engine.rs extern/cuzk/cuzk-core/src/pipeline.rs extern/supraseal-c2/cuda/groth16_cuda.cu extern/supraseal-c2/src/lib.rs && git commit -m "cuzk: route partition proofs to correct GPU via gpu_index parameter

Thread a gpu_index parameter through the entire proving stack (C++ -> supraseal-c2 -> bellperson -> pipeline -> engine) so that single-circuit partition proofs run on the GPU assigned to the Rust worker instead of always landing on GPU 0.

Previously, the C++ code computed n_gpus = min(ngpus(), num_circuits), which for single-circuit proofs always resolved to GPU 0 via select_gpu(0). This made per-GPU mutexes ineffective on multi-GPU systems: workers assigned to different GPUs could run CUDA kernels simultaneously on GPU 0, causing proof corruption (the original shared-mutex workaround serialized everything to one GPU).

Now gpu_index >= 0 pins work to that specific GPU, while -1 preserves the original multi-GPU fan-out for batched proofs. Also converts the global d_a_cache singleton to a per-GPU array to avoid thrashing when workers on different GPUs run concurrently." ```

The assistant then reports the commit result: [misc/cuzk-rseal-merge bf804060] cuzk: route partition proofs to correct GPU via gpu_index parameter with the same five-file change summary.

The Problem That Necessitated This Commit

To understand why this message was written, one must understand the bug it fixed. The CuZK proving engine uses multiple GPU workers to generate zero-knowledge proofs in parallel. Each worker is assigned a GPU ordinal by the Rust engine, and the system uses per-GPU mutexes to ensure that only one thread accesses a given GPU at a time. In theory, this should allow workers 0 and 1 to use GPU 0 while workers 2 and 3 use GPU 1, achieving true parallel utilization of both devices.

In practice, something was going wrong. PoRep (Proof of Replication) proofs were failing intermittently on multi-GPU systems. The initial investigation revealed that the C++ GPU proving code (groth16_cuda.cu) computed the number of GPUs to use as n_gpus = min(ngpus(), num_circuits). For single-circuit proofs — the common case for partition proofs — num_circuits is 1, so n_gpus always resolved to 1, and select_gpu(0) always selected GPU 0. The per-GPU mutexes in the Rust engine were rendered meaningless: workers assigned to GPU 1 would still enter the C++ code and land on GPU 0, where they would race with workers assigned to GPU 0, causing data corruption and proof failures.

The Shared Mutex Dead End

The first attempt at fixing this problem was a shared mutex — a single mutex that serialized all partition proofs onto GPU 0. This "fix" was a classic systems debugging trap: it made the symptom disappear (proofs stopped failing because there was no longer any concurrent GPU access) while actually making the system worse (it wasted the second GPU entirely, serializing work that should have been parallel). The assistant's todo list from earlier in the conversation shows this as a completed item: "Revert shared mutex hack from engine.rs — restore per-GPU mutexes." The very fact that this item existed reveals the arc of the debugging process: implement a quick workaround to unblock testing, then replace it with the proper fix.

The shared mutex hack's inadequacy became glaringly apparent when a SnapDeals workload with 16 identical partitions ran out of memory on a 20 GB RTX 4000 Ada host. Two workers were still entering the GPU code simultaneously on the same device, and the VRAM budget for a single SnapDeals partition was too large to allow concurrent kernel execution. The shared mutex had not actually fixed the routing problem — it had just added a coarse serialization lock that still allowed two workers to hit GPU 0 simultaneously. The system needed a proper fix: the C++ code needed to listen to which GPU the Rust engine had assigned, rather than always defaulting to GPU 0.

The Architecture of the Proper Fix

The commit message itself documents the architectural insight. The root cause was that "the C++ code computed n_gpus = min(ngpus(), num_circuits), which for single-circuit proofs always resolved to GPU 0 via select_gpu(0)." This is a subtle but critical design flaw: the GPU selection logic was based on the number of circuits in the current batch, not on the worker assignment from the engine. For batched proofs with multiple circuits, the C++ code could spread work across GPUs by using different loop indices. But for single-circuit proofs — which are the norm for partition-level proving — the loop always had one iteration, and that iteration always used GPU 0.

The fix required threading a gpu_index parameter through five layers of the software stack:

C++ (groth16_cuda.cu): Added an int gpu_index parameter to the two entry-point functions (generate_groth16_proofs_start_c and _c). When gpu_index >= 0, the code calls select_gpu(gpu_index) to pin work to that specific GPU. When gpu_index == -1, it preserves the original behavior of auto-spreading across GPUs based on the number of circuits. The d_a_cache — a device-side cache for precomputed data — was also converted from a global singleton to a per-GPU array, preventing cache thrashing when workers on different GPUs run concurrently.
Rust FFI (supraseal-c2/src/lib.rs): Added gpu_index: i32 to the start_groth16_proof and generate_groth16_proof wrapper functions, along with the corresponding FFI declarations that bridge Rust and C++.
Bellperson (supraseal.rs): Added gpu_index: i32 to the prove_start and prove_from_assignments functions, which are the entry points for proof generation in the bellperson library.
Pipeline (cuzk-core/src/pipeline.rs): Added gpu_index: i32 to gpu_prove and gpu_prove_start, the intermediate layer that orchestrates GPU proving within the pipeline architecture.
Engine (cuzk-core/src/engine.rs): The most significant change — reverted the shared mutex hack, restored the per-GPU mutexes, and now passes gpu_ordinal as i32 through the entire call chain. This is where the Rust worker's assigned GPU ordinal finally reaches the C++ code. The diff statistics tell the story: 75 insertions and 63 deletions across five files. The C++ file had the most changes (73 lines modified), reflecting the core logic change. The engine.rs file had 37 lines changed, mostly deletions — the shared mutex hack being torn out.

Verification: Proving the Fix Works

Before the commit, the assistant had already deployed and verified the fix on a remote test host. The verification process was rigorous and multi-layered:

GPU memory allocation: nvidia-smi showed both GPUs with 13 GB memory allocated (the d_a_cache plus SRS data). Previously, GPU 1 had only 700 MB — a clear sign it was barely being used.
Load balancing: journalctl logs showed worker_id=0 gpu=0 (6 jobs), worker_id=1 gpu=0 (7 jobs), worker_id=2 gpu=1 (6 jobs), worker_id=3 gpu=1 (5 jobs). Perfectly balanced across both GPUs.
d_a_cache allocation: Both GPUs showed d_a_cache allocated 4096 MiB on gpu 0 and gpu 1 at startup, confirming the per-GPU cache array was working.
Proof correctness: All four test proofs completed with self-check PASSED.
User confirmation: The user independently validated the fix via nvtop, reporting "load on both GPUs" and that it "seemed pretty quick." This verification was crucial. The assistant had initially been misled by the gpu_tid=0 log entries (which were just the loop iterator, not the physical GPU) and the d_a_cache allocated logs that only showed GPU 0 in a narrow time window. It took digging deeper into the journal — looking back 10 minutes instead of 3 — to find the initial allocation on both GPUs. This is a textbook example of how log analysis can mislead when you don't have the full picture.

The Thinking Process Visible in the Message

The commit message is remarkable for what it reveals about the assistant's understanding of the system. It is not a mechanical changelog; it is a diagnostic summary that explains the bug's root cause, the failed prior workaround, and the rationale for the fix. The assistant explicitly contrasts the old behavior ("always resolved to GPU 0 via select_gpu(0)") with the new behavior ("gpu_index >= 0 pins work to that specific GPU, while -1 preserves the original multi-GPU fan-out"). This shows a clear mental model of the C++ GPU selection logic and how it interacts with the Rust worker assignment.

The phrase "the original shared-mutex workaround serialized everything to one GPU" is a candid acknowledgment of a mistake. The assistant does not paper over the failed first attempt; it documents it as part of the fix's motivation. This intellectual honesty is a hallmark of good engineering: the commit message tells the full story, including the dead ends, so that future readers understand why the fix takes this particular form rather than some other approach.

The decision to make -1 mean "auto" (preserving original behavior) while >= 0 means "pin to specific GPU" is a well-considered API design choice. It maintains backward compatibility for all existing call sites that don't care about GPU routing (the old prove_batch_groth16 function, for example, passes -1), while enabling the new behavior where the engine's worker assignment is honored. The commit message's note about converting "the global d_a_cache singleton to a per-GPU array" shows awareness of a secondary issue: even if GPU routing were fixed, a shared cache would cause thrashing when workers on different GPUs allocate and free cache entries concurrently.

Assumptions and Their Validity

The fix makes several assumptions that deserve examination:

That the Rust engine's GPU ordinal assignment is correct. The fix assumes that when the engine assigns worker 2 to GPU 1, that assignment is the right one. This is a reasonable assumption — the engine's worker scheduling logic is a separate concern — but it does mean the fix is only as good as the assignment algorithm.
That -1 is a safe sentinel value. The commit uses -1 to mean "auto-select," which is a common convention in C/C++ APIs (e.g., shmget uses IPC_PRIVATE as a sentinel). This is safe as long as no valid GPU index is ever -1, which is true since GPU indices are non-negative.
That the per-GPU mutexes are sufficient for correctness. The fix restores the per-GPU mutexes that were replaced by the shared mutex hack. This assumes that the only source of data races was the GPU routing issue — that once each worker runs on its assigned GPU, the per-GPU mutexes will prevent concurrent access to that GPU's resources. This assumption held up under testing.
That the C++ code path for gpu_index >= 0 is equivalent to the original auto-select path. The fix adds a new branch in the C++ code for pinned GPU selection. If this branch diverges from the original logic in subtle ways (e.g., different memory allocation patterns, different error handling), new bugs could be introduced. The testing on the remote host — where proofs passed with self-check — provides evidence that the branches are functionally equivalent.

Input Knowledge Required

To understand this message, a reader needs substantial domain knowledge:

GPU programming concepts: Understanding of CUDA device selection (cudaSetDevice, select_gpu), GPU memory allocation (d_a_cache), and the concept of per-GPU mutexes for synchronization.
Zero-knowledge proof architecture: Knowledge of how Groth16 proofs are generated, what "partition proofs" are in the context of Filecoin's proof-of-replication (PoRep) and proof-of-spacetime (PoSt) systems, and the role of the "bellperson" library (a Rust implementation of the Bellman zk-SNARKs library).
Cross-language FFI: Understanding of how Rust calls into C++ code via FFI (Foreign Function Interface), and how parameters must be threaded through multiple abstraction layers.
The CuZK proving engine: Familiarity with the engine's worker model (GPU workers assigned by ordinal), the pipeline architecture, and the distinction between single-circuit and batched proofs.
The history of the bug: Knowledge that a shared mutex workaround was previously deployed and found inadequate, and that the SnapDeals OOM crash was the trigger for the proper fix. Without this context, the commit message reads as a straightforward description of a parameter-threading change. With it, the message becomes a window into a complex debugging journey spanning multiple days, multiple codebases, and multiple failed hypotheses.

Output Knowledge Created

This commit creates several forms of knowledge:

A documented fix in the git history. The commit message itself is a permanent record of the bug's root cause, the failed workaround, and the proper solution. Any developer who encounters GPU routing issues in the future can read this commit and understand why the gpu_index parameter exists.
A verified architectural pattern. The fix demonstrates that GPU routing for single-circuit proofs must be driven by the engine's worker assignment, not by the C++ code's internal circuit count. This is a reusable insight for any similar multi-GPU proving system.
A template for cross-layer parameter threading. The five-file change shows how a parameter must be propagated through C++ → FFI → library → pipeline → engine. Future changes that need to thread a parameter through the same stack can follow this pattern.
Validation data. The verification logs (GPU memory allocation, worker-to-GPU mapping, proof completion status) provide empirical evidence that the fix is correct. This data is valuable for regression testing and for understanding the system's behavior under load.

The Broader Significance

This message matters because it represents the moment when a complex, multi-layered bug was finally put to rest. The bug had manifested in multiple ways — intermittent proof failures, OOM crashes, wasted GPU resources — each of which had been individually investigated and partially addressed. The shared mutex hack was a band-aid; the SnapDeals OOM was a symptom of the underlying routing problem; the initial log analysis that showed only GPU 0 activity was a red herring that nearly derailed the investigation.

The fix required understanding the system at every level: from the C++ CUDA kernel selection logic, through the Rust FFI bindings, through the bellperson proof generation library, through the pipeline orchestration layer, and finally to the engine's worker scheduling. Each layer had to be modified in a consistent way, and the modifications had to be verified end-to-end on a remote multi-GPU system.

The assistant's final message before the commit — "All done. The proper fix is deployed and verified" — is a moment of closure. The commit that follows is not just a code change; it is the documentation of that closure, the permanent record of a hard-won engineering victory. The commit message's clarity and completeness reflect the depth of understanding that was required to solve the problem, and the verification data provides confidence that the fix is real.

In the end, 75 lines of insertions and 63 lines of deletions — a net change of 12 lines — resolved a bug that had caused days of intermittent failures, wasted GPU capacity, and at least one OOM crash. That ratio is typical of the best engineering work: the insight is large, but the change is small.