Chunk 3.0

In this chunk, the conversation pivoted from the initial PoRep proof failure fix to a deeper architectural issue. The earlier diagnosis had identified that the C++ GPU proving code always routes single-circuit proofs to GPU 0 regardless of which Rust worker submits them, causing data races on multi-GPU systems. The initial “fix” was a shared mutex that serialized all partition proofs onto GPU 0, effectively wasting the second GPU. However, when a new SnapDeals workload (16 identical partitions) OOM’d on a 20 GB RTX 4000 Ada host (p-dev-ngw-1), it became clear that the shared mutex was a lazy hack: two workers still entered the GPU code simultaneously, and the VRAM budget for a single SnapDeals partition was too large to allow concurrent kernel execution on the same device. The proper solution was to thread a `gpu_index` parameter through the entire call chain so that the C++ code uses the GPU assigned by the Rust engine instead of always defaulting to GPU 0. This required changes across multiple layers: the C++ `groth16_cuda.cu` (add `gpu_index` parameter, use `select_gpu(gpu_index)` for single-circuit proofs), the Rust FFI in `supraseal-c2/src/lib.rs`, the bellperson prover functions (`prove_start`, `prove_from_assignments`), the pipeline layer (`gpu_prove`, `gpu_prove_start`), and finally the engine’s GPU worker code in `engine.rs`. The shared mutex hack was reverted, and all call sites now pass either the assigned GPU ordinal or `-1` (auto) for non‑engine paths. The build succeeded, and the next step is to deploy the new binary to both the original test host (cs-calib) and the SnapDeals host (p-dev-ngw-1) to verify correct GPU load balancing and elimination of OOMs.

From Lazy Hack to Proper Architecture: The Multi-GPU Fix That Saved CuZK 2104 words

Message Articles

Subagent Sessions