The Dual-GPU-Worker Interlock: Formalizing Phase 8 of the cuzk Proving Engine

In the high-stakes world of Filecoin proof generation, every millisecond of GPU idle time is a tax on throughput. When the cuzk SNARK proving engine's Phase 7 per-partition dispatch architecture was benchmarked in late 2024, it revealed a tantalizing gap: the GPU was busy only 64.3% of the time. The remaining 35.7% was a mosaic of CPU-side overhead — mutex contention, proof serialization, memory trimming, and scheduler latency — that left the expensive CUDA hardware waiting. Message 2142 in this coding session marks the moment when that observation crystallized into a formal design specification. It is the summary and commit message for c2-optimization-proposal-8.md, a document that traces the exact root cause of GPU underutilization and prescribes a surgical fix: a dual-GPU-worker interlock that promises to push GPU efficiency from 64% to 98%.

The Message

The subject message is deceptively brief. After writing and committing the design document as commit 71f97bc7 on the feat/cuzk branch, the assistant reports:

Written and committed as 71f97bc7. Here's the summary:

>

## c2-optimization-proposal-8.md

>

The doc traces the full call path from Rust through FFI into generate_groth16_proofs_c and identifies the exact problem: the static std::mutex at groth16_cuda.cu:133 holds for the entire ~3.5s function, but only ~2.1s is actual CUDA kernels. The remaining ~1.3s is CPU work that could overlap with the next partition's GPU execution.

>

Key architectural points: - Two GPU workers per physical GPU, sharing a std::mutex* passed from Rust - Mutex scope narrowed to bracket only the per-GPU threads (NTT → batch-add → tail-MSM) - prep_msm_thread launches before lock acquisition — its CPU preprocessing overlaps with the other worker's CUDA kernels - b_g2_msm + epilogue run after lock release — hidden behind the other worker's CUDA execution - The sppark semaphore_t is a counting semaphore that latches notify() before wait(), so the barrier between prep_msm and per-GPU threads works correctly even when per-GPU launch is delayed by the lock

>

Recommended approach: Option 4 (passed mutex), ~75 lines of changes across 6 files. The C++ change is moving the lock scope from whole-function to just the CUDA kernel region, plus reordering prep_msm_thread.join() to happen after lock release.

This summary distills hundreds of lines of investigation, dozens of grep commands, multiple subagent task sessions, and careful code reading into six bullet points and a punchline. But to understand its significance, we must walk the path that led to it.

The Investigation: From "Jumpy" GPU to Root Cause

The story begins with the user's observation in message 2112: "looks like gpu use is pretty jumpy, generally high, but maybe there is some per-job overhead that could be cut? Or we could run two gpu-workers interlocked, one preparing to fire cuda code when the other one is actively running?" This is the seed of the Phase 8 idea — a hunch that the GPU's idle gaps are structural, not random.

The assistant immediately dives into quantitative analysis. In messages 2113–2116, it parses the TIMELINE instrumentation from the Phase 7 daemon log, computing gap statistics between GPU_START and GPU_END events. The results are stark: 110 GPU calls, 109 inter-GPU gaps, average gap of 2311ms, total gap of 251.9 seconds against 453.1 seconds of wall time. GPU efficiency: 64.3%. Eight gaps exceed 500ms, including a 125.9-second monster after the first proof (cold-start synthesis) and several 30–54 second cross-sector stalls.

But the user asks a critical clarifying question in message 2118: "Are gaps measuring time between full-blast cuda executions or whole gpu worker job processing?" This distinction is crucial. If the gaps measure only CUDA kernel idle time, the fix is one thing. If they measure whole-job processing (including CPU preamble and epilogue), the optimization strategy changes entirely.

The assistant investigates by reading the exact locations of GPU_START and GPU_END events in the engine code (messages 2119–2121). It discovers that GPU_START fires before spawn_blocking even schedules the blocking thread — it's on the async side, right before dispatching. GPU_END fires inside spawn_blocking, after gpu_prove() returns, which includes prove_from_assignments plus proof serialization. This means the measured gaps include everything: async scheduler overhead, mutex contention on the tracker, malloc_trim(0), channel mutex acquisition, span construction, and thread pool dispatch. The actual CUDA idle time is even larger than the measured gaps, because GPU_START fires before the blocking thread begins its CPU preamble.

This insight reframes the problem. The gaps aren't just CUDA idle — they're the entire inter-job overhead of the engine loop. And within each GPU job, only ~2.1s of the ~3.5s function is actual CUDA kernel execution. The remaining ~1.3s is CPU work (preprocessing, b_g2_msm, epilogue) that could, in theory, overlap with the next partition's GPU execution.

The Design Decision: Two Workers, One Mutex

The user's intuition in message 2122 sharpens the approach: "So seems like we want a dual-gpu-worker interlocked per-gpu, getting some lock/semaphore juuust before starting gpu work and releasing juuust after gpu work is done, before even b_g2_msm." The assistant confirms and immediately begins tracing the exact CUDA serialization point.

A subagent task (message 2126) traces the full call path from Rust's prove_from_assignments through the FFI boundary into the C++ CUDA code. The critical finding: a static std::mutex at groth16_cuda.cu:133 serializes the entire generate_groth16_proofs_c function. This mutex was presumably added to prevent concurrent GPU access, but it holds the lock for the full ~3.5s — including ~1.3s of CPU work that doesn't touch the GPU at all.

The design document evaluates multiple approaches for passing a finer-grained mutex into the C++ code. The recommended "Option 4" involves passing a std::mutex* from Rust through the FFI call chain, acquiring it only around the per-GPU thread launch and join (NTT → batch-add → tail-MSM), and releasing it before b_g2_msm and the epilogue. This allows two GPU workers to interleave: Worker A holds the GPU while Worker B does its CPU preamble; when A finishes its CUDA kernels and releases the lock, B immediately acquires it and launches its kernels, while A proceeds to b_g2_msm and proof serialization.

A critical verification step occurs in messages 2129–2140. The design depends on the sppark semaphore_t (used as a barrier between prep_msm and the per-GPU threads) being a counting semaphore that latches notify() before wait(). The assistant searches through the sppark source tree, eventually finding the class definition in thread_pool_t.hpp. It confirms that notify() increments a counter and signals a condition variable, while wait() blocks until the counter is positive then decrements it. This means a notify() issued before the corresponding wait() is safe — the notification is preserved even if the waiting thread is delayed by the mutex. This confirmation is added to the document before committing.

Why This Message Matters

Message 2142 is the formal handoff from investigation to implementation. It represents the moment when a performance hypothesis — "GPU is jumpy, maybe we can interleave two workers" — is transformed into a concrete, implementable specification backed by quantitative evidence and code-level analysis. The message itself is a summary, but it carries the weight of everything that preceded it: the timeline analysis that quantified 64.3% GPU efficiency, the code reading that identified the static mutex as the root cause, the call-path tracing that mapped the exact lock scope, and the semaphore verification that confirmed the design's safety.

The message also reveals the assistant's engineering philosophy: every optimization proposal must be grounded in measured data, trace the full causal chain from symptom to root cause, and specify the exact code changes required. The estimate of "~75 lines of changes across 6 files" is not a guess — it follows from the precise understanding of which files need modification and what each change entails.

Assumptions and Open Questions

The design rests on several assumptions that warrant scrutiny. First, it assumes that the static mutex at groth16_cuda.cu:133 is the only serialization point — that no other hidden serialization exists in the CUDA driver, the GPU hardware scheduler, or the memory allocator. If the CUDA driver itself serializes kernel launches from the same context, the fine-grained mutex may not help. Second, it assumes that the ~1.3s of CPU work (b_g2_msm, epilogue, proof serialization) is truly independent of the GPU state and can execute concurrently with another partition's CUDA kernels. If b_g2_msm reads GPU results that haven't fully synchronized, or if the proof serialization allocates GPU memory, hidden dependencies could emerge. Third, it assumes that two GPU workers sharing a single physical GPU via rapid lock handoff will not cause driver-level contention, context-switching overhead, or out-of-memory conditions from duplicated device allocations.

The message does not address these risks — they are implicit in the decision to proceed with the design. The real validation will come when the ~75 lines of changes are implemented and benchmarked.

Input and Output Knowledge

To fully understand this message, a reader needs familiarity with: Groth16 proof generation and its multi-scalar multiplication (MSM) and number-theoretic transform (NTT) kernels; the cuzk proving engine's architecture (partitioned pipeline, GPU workers, async/sync boundary via spawn_blocking); the Phase 7 per-partition dispatch that treats each of 10 PoRep partitions as an independent work unit; the timeline instrumentation that records GPU_START/GPU_END events; the b_g2_msm computation (a CPU-side MSM on the G2 curve, taking ~0.4s); the prep_msm_thread that preprocesses MSM inputs on the CPU; the FFI boundary between Rust's bellperson library and the C++ CUDA code in supraseal-c2; and the sppark library's semaphore_t counting semaphore.

The output knowledge created by this message is substantial. It formally documents the root cause of GPU underutilization in the Phase 7 pipeline, quantifies the optimization opportunity (~1.3s per ~3.5s job), specifies a concrete implementation plan (Option 4, ~75 lines, 6 files), and creates a permanent record in the git history (commit 71f97bc7). The design document becomes the blueprint for Phase 8, guiding the next round of implementation and benchmarking.

The Thinking Process

Although the subject message itself is a summary, the thinking process is vividly visible in the surrounding messages. The assistant demonstrates a methodical, measurement-first approach: it starts with raw timeline data, computes summary statistics, identifies anomalies, asks clarifying questions, reads source code to understand what the measurements actually mean, traces call paths through multiple layers of abstraction (Rust → FFI → C++ → CUDA), verifies critical assumptions about third-party library semantics, and only then writes the design document.

This is not a "guess and check" workflow. It is forensic performance engineering: every conclusion is backed by evidence, every assumption is tested against the code, and every design decision is evaluated against multiple alternatives. The message 2142 is the capstone of this process — the moment when analysis becomes specification.

Conclusion

Message 2142 is a masterclass in performance optimization communication. It takes a complex, multi-layered problem — GPU underutilization in a distributed proof generation pipeline — and distills it into a clear, actionable specification. The dual-GPU-worker interlock it describes is elegant in concept: use a fine-grained mutex to let two workers share a single GPU, overlapping one worker's CPU work with the other's GPU execution. But the elegance is earned through rigorous investigation. Every number in the summary (64.3% efficiency, ~2.1s CUDA time, ~1.3s CPU work, ~75 lines across 6 files) comes from direct measurement or code analysis. The message doesn't just say "we should do this" — it says "here is exactly why, here is exactly where, and here is exactly how." That is the hallmark of engineering work that can be trusted to deliver results.