The Dual-GPU-Worker Interlock: A Precision Insight in High-Performance Proving

In the middle of a deep optimization session for the cuzk SNARK proving engine, a single user message crystallized weeks of architectural exploration into a precise, implementable design. The message, brief and conversational, reads:

So seems like we want a dual-gpu-worker interlocked per-gpu, getting some lock/semaphore juuust before starting gpu work and releasing juuust after gpu work is done, before even b_g2_msm

This is not a casual observation. It is the distillation of a complex performance analysis into a concrete architectural proposal, delivered with the casual confidence of an engineer who has traced the exact boundaries of GPU and CPU work in a deeply layered codebase. To understand why this message matters, we must reconstruct the chain of reasoning that led to it and the engineering context that gives it weight.

The Context: Chasing GPU Utilization

The cuzk proving engine is a high-performance Groth16 proof generator for Filecoin's Proof-of-Replication (PoRep) protocol. It had recently undergone a major architectural shift in "Phase 7": instead of proving all 10 PoRep partitions in a single monolithic GPU call, each partition was now dispatched as an independent work unit flowing through a pipeline. This per-partition dispatch architecture was designed to enable cross-sector pipelining — overlapping the synthesis of one sector's partitions with the GPU proving of another's.

Initial benchmarks of Phase 7 showed promising results. The per-partition GPU calls achieved the predicted ~0.4s b_g2_msm (a CPU-side multi-scalar multiplication on the G2 curve) and throughput improved to ~45–50s per proof with concurrency 2–3. But the user observed that GPU utilization remained "pretty jumpy" ([msg 2112]), prompting a deeper investigation.

The Investigation: What the Gaps Really Mean

The assistant responded by analyzing timeline data from the daemon logs ([msg 2113]–[msg 2116]). The TIMELINE events tracked GPU_START and GPU_END for each partition's GPU call. The raw numbers revealed a GPU efficiency of only 64.3% — the GPU was idle 35.7% of the time. But the critical question, posed by the user in [msg 2118], was: "Are gaps measuring time between full-blast cuda executions or whole gpu worker job processing?"

This question forced a precise definition of what the timeline events actually measured. The assistant traced the code in engine.rs and pipeline.rs ([msg 2119]–[msg 2121]) and discovered a crucial fact: GPU_START fires before the spawn_blocking call that dispatches the blocking GPU work, while GPU_END fires inside spawn_blocking after gpu_prove() returns — which includes both CUDA kernel execution and CPU-side work like proof serialization, b_g2_msm, and malloc_trim. The gaps between GPU_END and the next GPU_START therefore included not just CUDA idle time, but also async scheduler overhead, mutex contention, channel operations, and thread dispatch latency.

More importantly, the assistant identified that within the prove_from_assignments FFI call (which bridges Rust to C++ CUDA code), the actual CUDA kernel execution is only a fraction of the total wall time. The b_g2_msm computation (~0.4s) is CPU work that happens inside the FFI call but does not use the GPU. The static std::mutex in generate_groth16_proofs_c holds for the entire ~3.5s function call, but only ~2.1s is actual CUDA kernel execution. The remaining ~1.3s is CPU preamble and epilogue that could theoretically overlap with another partition's GPU time.

The Insight: Interleaving CPU and GPU Work

The user's message in [msg 2122] is the direct response to this analysis. It contains several interlocking insights:

"dual-gpu-worker interlocked per-gpu" — The proposal is to have two worker tasks per physical GPU, rather than one. These workers are "interlocked" meaning they coordinate access to the GPU hardware using a synchronization primitive. The key architectural insight is that a single GPU can only execute one set of CUDA kernels at a time, but the CPU work surrounding those kernels (setting up pointer arrays, extracting density data, computing b_g2_msm, serializing proofs) can run in parallel with another worker's GPU kernels — if we can separate the two.

"getting some lock/semaphore juuust before starting gpu work and releasing juuust after gpu work is done" — The lock must bracket only the actual CUDA kernel execution, not the surrounding CPU work. This is the critical boundary. The "juuust" (deliberately misspelled for emphasis) signals the precision required: the lock should be held for the minimum possible duration, acquired immediately before the first CUDA kernel launch and released immediately after the last CUDA kernel completes.

"before even b_g2_msm" — This is the most technically significant detail. The b_g2_msm computation is a CPU-side multi-scalar multiplication that currently happens inside the locked region of generate_groth16_proofs_c. By releasing the lock before b_g2_msm, we allow the next worker to acquire the GPU and launch its CUDA kernels while the previous worker finishes its CPU computation. This is the key to achieving true overlap: one worker's CPU epilogue runs concurrently with another worker's GPU execution.## Assumptions Embedded in the Message

The user's message makes several implicit assumptions that reveal the depth of their understanding. First, it assumes that the CUDA kernel execution is the only truly serialized resource — that the GPU hardware itself cannot execute kernels from two different contexts simultaneously, but that CPU work can freely overlap. This is correct for a single GPU: CUDA contexts on the same device serialize kernel execution, while CPU threads can run independently on separate cores.

Second, the message assumes that the lock acquisition and release can be placed inside the C++ CUDA code, bracketing only the kernel launch region. This requires modifying the supraseal C++ library, not just the Rust orchestration layer. The user implicitly understands the FFI boundary and where the actual GPU work begins and ends within the foreign function calls.

Third, it assumes that the overhead of lock acquisition itself is negligible compared to the ~2.1s of GPU kernel time. A semaphore acquire/release pair typically costs microseconds; even with contention, it would be dwarfed by the milliseconds of GPU execution. This is a safe assumption.

Fourth, the message assumes that there is sufficient CPU work outside the CUDA kernel region to make the overlap worthwhile. The assistant's analysis had just revealed that ~1.3s of each ~3.5s generate_groth16_proofs_c call is CPU work (pointer setup, b_g2_msm, proof serialization). With two workers, one worker's 1.3s of CPU work can overlap with another worker's 2.1s of GPU work, potentially hiding the CPU overhead entirely and boosting GPU utilization from ~64% toward ~98%.

What the Message Creates: The Phase 8 Design

This single message effectively launched Phase 8 of the optimization project. The assistant immediately recognized the implications ([msg 2123]), elaborating: "Worker A does its CPU preamble (pointer setup, density extraction) while Worker B is on the GPU. When B's CUDA kernels finish, B releases the semaphore, A acquires it and immediately launches CUDA — then B does b_g2_msm + malloc_trim + assembler insert while A is on the GPU."

The assistant then traced the exact CUDA serialization point in the supraseal C++ code, verified that the semaphore_t in the sppark library is a counting semaphore with safe barrier semantics, and synthesized the findings into c2-optimization-proposal-8.md — a detailed specification for Phase 8. The document outlined the call path trace, the static mutex problem, and a recommended implementation approach (Option 4: passed mutex) requiring approximately 75 lines of changes across 6 files. The proposal promised to boost GPU efficiency from ~64% to ~98%, yielding a 3–10% throughput improvement. The document was committed to the feat/cuzk branch as 71f97bc7.

The Thinking Process: From Observation to Architecture

What makes this message remarkable is the thinking process it reveals. The user did not simply observe "GPU utilization is low" and propose "add more workers." Instead, they reasoned through the specific structure of the GPU work:

Observation: GPU utilization is "jumpy" — the GPU is active in bursts, then idle.
Hypothesis: The idle periods might be caused by per-job overhead (CPU work between GPU calls).
Refinement: The user's first message ([msg 2112]) suggested two possibilities — cutting per-job overhead, or running two interlocked GPU workers.
Investigation: The assistant confirmed that the gaps include significant CPU work (proof serialization, b_g2_msm, mutex contention, malloc_trim).
Precision question: The user asked whether gaps measured CUDA-to-CUDA or whole-job-to-whole-job ([msg 2118]), forcing a precise definition of the measurement boundary.
Root cause: The assistant traced the exact code paths and confirmed that the static mutex in generate_groth16_proofs_c holds CPU work that could be overlapped.
Architectural synthesis: The user's message in [msg 2122] combines all of this into a single coherent design — dual workers, fine-grained lock, release before b_g2_msm. This is not a linear process. It is a dialectic between measurement and architecture, where each measurement refines the understanding of what the architecture should be, and each architectural proposal suggests new measurements. The user's message is the synthesis point where the analysis crystallizes into a design.

Input Knowledge Required

To fully understand this message, one needs knowledge of: the Groth16 proving pipeline structure (partition synthesis, GPU proving, proof assembly); the CUDA execution model (kernel serialization on a single GPU, asynchronous stream behavior); the Rust async runtime (tokio's spawn_blocking, semaphore primitives); the FFI boundary between Rust and C++ CUDA code; the specific structure of the generate_groth16_proofs_c function and its static mutex; the timeline instrumentation framework; and the Phase 7 per-partition dispatch architecture. Without this context, the message reads as a vague suggestion; with it, it is a precise engineering specification.

Output Knowledge Created

The message created the specification for Phase 8 of the cuzk optimization project. It defined the exact lock boundary (bracketing only CUDA kernel execution, excluding b_g2_msm), the worker topology (two workers per GPU), the synchronization primitive (a semaphore or fine-grained mutex), and the implementation approach (modifying the C++ CUDA code to accept a passed-in mutex). This specification was then elaborated into a formal design document, committed to the repository, and became the roadmap for the next optimization cycle.

Broader Significance

The dual-GPU-worker interlock represents a shift from coarse-grained to fine-grained parallelism in the proving pipeline. Phase 7 had already broken the monolithic proof into per-partition units. Phase 8 goes further, breaking each partition's GPU work into its constituent CPU and GPU phases and overlapping them across partitions. This is the classic engineering pattern of Amdahl's law in reverse: identify the serial fraction, then find ways to parallelize it by restructuring the work boundaries.

The message also demonstrates a crucial engineering skill: knowing exactly where to place the synchronization boundary. The difference between "lock around the whole FFI call" and "lock around just the CUDA kernels" is the difference between 64% and 98% GPU utilization. That ~1.3s of CPU work that can be overlapped is the difference between a system that wastes a third of its GPU capacity and one that saturates it. The user's instinct to place the lock "juuust before starting gpu work" and release "juuust after gpu work is done, before even b_g2_msm" reflects a deep understanding of where the actual serial resource (the GPU) begins and ends.

In the broader narrative of the cuzk project, this message marks the transition from Phase 7's architectural restructuring (per-partition dispatch) to Phase 8's micro-architectural optimization (interleaving CPU and GPU work). It is the moment when the optimization focus shifts from "what work do we do" to "how do we schedule that work at the finest granularity." The dual-GPU-worker interlock is not a new algorithm or a new data structure — it is a new schedule, and that schedule is the key to unlocking the remaining GPU capacity.