From Per-Partition Pipeline to Dual-GPU Interlock: The Phase 7–8 Journey in the cuzk Proving Engine

Introduction

In the relentless pursuit of throughput for Filecoin proof generation, the cuzk SNARK proving engine has evolved through a series of increasingly sophisticated optimization phases. Each phase has targeted a specific bottleneck: Phase 3 introduced cross-sector batching for a 1.46× throughput improvement; Phase 4 delivered synthesis micro-optimizations yielding 13.2% end-to-end speedup; Phase 5 implemented the Pre-Compiled Constraint Evaluator to eliminate redundant constraint evaluation; and Phase 6 introduced the slotted partition pipeline for finer-grained synthesis/GPU overlap. By the end of Phase 6, the engine had achieved impressive throughput, but the fundamental architecture remained monolithic at the partition level — all 10 partitions of a PoRep C2 proof were synthesized and proved as a single batch.

Phase 7, the subject of this segment, represented a fundamental architectural shift: treating each of the 10 PoRep partitions as an independent work unit flowing through the engine pipeline. This change promised to eliminate the "thundering herd" problem where all partitions finished synthesis simultaneously and then queued for the GPU, and to enable true cross-sector pipelining where one proof's partitions could be proving on the GPU while another proof's partitions were still being synthesized on CPU cores.

This article traces the complete lifecycle of Phase 7 — from the architecture handoff that defined its specification, through the meticulous implementation across data structures and dispatch logic, to the benchmarks that validated its correctness and revealed its limitations. It then follows the diagnostic investigation that transformed a qualitative observation about "jumpy" GPU utilization into a precise, quantified metric — 64.3% GPU efficiency — and traces the root cause through layers of abstraction to a static std::mutex in the C++ CUDA layer. That diagnosis crystallized into the Phase 8 design: a dual-GPU-worker interlock promising to boost efficiency to ~98%. Both the Phase 7 implementation and the Phase 8 design document were committed to the feat/cuzk branch, marking two milestones in the engine's optimization journey.

The Architecture Handoff: Setting the Stage for Phase 7

The segment opens with a comprehensive architecture handoff message ([msg 2024]) that serves as both a summary of discoveries and a specification for what comes next. This message, analyzed in depth in [1], contains critical timing data that motivated the Phase 7 redesign. Per-partition synthesis takes 29–36 seconds, while per-partition GPU proving takes approximately 3 seconds with num_circuits=1 — achieving the predicted ~0.4s b_g2_msm that had been a target since Phase 6. In contrast, batch-all GPU proving takes 27 seconds with a 25-second b_g2_msm bottleneck, confirming that the monolithic approach serializes GPU work inefficiently.

The handoff documents the "thundering herd" problem in vivid detail: all 10 partitions start and finish synthesis simultaneously, leaving the GPU idle for 29–39 seconds before being slammed with all partitions at once. This burst of GPU work then completes in ~27 seconds, but the GPU utilization pattern is far from optimal — long idle periods followed by intense activity, rather than the steady, continuous utilization that maximizes throughput.

The Phase 7 specification, formalized as c2-optimization-proposal-7.md, lays out a six-step plan covering data structure changes, dispatch refactoring, GPU worker routing, error handling, benchmarking, and SnapDeals support. This document becomes the blueprint for the implementation that follows, representing the distillation of months of optimization work into a concrete, actionable plan. The architecture handoff is a critical moment of knowledge transfer — the point at which analysis crystallizes into engineering action.

Implementing Phase 7: From Specification to Running Code

The implementation of Phase 7 unfolds across dozens of messages, each representing a deliberate step in the six-step plan. The agent systematically extends data structures — adding partition_index, total_partitions, and parent_job_id fields to SynthesizedJob ([msg 2036], [msg 2037]), creating the PartitionedJobState struct ([msg 2038]), and adding an assemblers map to JobTracker. Each change is small and focused, but together they create the foundation for per-partition routing.

The dispatch refactoring in process_batch() represents the heart of Phase 7. The agent replaces the old monolithic dispatch with a semaphore-gated pool of 20 spawn_blocking workers ([msg 2047], [msg 2048]). When partition_workers > 0 and a single-sector PoRep C2 arrives, the new path parses the C1 output once, registers a ProofAssembler, and dispatches 10 independent tasks — each acquiring a semaphore permit, running synthesize_partition() via spawn_blocking, and sending results to the GPU worker channel. The dispatch is non-blocking: process_batch() returns immediately, allowing the engine to accept new work while partitions are still being synthesized.

The GPU worker routing adds partition-aware logic after gpu_prove() returns ([msg 2062], [msg 2066]). Instead of delivering a complete proof directly, the worker now checks the partition's parent_job_id, routes the partition proof to the corresponding ProofAssembler, and only delivers the final assembled proof when is_complete() returns true. A malloc_trim(0) call after each partition releases memory back to the system, preventing the memory footprint from growing unboundedly across multiple proofs.

Error handling is integrated throughout: a failed flag on PartitionedJobState allows synthesis or GPU failures to propagate, and subsequent partitions for a failed job skip their work. The Phase 6 slotted pipeline is retained as a fallback for partition_workers=0, slot_size>0, ensuring backward compatibility.

The implementation culminates in a clean compilation and a commit that changes 4 files with 578 insertions and 22 deletions ([msg 2086]). The commit message — "feat(cuzk): Phase 7 — engine-level per-partition pipeline" — marks the moment when the architecture becomes reality.

Benchmarking Phase 7: Validation and Discovery

With Phase 7 committed, the user's instruction is simple: "Do some test runs!" ([msg 2089]). The agent methodically prepares for benchmarking — checking available test configurations, verifying binaries, rebuilding the bench tool, creating a Phase 7-specific configuration with partition_workers=20, and starting the daemon after waiting for SRS preloading.

The single-proof latency test ([msg 2106]) confirms the core mechanism works correctly: 72.8 seconds total, with 38.8 seconds of GPU prove time spread across 10 partitions averaging ~3.9 seconds each. Each GPU call uses num_circuits=1, achieving the predicted fast b_g2_msm. The final proof is correctly assembled at 1920 bytes (10 × 192), confirming that the ProofAssembler correctly concatenates the partition proofs.

The timeline analysis ([msg 2109]) reveals the pipeline dynamics in detail. All 10 partitions synthesize in parallel (~33–35.5 seconds), then GPU-prove sequentially (~39 seconds total). The first GPU partition takes 5.8 seconds due to CPU contention with still-running synthesis workers, but subsequent partitions run cleanly in 3.3–4.0 seconds. This is the first hint of the inefficiency that will drive Phase 8 — the GPU is not continuously saturated because CPU-side work (synthesis completion, proof serialization, memory management) creates gaps between GPU calls.

The multi-proof throughput tests ([msg 2110], [msg 2111]) show significant improvement: ~45–50 seconds per proof wall-clock time at concurrency 2–3, compared to the single-proof latency of 72.8 seconds. The cross-sector pipelining is working — while one proof's partitions occupy the GPU, another proof's partitions are being synthesized on CPU cores. But the variance is telling: proof times range from 114.5s to 177.2s in the j=3 run, a 55% spread that indicates systemic instability. The GPU utilization is "pretty jumpy" as the user observes — sometimes saturated, sometimes idle.## The Diagnostic Pivot: From "Jumpy" to 64.3%

The user's observation in [msg 2112] — "looks like gpu use is pretty jumpy, generally high, but maybe there is some per-job overhead that could be cut? Or we could run two gpu-workers interlocked" — is the pivot point of the entire segment. This single message transforms the conversation from celebration of Phase 7's success into deep diagnostic investigation that will define Phase 8.

The agent responds with disciplined measurement ([msg 2113]), extracting precise GPU_START and GPU_END timestamps from the timeline logs. The awk script in [msg 2114] computes inter-partition gaps with millisecond precision, revealing a heterogeneous landscape: some gaps are near-instantaneous (12–29ms), while others stretch to 988ms or even 32.8 seconds (the cross-proof cold-start gap). The refined analysis in [msg 2115] quantifies GPU efficiency at 64.3% — meaning the GPU is idle for more than a third of the total benchmark time.

The 64.3% figure becomes the north star for the next optimization phase. It means that for every 10 seconds of benchmark time, the GPU does useful work for only 6.4 seconds. The remaining 3.6 seconds are pure waste — the most expensive resource in the system sitting idle while the CPU does other work. But the assistant doesn't stop at the headline number. The gap distribution tells a nuanced story: 87 out of 109 gaps are under 50ms (the pipeline flowing smoothly for most transitions), 14 are between 50–500ms (moderate stalls), and 8 exceed 500ms (large outliers). The two largest gaps — 125,976ms and 54,117ms — are cross-sector synthesis stalls where the GPU finishes all partitions for one proof and must wait for the next proof's synthesis to complete. The remaining 200–700ms gaps represent per-partition overhead: the CPU work that happens between one partition's GPU completion and the next partition's GPU launch.

This classification is not arbitrary. It reflects the assistant's understanding that different gap sizes have different root causes and would require different solutions. The synthesis stalls might be addressed by increasing synthesis_lookahead or overlapping synthesis across proofs. The per-partition overhead requires understanding exactly what the CPU is doing between GPU calls — which leads to the deeper investigation.

The Precision Question: What Do the Gaps Actually Measure?

The user's clarifying question in [msg 2118] is the pivot point: "Are gaps measuring time between full-blast cuda executions or whole gpu worker job processing?" This distinction is critical. If the gaps measure only CUDA kernel idle time, the fix is one thing. If they measure whole-job processing — including CPU preamble, proof serialization, and scheduler overhead — the optimization strategy changes entirely.

The assistant investigates by reading the exact locations of the GPU_START and GPU_END events in the engine code ([msg 2121]). The discovery is revealing: GPU_START fires before spawn_blocking even schedules the blocking thread — it's on the async side, right before dispatching. GPU_END fires inside spawn_blocking, after gpu_prove() returns, which includes prove_from_assignments plus proof serialization. This means the measured gaps include everything: async scheduler overhead, mutex contention on the tracker, malloc_trim(0), channel mutex acquisition, span construction, and thread pool dispatch.

Even more critically, the assistant discovers that the GPU wall time itself contains significant CPU work. The generate_groth16_proofs_c function — the FFI call into C++ CUDA code — holds a static std::mutex for its entire ~3.5-second duration, but only ~2.1 seconds of that is actual CUDA kernel execution. The remaining ~1.3 seconds is CPU work: preprocessing (pointer array setup, density extraction), the b_g2_msm computation (~0.4 seconds of CPU-side multi-scalar multiplication on the G2 curve), proof serialization, and background deallocation thread spawning.

This is the root cause. The static mutex serializes the entire proving pipeline, preventing any overlap between one partition's CPU work and another partition's GPU execution. The GPU is idle not because there is no work to do, but because the mutex forces a strict sequential ordering: CPU work → GPU kernels → CPU work → GPU kernels → ... The GPU spends ~1.3 seconds of every ~3.5-second proving cycle waiting for CPU work to complete, even though those CPU operations could theoretically run concurrently with another partition's GPU kernels.

The Dual-GPU-Worker Interlock: A Design Emerges

The user's insight in [msg 2122] sharpens the approach: "So seems like we want a dual-gpu-worker interlocked per-gpu, getting some lock/semaphore juuust before starting gpu work and releasing juuust after gpu work is done, before even b_g2_msm."

The assistant immediately recognizes the architecture. Two GPU worker tasks per physical GPU would share a fine-grained mutex that brackets only the CUDA kernel region. While Worker A holds the mutex and runs CUDA kernels (NTT → batch-add → tail-MSM), Worker B performs its CPU preamble (pointer setup, density extraction). When Worker A's kernels finish, it releases the mutex, Worker B acquires it and immediately launches CUDA kernels — then Worker A performs its CPU postamble (b_g2_msm, proof serialization, malloc_trim) while Worker B is on the GPU. This interleaving could theoretically boost GPU efficiency from 64% to over 98%.

But the design requires precise knowledge of the call path. The assistant dispatches a subagent task in [msg 2126] to trace the full path from Rust's prove_from_assignments through the FFI boundary into the C++ CUDA code, returning exact line numbers for each function. The task confirms the critical bottleneck: the static std::mutex at groth16_cuda.cu:133 serializes the entire generate_groth16_proofs_c function.

The Semaphore Verification: Confirming the Foundation

Before committing the design document, the assistant pauses to verify one critical assumption. The proposed interlock depends on the existing semaphore_t barrier mechanism in the CUDA code — used to coordinate between the prep_msm_thread (CPU preprocessing) and the per-GPU threads (CUDA kernel launches). The design requires that notify() be callable before wait() — that the semaphore can "latch" a notification so that a later wait() will see it immediately.

The grep session that follows ([msg 2129] through [msg 2139]) is a beautiful example of open-source detective work. The assistant searches for barrier and semaphore_t across the entire codebase, tracing the definition from groth16_cuda.cu through groth16_srs.cuh and finally to sppark/util/thread_pool_t.hpp. There, at line 25, it finds the class definition: a counting semaphore with a counter, a std::mutex, and a std::condition_variable. The notify() method increments the counter and wakes one waiter; the wait() method blocks until the counter is positive, then decrements it.

This confirmation is critical. If semaphore_t were a one-shot event (like std::promise/std::future), the notify() before wait() pattern could lose the notification. But because it is a counting semaphore, the notification is stored in the counter and persists until consumed. The design is safe.

Formalizing the Diagnosis: The Phase 8 Design Document

The Phase 8 design document, c2-optimization-proposal-8.md, is committed as 71f97bc7 on the feat/cuzk branch ([msg 2141]). The commit message captures the three key findings:

The static mutex problem: The mutex in groth16_cuda.cu covers the entire ~3.5-second function, but actual CUDA kernel time is only ~2.1 seconds. The remaining ~1.3 seconds of CPU work could overlap with another partition's GPU execution.
The semaphore verification: The semaphore_t in sppark is a counting semaphore that latches notify() before wait(), confirming safe barrier semantics for the proposed restructuring.
The recommended approach: Option 4 (passed mutex), requiring approximately 75 lines of changes across 6 files. Pass a std::mutex* from Rust through FFI, acquire it before per-GPU thread launch, release after per-GPU thread join, leaving b_g2_msm and epilogue outside the lock. The estimated impact: GPU efficiency from ~64% to ~98%, with a 3–10% throughput improvement on top of Phase 7.

Engineering Principles in Action

Several engineering principles emerge from this segment that are worth articulating explicitly.

Measure before optimizing. The assistant does not start implementing the dual-worker interlock based on the user's hunch. It first quantifies the problem (64.3% efficiency), then classifies the gaps by size and root cause, then traces the exact code paths responsible. Every design decision is grounded in measured data.

Understand what your measurements actually mean. The critical insight comes from understanding that the GPU_START/GPU_END gaps measure whole-job processing time, not pure CUDA idle time. The assistant reads the source code to determine exactly where those events fire relative to the actual CUDA kernel execution. Without this understanding, the optimization would have targeted the wrong metric.

Verify assumptions before committing. The semaphore verification — a simple grep followed by reading the class definition — prevents a potentially flawed design from being committed. The assistant could have assumed the barrier semantics were correct and moved on. Instead, it takes the time to verify, confirming that the entire Phase 8 architecture rests on a solid foundation.

Document the diagnosis, not just the solution. The Phase 8 design document does not just describe what to change — it traces the full call path, identifies the exact lock points, quantifies the optimization opportunity, and evaluates multiple implementation approaches. This documentation ensures that the design can be reviewed, discussed, and implemented correctly by anyone who reads it.

Conclusion

The transition from Phase 7 to Phase 8 in the cuzk proving engine represents a model of disciplined performance engineering. It began with a qualitative observation ("GPU use is pretty jumpy"), transformed it into a quantified metric (64.3% efficiency), traced the root cause through multiple layers of abstraction (from Rust async to C++ mutex to CUDA kernel), designed a precise intervention (the dual-GPU-worker interlock), verified the enabling technology (the counting semaphore), and formalized the plan in a committed design document.

The 64.3% threshold was not just a number — it was the catalyst that drove an entire optimization phase. It validated the user's intuition, quantified the opportunity, and provided a baseline for measuring improvement. The Phase 8 design, promising to boost efficiency to ~98%, is a direct response to the gap analysis performed in those early diagnostic messages.

In the broader narrative of the cuzk project, this segment represents the pivot point between architecture implementation (Phase 7) and fine-grained concurrency optimization (Phase 8). It demonstrates that in high-performance systems engineering, the most impactful tool is often not a complex profiler or a specialized analyzer, but a well-crafted command-line pipeline that asks the right questions of the right data — followed by the discipline to verify every assumption before committing to a design.

The Phase 7 implementation is complete and committed. The Phase 8 design is specified and ready. The proving engine continues its march toward the theoretical limit of continuous, memory-efficient, GPU-saturated proof generation. Each optimization cycle produces not just better performance but deeper understanding of the system's behavior — and that understanding, captured in design documents and commit messages, is the lasting contribution of this work.