The Async Overlap Pipeline: How Phase 2 of the cuzk Proving Engine Transformed Sequential Proof Generation into a Concurrent Powerhouse

Introduction

In the world of zero-knowledge proofs on Filecoin, every second of proving time translates directly into operational cost. The cuzk project set out to build a pipelined SNARK proving daemon capable of generating Groth16 proofs for Proof-of-Replication (PoRep), WinningPoSt, WindowPoSt, and SnapDeals — the four proof types that underpin Filecoin's storage verification system. Phase 1 had established a working monolithic engine: each GPU worker would sequentially fetch a job, run CPU-bound circuit synthesis, then run GPU-bound proving, then repeat. It worked, but it left the GPU idle during synthesis and the CPU idle during proving — a classic resource utilization problem.

Phase 2 was designed to fix this. The core insight was simple but powerful: CPU-bound circuit synthesis and GPU-bound proof computation use entirely different hardware resources, so they should be able to run concurrently. If synthesis of proof N+1 could overlap with GPU proving of proof N, the pipeline could achieve higher throughput without changing the underlying cryptographic algorithms. This article traces the complete arc of Phase 2's implementation, from the batch-mode pipeline rewrite through the async overlap architecture to the final GPU validation that proved the design on real hardware.

The Foundation: Committing the Batch-Mode Pipeline

Before the async overlap could be built, the proving pipeline needed a solid foundation. The assistant began by examining the workspace state with git log and git status ([msg 607]), discovering six modified files with 918 insertions and 209 deletions — a substantial body of uncommitted work from the previous session. This was the batch-mode pipeline rewrite, which had converted the engine from per-partition sequential synthesis (process each of 10 PoRep partitions one at a time) to batch-mode synthesis (all 10 partitions synthesized together in a single rayon-parallel call, then sent to the GPU in a single pass).

The decision to commit this work before implementing async overlap was deliberate and disciplined. As the assistant noted in [msg 608], the plan was threefold: verify the code compiles, commit the current work, and then implement true async overlap. The compilation check ([msg 609]) and test run ([msg 610]) both passed cleanly — zero warnings, all 15 unit tests passing. With this validation in hand, the assistant executed the commit in [msg 612].

The commit message itself is a miniature architecture document [8]. It describes the batch-mode approach for all four proof types:

The Core Architecture: Designing the Async Overlap Pipeline

With the batch pipeline committed, the assistant turned to the core architectural challenge of Phase 2: true async overlap between CPU-bound synthesis and GPU-bound proving. The design was laid out in [msg 617] with ASCII diagrams contrasting the current and target architectures [13].

The current architecture was sequential per-worker:

Worker: pull_request → SRS_load → synthesize → GPU_prove → complete → repeat

The target architecture was a two-stage pipeline:

Synthesis task:  pull_request → SRS_load → synthesize → push to channel
                 pull_request → SRS_load → synthesize → push to channel (overlapping with GPU)
GPU worker:      pull from channel → GPU_prove → complete
                 pull from channel → GPU_prove → complete

Six explicit design decisions guided the implementation:

  1. A synthesis task pulls from the scheduler, runs synthesis on spawn_blocking, and pushes synthesized jobs into a bounded async channel. This separates job acquisition, CPU work, and inter-stage communication.
  2. Each GPU worker pulls from the channel and runs gpu_prove. The GPU worker loop is simple — it just consumes ready-to-prove jobs.
  3. Channel capacity = config.pipeline.synthesis_lookahead (default: 1). The bounded channel provides natural backpressure: if the GPU workers are busy and the channel is full, the synthesis task blocks, preventing runaway memory consumption.
  4. The synthesis task respects the shutdown signal, ensuring graceful daemon termination.
  5. SRS is loaded before synthesis via the existing SrsManager, keeping the Structured Reference String management unchanged.
  6. The pipeline is per-GPU — each GPU gets its own synthesis task + channel + GPU worker pair, with CUDA_VISIBLE_DEVICES isolation. However, the assistant immediately refined this design in [msg 618]. The per-GPU pipeline would create N synthesis tasks for N GPUs, but since PoRep C2 synthesis saturates all available CPU cores (~142 cores), multiple synthesis tasks would simply serialize on CPU resources. The refined design used a single synthesis task feeding a shared bounded channel that all GPU workers compete on. This was a critical insight: synthesis is a global CPU bottleneck that cannot benefit from parallelism, so a single synthesis task is both simpler and more efficient [14].

The Implementation: Rewriting the Engine

The implementation unfolded across a series of surgical edits to engine.rs. The assistant restructured the start() method ([msg 620]) to spawn a dedicated synthesis task and per-GPU worker tasks, connected by a tokio::sync::mpsc channel. The SynthesizedJob type was introduced as the bridge between stages, carrying the intermediate state and an SRS reference [15].

Each edit was followed by a compilation check to catch errors early. A minor fix was needed — an unused mut warning — which was promptly corrected ([msg 622]). The assistant then ran the full test suite ([msg 624]), confirming all 15 unit tests passed. A final warning check ([msg 631]) filtered for cuzk-specific warnings and confirmed zero.

The commit in [msg 634] captured the transformation: 493 insertions and 203 deletions across two files (engine.rs and cuzk.example.toml). The commit message is unusually detailed, reading more like a changelog entry than a typical one-liner [30]:

Restructure the engine to use a two-stage pipeline architecture when pipeline mode is enabled:

>

Stage 1 (synthesis task): Pulls requests from the scheduler, runs CPU-bound circuit synthesis on a blocking thread, pushes the SynthesizedJob (intermediate state + SRS ref) to a bounded channel.

>

Stage 2 (GPU workers): One per GPU, pull SynthesizedJob from the shared channel, run gpu_prove on a blocking thread pinned to their GPU via CUDA_VISIBLE_DEVICES, complete the job.

>

The bounded channel (capacity = synthesis_lookahead config, default 1) provides backpressure: when GPU workers are busy and the channel is full, the synthesis task blocks — preventing OOM from unbounded pre-synthesized proofs.

The commit also preserved the monolithic (Phase 1) path as a fallback when pipeline.enabled = false, demonstrating careful risk management.

The Moment of Truth: Validating on Real GPU Hardware

With the implementation committed, the assistant turned to the critical question: does it actually work on real hardware? The E2E GPU test began with building a CUDA release binary ([msg 638]), which required resolving a bellperson dependency issue — the fork needed to be referenced by path rather than by git revision [34]. After fixing the build, the assistant started the daemon with the new pipeline configuration and submitted proofs.

The first single-proof test ([msg 649]) completed in ~90 seconds with correct proof bytes (1920 bytes, matching the expected Groth16 BLS12-381 format). This confirmed the pipeline produced correct proofs. But the real test was the batch run: three consecutive 32 GiB PoRep C2 proofs submitted concurrently.

The results, documented in [msg 652] and [msg 653], were striking [48][49]:

The Final Verification: Clean Git State and Project Handoff

After the GPU validation, the assistant performed a systematic cleanup and verification sequence. The daemon was stopped ([msg 653]), the todo list was updated to mark all Phase 2 tasks as completed ([msg 654]), and the git state was verified ([msg 655], [msg 656]).

The git log --oneline -8 command in [msg 655] revealed the complete Phase 2 commit chain [51]:

  1. f258e8c7 — bellperson fork (split synthesis/GPU API)
  2. beb3ca9c — initial pipelined synthesis/GPU prover for PoRep C2
  3. 698c32b3 — batch pipeline for all proof types
  4. 5ba4250f — async overlap pipeline (synthesis ∥ GPU) The git status --short in [msg 656] confirmed a clean working tree — only untracked files from before the session, none from the cuzk project [52]. This was the final stamp on Phase 2: the code was committed, the tests passed, the GPU validation confirmed the speedup, and the repository was in a known good state. The session concluded with a comprehensive summary in [msg 657] that documented the validated performance, the clean git state, and the roadmap for Phases 3-5 (cross-sector batching, compute optimizations, and PCE) [53]. The user's response in [msg 658] was silence — the most powerful possible signal of satisfaction and acceptance [54].

Architectural Decisions and Their Rationale

Several key architectural decisions shaped Phase 2, each with explicit rationale:

Why a bounded channel? The bounded channel provides backpressure. PoRep C2 synthesis consumes approximately 20 GiB of intermediate state per proof. If the synthesis task could run ahead of the GPU workers unboundedly, it would accumulate synthesized proofs in memory, each holding hundreds of GiB. A bounded channel with capacity 1 (default) means at most one proof's worth of intermediate state is buffered. When the channel is full, the synthesis task blocks at the send() call, naturally throttling itself.

Why a single synthesis task? The assistant considered per-GPU synthesis tasks but rejected the idea because PoRep C2 synthesis saturates all ~142 CPU cores. Multiple synthesis tasks would simply compete for the same CPU resources, adding overhead without improving throughput. A single synthesis task feeding a shared channel is both simpler and more efficient.

Why preserve the monolithic fallback? The pipeline.enabled = false path preserves the Phase 1 architecture. This allows operators to revert to known-working behavior if the pipeline introduces issues, and it provides a clean baseline for performance comparisons. It is a textbook example of risk management in architectural evolution.

Why spawn_blocking for both stages? Circuit synthesis and GPU kernel launches are inherently blocking, CPU-bound operations. Wrapping them in spawn_blocking allows the async runtime to manage these as background tasks without occupying the async executor's threads. The GPU workers additionally pin themselves via CUDA_VISIBLE_DEVICES to ensure each worker stays on its assigned device.

Assumptions and Limitations

The Phase 2 validation, while successful, rests on several assumptions worth examining:

The benchmark is representative. The test used 3 consecutive 32 GiB PoRep C2 proofs on a single RTX 5070 Ti. This validates the architecture for the most computationally intensive proof type, but it does not test multi-GPU configurations, mixed proof types, or sustained production workloads.

The 1.27× speedup is a baseline, not a ceiling. The measured improvement is specific to the current hardware and configuration (synthesis_lookahead=1). With longer runs, the steady-state throughput would approach the synthesis-bound limit of ~55-60s per proof, yielding a speedup closer to 1.5× (90s / 60s). The first proof always pays a cold-start penalty, so the benefit increases with batch size.

CPU contention during overlap is real but manageable. The 2nd and 3rd synthesis runs were slightly slower (~60s vs ~55s) due to CPU contention with the GPU phase's finalization. On machines with fewer hardware threads, this contention could be more significant.

The remaining proof types were not tested on GPU. The WinningPoSt, WindowPoSt, and SnapDeals synthesis and GPU functions were implemented and routed through the same pipeline architecture, but only PoRep C2 was validated on real hardware. The assistant assumes these paths are correct by construction.

Conclusion

Phase 2 of the cuzk proving engine represents a fundamental architectural transformation. The engine evolved from a monolithic, sequential per-GPU worker model to a two-stage pipeline where CPU-bound synthesis and GPU-bound proving run concurrently, connected by a bounded channel that provides both communication and backpressure. The 1.27× throughput improvement on real GPU hardware validates the core insight: synthesis and GPU proving use different resources and can safely overlap in time.

The engineering process itself is a case study in disciplined development. The assistant committed the batch pipeline before implementing async overlap, creating a clean foundation. Each edit was followed by a compilation check. The full test suite was run multiple times. The GPU validation was performed with real proofs on real hardware. The git state was verified at the end. These practices — commit early, verify often, test on target hardware, leave a clean workspace — are the hallmarks of professional engineering, whether performed by a human or an AI.

With Phase 2 complete and validated, the project is ready for Phase 3 (cross-sector batching), where the async overlap architecture will be extended to batch multiple sectors' circuits together, compounding the throughput gains. The foundation laid in Phase 2 — the bounded channel, the two-stage pipeline, the synthesis-bound steady-state model — will serve as the platform for everything that follows.