The Last Mile: Wiring the Pipelined Prover into the cuzk Engine

"I need to update the blocking task to check pipeline_enabled and for PoRep use the pipeline path. The pipelined prover needs the SRS, which is loaded by the SrsManager. Let me update the worker code."

At first glance, message [msg 512] appears unremarkable — a brief note followed by a file edit that succeeds. Yet this message represents the culmination of a complex, multi-session engineering effort spanning weeks of design, analysis, and implementation. It is the moment when the Phase 2 pipelined proving architecture for the cuzk SNARK proving daemon finally becomes operational, connecting all the pieces that were built over dozens of preceding messages.

The Context: A Pipelined Revolution

To understand why this message was written, one must appreciate the scope of what preceded it. The cuzk project (short for "cuzk," a pipelined SNARK proving daemon for Filecoin) had been under development across multiple segments. Phase 0 established the gRPC scaffold and proved that a single monolithic PoRep C2 proof could be generated through the daemon. Phase 1 extended this to all four Filecoin proof types (WinningPoSt, WindowPoSt, SnapDeals) and added multi-GPU worker pool support. But the architecture remained fundamentally monolithic: each proof was synthesized and proven as a single, indivisible unit, consuming up to ~200 GiB of peak memory.

Phase 2 was designed to change this fundamentally. The key insight, documented in the earlier analysis (see [chunk 0.0]), was that the Groth16 proof pipeline could be split into two distinct phases — CPU-bound circuit synthesis and GPU-bound number-theoretic transforms (NTT) and multi-scalar multiplications (MSM) — with a bounded channel between them. For PoRep C2 specifically, the 10 partitions could be processed individually, reducing peak intermediate memory from ~136 GiB to ~13.6 GiB per partition.

The implementation plan, laid out in cuzk-phase2-design.md, consisted of seven steps. Steps 1 and 2 (bellperson fork and workspace wiring) were completed in a prior commit. Steps 3 and 4 (SRS manager and pipeline module) were implemented earlier in this segment, creating the srs_manager.rs module for direct SRS loading via SuprasealParameters and the pipeline.rs module containing the split synthesize_porep_c2_partition() and gpu_prove() functions. Step 5 — refactoring the engine to support pipeline mode — was the most complex piece, and message [msg 512] represents its final act.

The Architecture of the Engine Refactoring

The engine refactoring was approached methodically across seven edits to engine.rs (messages [msg 505] through [msg 512]). The design decisions reveal careful architectural thinking:

Preserving backward compatibility: The engine was designed to support both Phase 1 (monolithic) and Phase 2 (pipelined) modes via a pipeline.enabled configuration flag. When disabled, the existing prover::prove_porep_c2() path is used unchanged. This allows operators to switch between modes for benchmarking and gradual rollout.

Shared SRS management: The SrsManager is wrapped in Arc<Mutex<SrsManager>> and shared across all GPU workers. This is a critical design choice — SRS parameters (the 45 GiB PoRep params file) are loaded once and shared, rather than each worker loading its own copy. The SRS manager provides explicit control over parameter residency, mapping CircuitId values to exact .params filenames on disk and supporting preload/evict operations with memory budget tracking.

Worker model unchanged: The same worker-per-GPU model from Phase 1 is preserved. Each worker is bound to a specific GPU via CUDA_VISIBLE_DEVICES and runs a blocking task loop. The change is internal to the worker: when pipeline mode is enabled and the proof kind is PoRepSealCommit, the worker calls pipeline::prove_porep_c2_pipelined() instead of prover::prove_porep_c2().

SRS preloading at startup: The start() method was updated to preload SRS via the SRS manager when in pipeline mode, ensuring parameters are resident in memory before any proof requests arrive.

What Message 512 Actually Does

The edit in message [msg 512] is the final integration point. The assistant had already:

  1. Added the SrsManager to the Engine struct ([msg 506])
  2. Updated the constructor to initialize it ([msg 507])
  3. Added SRS preloading in start() ([msg 508])
  4. Passed srs_manager and pipeline_enabled into the worker spawn ([msg 510]) What remained was the worker's blocking task itself — the inner loop where proof requests are actually processed. This is the code that receives a ProofRequest, extracts the parameters (randomness, partition index, commitments), and dispatches to the prover. The edit adds a conditional branch: if pipeline_enabled is true and the proof kind is PoRepSealCommit, route through the new pipeline::prove_porep_c2_pipelined() function; otherwise, fall through to the existing monolithic path. The assistant's reasoning is explicit: "The pipelined prover needs the SRS, which is loaded by the SrsManager." This acknowledges a critical dependency chain — the pipeline module's gpu_prove() function requires loaded SRS parameters, which the SRS manager provides. Without this wiring, the pipeline module would fail at runtime when it tries to access unloaded parameters.

Assumptions and Design Trade-offs

Several assumptions underpin this message:

The pipeline module is correct: The assistant assumes that pipeline::prove_porep_c2_pipelined() works correctly — that it properly handles per-partition synthesis, manages the intermediate SynthesizedProof type, and calls gpu_prove() with the right SRS. This assumption is validated only in the next message ([msg 513]) when compilation succeeds, and in [msg 514] when all 15 tests pass.

Per-partition pipelining is sufficient: The assistant explicitly notes in [msg 504] that "true overlapping (synthesis of next proof while GPU processes current one)" would require a separate synthesis thread pool feeding a bounded channel. However, the decision is made that "per-partition pipelining approach already provides significant benefit: within a single proof, partition N+1 can be synthesized while partition N is on the GPU." This is a pragmatic trade-off — implementing true cross-proof overlap is deferred to a future phase.

PoRep C2 is the only pipeline target: The pipeline is specifically optimized for PoRep C2. PoSt and SnapDeals proof types still fall back to the Phase 1 monolithic prover. This reflects the analysis from [chunk 0.0] which identified PoRep C2 as the dominant memory consumer (~200 GiB peak) and the primary target for optimization.

The SRS manager bypasses GROTH_PARAM_MEMORY_CACHE: The SRS manager loads parameters directly via SuprasealParameters::new(param_path), bypassing the private GROTH_PARAM_MEMORY_CACHE in filecoin-proofs. This is necessary because the cache's get_stacked_params() function is pub(crate) and inaccessible from outside the crate. The assumption is that direct loading is equivalent in behavior — a reasonable assumption given that SuprasealParameters is the underlying type that the cache ultimately returns.

The Significance of a Small Edit

Message [msg 512] is, in isolation, a trivial edit — a few lines added to a conditional branch. But its significance lies in what it completes. The Phase 2 pipelined architecture, designed across multiple sessions and implemented over dozens of edits, is now fully wired. The SRS manager can load parameters. The pipeline module can synthesize partitions and prove them on the GPU. The engine can route jobs through the new path. All that remains is integration testing with real GPU hardware.

The assistant's next actions confirm this: [msg 513] checks compilation (zero errors, zero warnings from cuzk code), and [msg 514] runs the full test suite (all 15 tests pass). The Phase 2 core implementation is complete.

Knowledge Boundaries

To fully understand this message, one needs input knowledge spanning several domains: the Filecoin proof pipeline architecture (PoRep C1/C2, partitions, Groth16), the bellperson library's internal synthesis/GPU split, the supraseal CUDA backend, the cuzk engine's worker pool model, and the SRS parameter loading mechanism. The assistant draws on all of this accumulated knowledge, built up over the preceding segments and documented in the project's design documents.

The output knowledge created by this message is the integrated, working pipeline — a proving engine that can process PoRep C2 proofs partition-by-partition, reducing peak memory by an order of magnitude while maintaining the same worker model and backward compatibility. This is the foundation upon which future optimizations (cross-sector batching, PCE, micro-optimizations) will be built.

Conclusion

Message [msg 512] is a study in the nature of complex system integration. The most critical edits are often the smallest — the final wire that connects months of design and implementation into a working whole. The assistant's methodical approach — planning the refactoring, implementing each piece, testing incrementally, and finally connecting everything — exemplifies disciplined engineering. The result is a pipelined proving engine that transforms the economics of Filecoin proof generation, enabling memory-efficient operation on 128 GiB machines where previously 200+ GiB was required.