Engineering the Split: How the cuzk Phase 2 Pipeline Replaced Filecoin's Monolithic Prover

Introduction

In the previous installment of this series, we examined the architectural significance of the cuzk pipelined proving engine — how per-partition pipelining transformed Filecoin's SNARK proving from a monolithic, memory-hungry process into a lean, overlapping pipeline. This article takes a different perspective. Instead of focusing on the what and why, it focuses on the how: the engineering process, the research campaign, the debugging journey, and the design decisions that shaped the Phase 2 implementation.

The work described here — captured across a dense sequence of approximately 100 messages in an opencode coding session — represents the core implementation of Phase 2 (Steps 3-7) of the cuzk project. In a single sustained session, the assistant created two new modules totaling over 900 lines of Rust, modified seven existing files, resolved dozens of compilation errors, and produced a clean commit (beb3ca9c) with 15 passing tests and zero warnings. This article reconstructs that engineering journey, extracting the patterns and decisions that made it successful.

The Starting Point: What Was Already Done

Before diving into the implementation, it is essential to understand what the Phase 2 session inherited. The cuzk project had already completed Phase 0 (scaffold with gRPC API and monolithic PoRep C2 proving) and Phase 1 (multi-GPU worker pool with all four Filecoin proof types). The foundation for Phase 2 had been laid in two critical prerequisites:

The bellperson fork (commit f258e8c7) was a minimal fork of the core Groth16 proving library that exposed three things previously kept private: synthesize_circuits_batch() (the CPU circuit synthesis function), ProvingAssignment<Scalar> (the intermediate data structure carrying a/b/c evaluations and density trackers), and a new prove_from_assignments() function (the GPU proving phase extracted from the monolithic create_proof_batch_priority_inner()). The fork was deliberately minimal — it changed nothing about the proving logic itself, only the visibility of existing internal functions and types.

The Phase 2 design document (cuzk-phase2-design.md) laid out a seven-step implementation plan. Steps 1 (bellperson fork) and 2 (wire up workspace) were complete. Steps 3-7 — the core implementation — were the target of this session.

The assistant began the session by reading the Phase 2 design document and all the existing source files it would need to modify: engine.rs, types.rs, prover.rs, config.rs, lib.rs, and the workspace Cargo.toml. This reading phase was not superficial — the assistant examined every struct definition, every function signature, every import path. Understanding the existing architecture at this level of detail was the prerequisite for making correct modifications.

The Research Campaign: Three Parallel Deep Dives

Before writing a single line of pipeline code, the assistant launched a systematic research campaign to understand the upstream proving APIs. This was not optional browsing — it was a prerequisite for writing correct code. The assistant needed to understand exactly how seal_commit_phase2 in filecoin-proofs-api called into filecoin-proofs, which called into storage-proofs, which called into bellperson. Every function signature, every type parameter, every intermediate data structure had to be mapped before the pipeline could be built.

The research unfolded across three parallel subagent tasks, each dispatched in the same round ([msg 435], [msg 436], [msg 437]):

Task 1 targeted the seal_commit_phase2_circuit_proofs function in filecoin-proofs — the internal function that takes already-constructed circuits and produces proofs. The subagent searched the cargo registry source files, found the function signature, and returned its generic type parameters (Tree: MerkleTreeTrait), its input arguments (commitment, replica id, prover id, sector id, etc.), and its return type.

Task 2 traced the full call chain from filecoin-proofs-api's seal_commit_phase2 entry point down through storage-proofs to bellperson's circuit construction layer. This revealed how the monolithic prover constructed circuits, synthesized witnesses, and handed off to the GPU for NTT and MSM operations.

Task 3 gathered the data types: the PoRepConfig struct (with its sector_size, partition_count, and porep_id fields), the CircuitId type that maps to .params filenames on disk, the PrivateInputs and PublicInputs types used in circuit synthesis, and the partition iteration structure.

These three tasks, completed in parallel, gave the assistant a thorough understanding of the entire proving pipeline from API entry point to GPU kernel call. The research was exhaustive — it even discovered that the existing GROTH_PARAM_MEMORY_CACHE was a private global, confirming that a new SRS manager would need to bypass it entirely.

The SRS Manager: Taking Control of Parameter Residency

The first module created was srs_manager.rs ([msg 447]). The SRS (Structured Reference String) parameters are the largest data dependency in the proving pipeline — a single PoRep 32G parameter file is 45 GiB. The existing system loaded them lazily via GROTH_PARAM_MEMORY_CACHE, an unbounded HashMap with no eviction policy that was populated on first proof call and never cleared. Worse, the functions that accessed this cache (get_stacked_params() and get_post_params()) were pub(crate) — invisible from outside the filecoin-proofs crate.

The SRS manager solved this by loading parameters directly via SuprasealParameters::new(path), bypassing the private cache entirely. This gave the engine explicit control over which parameters were resident and when they were evicted. The manager mapped CircuitId values to exact .params filenames on disk, supporting preload operations (load parameters at startup before any proof requests arrive) and evict operations (unload parameters to free memory) with memory budget tracking.

The CircuitId enum encoded the knowledge about which parameter file corresponded to which circuit type — a mapping that had previously been implicit in the private cache. For 32 GiB V1.1 sectors, the correct parameter file was the 8-8-0 variant (not 8-8-2, which is V1.2). Getting this mapping wrong would produce incorrect proofs or runtime errors.

The SRS manager was also the first module to deal with the cuda-supraseal feature gating. The SuprasealParameters type is only available when the cuda-supraseal feature is enabled, so the entire module was conditionally compiled. For non-CUDA builds, a stub SrsManager was provided that simply returned errors on any operation — allowing the engine to compile and test without GPU hardware.

The Pipeline Module: Per-Partition Synthesis and GPU Proving

The heart of Phase 2 was pipeline.rs ([msg 468]), containing the SynthesizedProof type and the two split functions: synthesize_porep_c2_partition() and gpu_prove().

SynthesizedProof was the intermediate data structure that carried the output of CPU synthesis to the GPU proving phase. It wrapped the ProvingAssignment<Scalar> type from the bellperson fork — the a/b/c evaluations and density trackers that represent the circuit's constraint system — along with the input/aux assignments and the circuit-specific SRS parameters needed for GPU proving.

synthesize_porep_c2_partition() performed the CPU-bound circuit synthesis for a single partition. It called the bellperson fork's synthesize_circuits_batch() function, which ran circuit.synthesize() in parallel via rayon across available CPU cores. The output was a SynthesizedProof containing the intermediate assignment data — approximately 13.6 GiB for a single partition of a 32 GiB PoRep.

gpu_prove() performed the GPU-bound proving phase. It took a SynthesizedProof, called the bellperson fork's prove_from_assignments() function, which packed the assignment data into raw pointers and called supraseal_c2::generate_groth16_proof() for the NTT (Number Theoretic Transform) and MSM (Multi-Scalar Multiplication) computations on the GPU. The output was a 1920-byte Groth16 proof — the final output of the proving pipeline.

The split was clean: synthesis was CPU-only and could run in parallel with other CPU work, while GPU proving was GPU-only and could run while the next partition's synthesis was in progress. This overlap was the key to the throughput improvement.

The Debugging Journey: From First Compilation to Clean Build

The path from the first implementation attempt to a clean compilation was not smooth. The assistant encountered and resolved numerous compilation errors, each revealing a subtle aspect of the Rust type system or the dependency graph. This section documents the major debugging episodes.

The Dependency Wiring Problem

The first compilation attempt ([msg 451]) failed with a cryptic error: storage-proofs-porep did not have the cuda-supraseal feature. The assistant had assumed that all storage-proofs crates supported this feature, but investigation revealed that only filecoin-proofs and storage-proofs-core had cuda-supraseal. The porep, post, and update crates only had cuda. The solution was to propagate features correctly: cuda-supraseal for the core crates, cuda for the proof-type-specific crates ([msg 453]).

The Feature Flag Mismatch

The bellperson fork used #[cfg(feature = "cuda-supraseal")] to conditionally compile the GPU proving code. The cuzk workspace needed to ensure this feature was propagated correctly from the workspace-level configuration to the bellperson dependency. Getting this wrong meant the GPU proving functions were not compiled at all, resulting in "function not found" errors.

The Generic Type Parameter Problems

The pipeline functions initially used generic type parameters (Tree: MerkleTreeTrait) to support multiple sector sizes. However, this created a type mismatch: SealCommitPhase1Output.replica_id is concretely PoseidonDomain (the domain type of DefaultTreeHasher), but the generic function expected <Tree::Hasher as Hasher>::Domain. Since SectorShape32GiB uses PoseidonHasher, the types were the same — but the compiler could not prove this for an arbitrary Tree.

The assistant's solution was pragmatic: eliminate the generic function and hard-code SectorShape32GiB ([msg 494]). This simplified the type signatures, avoided complex trait bounds, and was justified by the fact that Phase 2 only targets 32 GiB sectors. Support for 64 GiB sectors could be added later when needed.

The Missing Module Declaration

After creating srs_manager.rs and pipeline.rs, the assistant forgot to declare them in lib.rs. This caused the modules to be silently excluded from compilation — their tests didn't run, and their functions weren't available to the engine. The fix was a simple addition to lib.rs ([msg 459]).

The Import Path Corrections

Several functions and types had moved or been renamed between versions of the upstream crates. The assistant had to trace import paths through the dependency graph to find the correct module locations. For example, try_from_bytes was a method on the Domain trait from filecoin-hashers, but the initial code tried to call it as a standalone function. The rand_core::OsRng import path differed between versions. The filecoin_hashers::Hasher trait needed to be imported explicitly rather than accessed through filecoin_proofs.

Each error was diagnosed and fixed iteratively: compile, read the error, identify the root cause, edit the source, compile again. The assistant's pattern of reading source files to understand type signatures before writing code paid off repeatedly — most errors were simple typos or missing imports, not architectural flaws.

The Engine Refactoring: Wiring the Pipeline into the Daemon

With the SRS manager and pipeline module created, the engine needed to be refactored to support a pipeline.enabled configuration flag ([msg 500]-[msg 512]). This was a backward-compatible design: when pipeline.enabled was false (the default), the engine used the Phase 1 monolithic prover for all proof types. When true, PoRep C2 jobs were routed through the new pipeline, while PoSt and SnapDeals proofs continued to use the monolithic path.

The engine refactoring touched several files:

config.rs: A new pipeline configuration section was added with fields for enabled, max_in_flight_partitions, and srs_memory_budget. These settings controlled the pipeline's behavior at runtime.

engine.rs: The central coordinator gained a pipeline field containing the SrsManager and pipeline state. The job dispatch logic was extended with a branch that checked pipeline.enabled for PoRep C2 jobs and routed them through synthesize_porep_c2_partition() + gpu_prove() instead of the monolithic prove_porep_c2().

Cargo.toml: New dependencies were added: filecoin-proofs, storage-proofs-porep, storage-proofs-post, bellperson (the fork), blstrs, rayon, and ff.

The SRS manager was wired into the engine's startup sequence. When the engine initialized with pipeline.enabled, it created an SrsManager instance and preloaded the SRS parameters for PoRep C2 circuits. This meant the first proof request would not incur the ~15-second SRS loading penalty — the parameters were already resident in memory.

The Validation: 15 Tests, Zero Warnings

The final validation was comprehensive. The assistant ran:

  1. cargo check — to verify the code compiled without GPU features (no CUDA required).
  2. cargo test — to run all unit tests in the cuzk workspace.
  3. cargo check --features cuda-supraseal — to verify the GPU-specific code compiled correctly.
  4. cargo check 2>&1 | grep -i warning — to ensure zero warnings from the cuzk crates. All 15 tests passed. Zero warnings were emitted from any cuzk crate. The code compiled cleanly both with and without the cuda-supraseal feature flag. The commit beb3ca9c captured the complete Phase 2 implementation: the SRS manager, the pipeline module, the refactored engine, the configuration support, and all the dependency wiring. The commit message documented the architecture, the key design decisions, and the status of each Phase 2 step.

Design Decisions That Shaped the Architecture

Several design decisions emerged during the implementation that are worth examining as lessons in systems engineering.

Per-partition pipelining over per-sector pipelining: The design could have pipelined at the sector level (synthesize one sector while proving another) or at the partition level (synthesize one partition while proving another). The partition-level choice was driven by memory: a single partition's intermediate state is ~13.6 GiB, which fits comfortably alongside the SRS parameters (~45 GiB) and the operating system on a 128 GiB machine. Sector-level pipelining would require ~136 GiB of intermediate state, which would not fit.

Explicit SRS loading over implicit caching: The decision to bypass GROTH_PARAM_MEMORY_CACHE and load parameters directly via SuprasealParameters::new() was driven by the need for explicit control. The private cache was designed for the monolithic prover's "load on first use, never evict" pattern. The pipeline needed to preload parameters at startup, evict them when memory was needed, and potentially share them across multiple concurrent pipeline instances. The explicit approach gave the engine this control.

Concrete types at the pipeline boundary: The pipeline functions could have been generic over the pairing engine type (E: MultiMillerLoop), which would have made them more reusable. The assistant chose to concretize them at the pipeline boundary using Bls12 from blstrs. This simplified the type signatures, avoided complex trait bounds, and was justified by the fact that Filecoin uses BLS12-381 exclusively.

Backward compatibility as a risk mitigation strategy: The pipeline.enabled flag was not just a convenience feature — it was a deliberate risk mitigation strategy. If the pipeline had bugs, operators could disable it and fall back to the proven Phase 1 prover. This allowed the pipeline to be deployed incrementally, reducing the blast radius of any undetected issues.

The Scope Boundaries: What the Pipeline Does and Doesn't Do

The current pipeline is specifically optimized for PoRep C2. PoSt (WinningPoSt and WindowPoSt) and SnapDeals proof types still fall back to the Phase 1 monolithic prover. This was a deliberate scoping decision: PoRep C2 is by far the most memory-intensive and time-consuming proof type, consuming ~200 GiB of RAM and taking over 90 seconds on a modern GPU. PoSt proofs are much smaller (WinningPoSt ~0.45 GiB intermediate state, WindowPoSt ~16 GiB) and benefit less from pipelining.

True cross-proof overlap — synthesizing the next job while the GPU finishes the current one — remains a future enhancement for Phase 3. The current pipeline overlaps partitions within a single proof, which already provides significant benefit for PoRep (10 partitions, each taking ~9 seconds of synthesis and ~9 seconds of GPU time). The overlap reduces the per-proof wall-clock time by approximately the synthesis time of one partition, since the GPU can start proving partition 0 while partition 1 is still being synthesized.

The Next Steps: From Implementation to Integration

With the Phase 2 core implementation complete, the immediate next step is end-to-end integration testing with a GPU build (--features cuda-supraseal) against the golden test data in /data/32gbench/. This testing will validate:

Conclusion

The Phase 2 implementation of the cuzk pipelined proving engine represents a significant engineering achievement. In a single sustained session, the assistant:

References

[1] "The Pipeline That Split the Prover: How Per-Partition Pipelining Transformed Filecoin's SNARK Proving Engine" — Chunk article for Segment 8, covering the architectural significance of the Phase 2 pipeline implementation.