The Pipeline That Split the Prover: How Per-Partition Pipelining Transformed Filecoin's SNARK Proving Engine

Introduction

In the world of decentralized storage, cryptographic proofs are the bedrock of trust. Filecoin, the largest decentralized storage network, requires its storage providers to generate Groth16 zk-SNARKs to prove they are honestly storing data. These proofs are computationally expensive — a single 32 GiB sector's Proof-of-Replication (PoRep) C2 proof consumes approximately 200 GiB of RAM and takes over 90 seconds on a modern GPU. The cost is so high that it shapes the economics of the entire network: storage providers must invest in high-memory machines, and the throughput of proof generation directly limits how much data they can seal per day.

The cuzk project set out to solve this problem by building a pipelined SNARK proving daemon — a persistent process that keeps GPU resources warm, manages multi-gigabyte parameter files in memory, and pipelines the CPU synthesis and GPU proving phases to maximize throughput. This article examines the core implementation of Phase 2 of that project, a multi-week engineering effort that replaced the monolithic PoRep C2 prover with a per-partition pipelined architecture. The work, captured across a dense sequence of messages in an opencode coding session, represents a fundamental transformation in how Filecoin proofs are generated — one that reduces peak memory by an order of magnitude and enables the pipeline to run on machines with 128 GiB of RAM instead of the 256+ GiB previously required.

The Problem: A Monolithic Prover That Wastes Both Memory and Time

To understand what Phase 2 accomplished, one must first understand the architecture it replaced. In Phase 0 and Phase 1 of the cuzk project, the proving engine operated monolithically: for each PoRep C2 proof request, the engine called seal_commit_phase2 from the filecoin-proofs-api library, which performed all steps of proof generation in a single blocking call. This meant:

  1. The GPU sat idle while the CPU synthesized circuits — for tens of seconds on a 32 GiB PoRep.
  2. All ten partitions were synthesized at once, consuming ~136 GiB of intermediate memory for the circuit assignments (a/b/c vectors and density trackers).
  3. SRS parameters were loaded lazily via a private global cache (GROTH_PARAM_MEMORY_CACHE) that could not be preloaded or evicted explicitly.
  4. There was no overlap between consecutive proofs — the GPU finished one proof entirely before the next proof's synthesis began. The performance numbers told the story. Cold SRS (first proof after daemon start) took 116.8 seconds total, including ~15 seconds for SRS loading from disk. Warm SRS (parameters cached) took 92.8 seconds — a 20.5% improvement that validated the value of parameter residency. But even warm, the GPU was active for only a fraction of that time, with the rest consumed by CPU synthesis that could not overlap with GPU work. The Phase 2 design document laid out a clear vision: split the monolithic prover into two phases — CPU circuit synthesis and GPU NTT+MSM proving — and pipeline them so that the GPU never waits. The key insight was per-partition processing. A 32 GiB PoRep is divided into 10 partitions, each with ~106 million constraints. The monolithic prover synthesized all 10 partitions together, producing ~136 GiB of intermediate state. By processing partitions one at a time, the peak intermediate memory drops to ~13.6 GiB per partition — a 10× reduction that fits comfortably within a 128 GiB machine.

The Foundation: A Minimal Bellperson Fork

Before Phase 2 could begin, a critical prerequisite had to be met: the bellperson library (the core Groth16 proving library used by Filecoin) needed to expose its internal synthesis/GPU split point. In the upstream bellperson, the create_proof_batch_priority_inner() function performed both synthesis and GPU proving in a single call, with the intermediate ProvingAssignment type kept crate-private.

The solution was a minimal fork of bellperson, created in Phase 2 steps 1-2. The fork made three changes:

The Research Campaign: Mapping the Upstream API

With the bellperson fork in place, the assistant embarked on a systematic research campaign to understand the upstream proving APIs. This was not optional browsing — it was a prerequisite for writing correct pipeline code. The assistant needed to understand exactly how seal_commit_phase2 in filecoin-proofs-api called into filecoin-proofs, which called into storage-proofs, which called into bellperson. Every function signature, every type parameter, every intermediate data structure had to be mapped before the pipeline could be built.

The research unfolded across three parallel subagent tasks:

Task 1 (msg 435) targeted the seal_commit_phase2_circuit_proofs function in filecoin-proofs — the internal function that takes already-constructed circuits and produces proofs. The subagent searched the cargo registry source files, found the function signature, and returned its generic type parameters (Tree: MerkleTreeTrait), its input arguments (commitment, replica id, prover id, sector id, etc.), and its return type.

Task 2 (msg 436) traced the full call chain from filecoin-proofs-api's seal_commit_phase2 entry point down through storage-proofs to bellperson's circuit construction layer. This revealed how the monolithic prover constructed circuits, synthesized witnesses, and handed off to the GPU for NTT and MSM operations.

Task 3 (msg 437) gathered the data types: the PoRepConfig struct (with its sector_size, partition_count, and porep_id fields), the CircuitId type that maps to .params filenames on disk, the PrivateInputs and PublicInputs types used in circuit synthesis, and the partition iteration structure.

These three tasks, completed in parallel, gave the assistant a thorough understanding of the entire proving pipeline from API entry point to GPU kernel call. The research was exhaustive — it even discovered that the existing GROTH_PARAM_MEMORY_CACHE was a private global, confirming that a new SRS manager would need to bypass it entirely.

The SRS Manager: Taking Control of Parameter Residency

The first module created in the Phase 2 implementation was srs_manager.rs. The SRS (Structured Reference String) parameters are the largest data dependency in the proving pipeline — a single PoRep 32G parameter file is 45 GiB. The existing system loaded them lazily via GROTH_PARAM_MEMORY_CACHE, an unbounded HashMap with no eviction policy that was populated on first proof call and never cleared. Worse, the functions that accessed this cache (get_stacked_params() and get_post_params()) were pub(crate) — invisible from outside the filecoin-proofs crate.

The SRS manager solved this by loading parameters directly via SuprasealParameters::new(path), bypassing the private cache entirely. This gave the engine explicit control over which parameters were resident and when they were evicted. The manager mapped CircuitId values to exact .params filenames on disk, supporting preload operations (load parameters at startup before any proof requests arrive) and evict operations (unload parameters to free memory) with memory budget tracking.

The CircuitId enum encoded the knowledge about which parameter file corresponded to which circuit type — a mapping that had previously been implicit in the private cache. For 32 GiB V1.1 sectors, the correct parameter file was the 8-8-0 variant (not 8-8-2, which is V1.2). Getting this mapping wrong would produce incorrect proofs or runtime errors.

The Pipeline Module: Per-Partition Synthesis and GPU Proving

The heart of Phase 2 was pipeline.rs, which contained the SynthesizedProof type and the two split functions: synthesize_porep_c2_partition() and gpu_prove().

SynthesizedProof was the intermediate data structure that carried the output of CPU synthesis to the GPU proving phase. It wrapped the ProvingAssignment<Scalar> type from the bellperson fork — the a/b/c evaluations and density trackers that represent the circuit's constraint system — along with the input/aux assignments and the circuit-specific SRS parameters needed for GPU proving.

synthesize_porep_c2_partition() performed the CPU-bound circuit synthesis for a single partition. It called the bellperson fork's synthesize_circuits_batch() function, which ran circuit.synthesize() in parallel via rayon across available CPU cores. The output was a SynthesizedProof containing the intermediate assignment data — approximately 13.6 GiB for a single partition of a 32 GiB PoRep.

gpu_prove() performed the GPU-bound proving phase. It took a SynthesizedProof, called the bellperson fork's prove_from_assignments() function, which packed the assignment data into raw pointers and called supraseal_c2::generate_groth16_proof() for the NTT (Number Theoretic Transform) and MSM (Multi-Scalar Multiplication) computations on the GPU. The output was a 1920-byte Groth16 proof — the final output of the proving pipeline.

The split was clean: synthesis was CPU-only and could run in parallel with other CPU work, while GPU proving was GPU-only and could run while the next partition's synthesis was in progress. This overlap was the key to the throughput improvement.

The Engine Refactoring: Wiring the Pipeline into the Daemon

With the SRS manager and pipeline module created, the engine needed to be refactored to support a pipeline.enabled configuration flag. This was a backward-compatible design: when pipeline.enabled was false (the default), the engine used the Phase 1 monolithic prover for all proof types. When true, PoRep C2 jobs were routed through the new pipeline, while PoSt and SnapDeals proofs continued to use the monolithic path.

The engine refactoring touched several files:

The Debugging Journey: From First Compilation to Clean Build

The path from the first implementation attempt to a clean compilation was not smooth. The assistant encountered and resolved numerous compilation errors, each revealing a subtle aspect of the Rust type system or the dependency graph.

The dependency wiring problem: The bellperson fork needed to be accessible from cuzk-core, but the workspace's [patch.crates-io] configuration in the root Cargo.toml was not being picked up by the individual crate. This required careful debugging of Cargo's feature flag resolution and workspace member configuration.

The feature flag mismatch: The bellperson fork used #[cfg(feature = "cuda-supraseal")] to conditionally compile the GPU proving code. The cuzk workspace needed to ensure this feature was propagated correctly from the workspace-level configuration to the bellperson dependency. Getting this wrong meant the GPU proving functions were not compiled at all, resulting in "function not found" errors.

The generic type parameter problems: The pipeline functions needed to work with the generic E: MultiMillerLoop type parameter from bellperson, but the engine's types were concrete (using Bls12 from blstrs). The assistant initially tried to propagate generics through the pipeline, then realized that concretizing the types at the pipeline boundary was simpler and avoided complex trait bounds.

The missing module declaration: After creating srs_manager.rs and pipeline.rs, the assistant forgot to declare them in lib.rs. This caused a compilation error that was quickly fixed once identified.

The import path corrections: Several functions and types had moved or been renamed between versions of the upstream crates. The assistant had to trace import paths through the dependency graph to find the correct module locations.

Each error was diagnosed and fixed iteratively: compile, read the error, identify the root cause, edit the source, compile again. The assistant's pattern of reading source files to understand type signatures before writing code paid off repeatedly — most errors were simple typos or missing imports, not architectural flaws.

The Validation: 15 Tests, Zero Warnings

The final validation was comprehensive. The assistant ran:

  1. cargo check — to verify the code compiled without GPU features (no CUDA required).
  2. cargo test — to run all unit tests in the cuzk workspace.
  3. cargo check --features cuda-supraseal — to verify the GPU-specific code compiled correctly.
  4. cargo check 2>&1 | grep -i warning — to ensure zero warnings from the cuzk crates. All 15 tests passed. Zero warnings were emitted from any cuzk crate. The code compiled cleanly both with and without the cuda-supraseal feature flag. The commit beb3ca9c captured the complete Phase 2 implementation: the SRS manager, the pipeline module, the refactored engine, the configuration support, and all the dependency wiring. The commit message documented the architecture, the key design decisions, and the status of each Phase 2 step.

The Architectural Significance: What Changed and Why It Matters

The Phase 2 implementation represents a fundamental architectural shift in how Filecoin proofs are generated. The changes can be summarized across several dimensions:

Memory efficiency: Peak intermediate memory for a 32 GiB PoRep C2 proof dropped from ~136 GiB (all 10 partitions synthesized together) to ~13.6 GiB (one partition at a time). This 10× reduction means the pipeline can run on 128 GiB machines that were previously unable to handle PoRep C2 proving.

Parameter management: The private, lazy, unbounded GROTH_PARAM_MEMORY_CACHE was replaced by an explicit SrsManager with preload/evict operations and memory budget tracking. This gives the engine fine-grained control over which parameters are resident, enabling proactive loading at startup and intelligent eviction under memory pressure.

GPU utilization: The monolithic prover serialized CPU synthesis and GPU proving, leaving the GPU idle for tens of seconds per proof. The pipeline enables overlap: while the GPU is proving partition N, the CPU can be synthesizing partition N+1. This overlap is the primary mechanism for the targeted 1.5-1.8× throughput improvement.

Backward compatibility: The pipeline is gated on a pipeline.enabled configuration flag. When disabled, the engine behaves exactly as Phase 1 — using the monolithic prover for all proof types. This allows operators to adopt the pipeline incrementally, validating its correctness and performance before committing to it fully.

Scope boundaries: The current pipeline is specifically optimized for PoRep C2. PoSt (WinningPoSt and WindowPoSt) and SnapDeals proof types still fall back to the Phase 1 monolithic prover. True cross-proof overlap — synthesizing the next job while the GPU finishes the current one — remains a future enhancement for Phase 3.

The Design Decisions That Shaped the Architecture

Several design decisions emerged during the implementation that are worth examining as lessons in systems engineering.

Per-partition pipelining over per-sector pipelining: The design could have pipelined at the sector level (synthesize one sector while proving another) or at the partition level (synthesize one partition while proving another). The partition-level choice was driven by memory: a single partition's intermediate state is ~13.6 GiB, which fits comfortably alongside the SRS parameters (~45 GiB) and the operating system on a 128 GiB machine. Sector-level pipelining would require ~136 GiB of intermediate state, which would not fit.

Explicit SRS loading over implicit caching: The decision to bypass GROTH_PARAM_MEMORY_CACHE and load parameters directly via SuprasealParameters::new() was driven by the need for explicit control. The private cache was designed for the monolithic prover's "load on first use, never evict" pattern. The pipeline needed to preload parameters at startup, evict them when memory was needed, and potentially share them across multiple concurrent pipeline instances. The explicit approach gave the engine this control.

Concrete types at the pipeline boundary: The pipeline functions could have been generic over the pairing engine type (E: MultiMillerLoop), which would have made them more reusable. The assistant chose to concretize them at the pipeline boundary using Bls12 from blstrs. This simplified the type signatures, avoided complex trait bounds, and was justified by the fact that Filecoin uses BLS12-381 exclusively.

Backward compatibility as a risk mitigation strategy: The pipeline.enabled flag was not just a convenience feature — it was a deliberate risk mitigation strategy. If the pipeline had bugs, operators could disable it and fall back to the proven Phase 1 prover. This allowed the pipeline to be deployed incrementally, reducing the blast radius of any undetected issues.

The Next Steps: From Implementation to Integration

With the Phase 2 core implementation complete, the immediate next step is end-to-end integration testing with a GPU build (--features cuda-supraseal) against the golden test data in /data/32gbench/. This testing will validate:

Conclusion

The Phase 2 implementation of the cuzk pipelined proving engine represents a significant architectural achievement. In replacing the monolithic PoRep C2 prover with a per-partition pipelined architecture, the project addressed the fundamental throughput and memory bottlenecks that limited Filecoin proof generation. The new SRS manager gives explicit control over parameter residency, the pipeline module enables overlap of CPU synthesis and GPU proving, and the backward-compatible engine design allows incremental adoption.

The work was not just about writing code — it was about understanding a complex system deeply enough to decompose it safely. The research campaign that preceded the implementation mapped the entire proving pipeline from API entry point to GPU kernel call. The bellperson fork exposed the critical split point with minimal changes. The SRS manager and pipeline modules were designed to be independent, testable, and composable. And the engine refactoring preserved backward compatibility while enabling the new architecture.

The result is a proving engine that can run on more modest hardware (128 GiB machines instead of 256+ GiB), utilize GPU resources more efficiently (overlapping synthesis and proving), and provide explicit control over parameter management (preload, evict, budget tracking). These improvements directly benefit Filecoin storage providers by reducing hardware costs and increasing sealing throughput.

The commit beb3ca9c with 15 passing tests and zero warnings is not the end of the story — it is the beginning of the next phase. The pipeline must be validated on real GPU hardware, benchmarked against the Phase 1 baseline, and extended to support cross-proof overlap and additional proof types. But the foundation is solid. The pipeline that split the prover is ready.