Phase 2 Complete: How a Pipelined SNARK Proving Engine Achieved 1.27x Throughput Through Deep Integration and Empirical Discovery

Introduction

In the demanding world of Filecoin proof generation, every second counts. The standard approach—spawning a fresh child process per proof, loading 45 GiB of SRS parameters from disk, running the proof, then exiting—wastes 30–90 seconds per proof on redundant initialization alone. The cuzk project was conceived to eliminate this waste: a persistent, GPU-resident proving daemon that accepts SNARK proof requests over gRPC, manages Groth16 SRS parameter residency in tiered memory, and schedules work across heterogeneous GPUs with priority awareness.

This chunk documents the completion of Phase 2 of the cuzk project, marking a major architectural milestone. The central achievement is the implementation of a true async overlap pipeline where CPU-bound circuit synthesis for proof N+1 runs concurrently with GPU-bound proving for proof N, mediated by a bounded tokio channel for backpressure. E2E validation on an RTX 5070 Ti with real 32 GiB PoRep data confirmed a 1.27x speedup over sequential execution, with three consecutive proofs completing in 212.7 seconds (~60s/proof steady-state throughput).

But the story of this chunk is not just about performance numbers. It is about the deep integration work required to achieve them: reverse-engineering serialization formats, forking a core dependency (bellperson) to expose private APIs, discovering that per-partition pipelining was 6.6x slower than batch synthesis, and consolidating every hard-won discovery into a comprehensive knowledge document that would serve as the foundation for Phases 3 through 5. This article examines the work documented in message 659—a 10,000+ word knowledge consolidation that captures the complete architecture of the pipelined proving engine at the moment of Phase 2 completion [1].

The Phase 2 Milestone: Async Overlap Pipeline

Phase 2's core innovation is the two-stage async overlap pipeline. The architecture is elegantly simple: a single synthesis task pulls proof requests from a priority scheduler, runs CPU-bound circuit synthesis on tokio's spawn_blocking thread pool, and pushes the resulting SynthesizedJob into a bounded mpsc channel. GPU workers pull from this shared channel, run the GPU-bound proving phase (NTT + MSM + proof assembly) with CUDA_VISIBLE_DEVICES pinning for GPU isolation, and return the final proof.

The bounded channel is the critical synchronization primitive. Its capacity is configurable via synthesis_lookahead (default 1), providing backpressure that prevents the synthesis task from producing more intermediate state than the GPU can consume. Without this backpressure, the system would quickly exhaust its ~200 GiB RSS budget—each PoRep proof's intermediate state alone occupies ~136 GiB across all 10 partitions.

The performance numbers tell a clear story:

• Single proof (pipeline mode): ~90s total (synthesis=55s, GPU=35s)
• Steady-state pipeline throughput: ~60s/proof (synthesis-bound, GPU time hidden)
• 3 proofs pipeline total: 212.7s (vs ~270s sequential = 1.27x speedup)
• Cold SRS (first proof): ~117s total (includes ~15s SRS load from disk) The 1.27x figure is conservative—it includes the first proof which has no overlap benefit (the pipeline is empty at startup). The steady-state throughput improvement is closer to 1.5x, with the GPU phase fully hidden behind synthesis for subsequent proofs.

The Critical Discovery: Batch vs Per-Partition Pipelining

One of the most important findings documented in this chunk is the discovery that per-partition pipelining is 6.6x slower than batch synthesis. The initial Phase 2 implementation attempted to pipeline at the partition level: synthesize partition 0, prove partition 0 on GPU, synthesize partition 1 while GPU proves partition 0, and so on across all 10 partitions. The intuition was that this would maximize overlap between CPU and GPU work.

The reality was dramatically different. Synthesis is highly parallelizable—rayon can synthesize all 10 partitions simultaneously using ~142 CPU cores. Per-partition pipelining serializes this parallel work, making synthesis itself 6.6x slower. The GPU, meanwhile, finishes its work faster than the serialized synthesis can keep up, resulting in no overlap benefit and a net throughput loss.

The solution was to switch to batch synthesis as the default: all 10 partitions are synthesized at once via rayon, matching monolithic performance. The pipeline's throughput win then comes from overlapping different proofs—synthesizing proof N+1's 10 partitions while the GPU proves proof N's 10 partitions—not partitions within a single proof. This finding is a textbook example of why empirical validation is essential in performance engineering: the intuitive approach (per-partition pipelining) was catastrophically wrong, and only measurement revealed the correct strategy.

The Bellperson Fork: Exposing Private APIs

Phase 2 required splitting bellperson's monolithic create_proof() function into separate synthesis and GPU phases. The stock version of bellperson (0.26.0 from crates.io) already had this split internally—synthesize_circuits_batch() ran CPU synthesis, and the GPU phase was embedded in create_proof_batch_priority_inner—but both were private APIs.

The solution was a fork of bellperson stored in extern/bellperson/, with three key changes:

1. synthesize_circuits_batch() made public: The CPU-only function that runs circuit.synthesize() in parallel via rayon
2. ProvingAssignment<Scalar> made public with all fields: The intermediate state (a/b/c evaluations, density trackers, input/aux assignments) that bridges synthesis and GPU proving
3. New prove_from_assignments() function: Extracted the GPU-phase code (NTT + MSM + proof assembly) into a standalone public function that accepts pre-synthesized assignments This fork is a pragmatic decision. Upstream contributions to bellperson would take months to review and release. By maintaining a fork, the project can move forward immediately while keeping the option to upstream the changes later. The fork is tracked in git alongside the main cuzk workspace, ensuring reproducibility.

Deep Integration: Serialization, SRS Management, and CID Parsing

The chunk reveals extensive reverse-engineering work required to integrate with the Filecoin proving stack. Three areas stand out:

The JSON-Within-JSON Serialization Format

The C1 output file (c1.json) uses a double serialization pattern that is easy to get wrong. The outer format is a Go struct serialized by encoding/json: {"SectorNum":1,"Phase1Out":"<base64>","SectorSize":34359738368}. The Phase1Out field is a []byte that Go encodes as base64. But those bytes themselves contain JSON—the Rust SealCommitPhase1Output struct serialized by serde_json. The Rust C2 FFI function expects the raw inner JSON bytes via serde_json::from_slice(), not bincode or any other format. Getting this wrong would cause opaque deserialization failures that are nearly impossible to debug without understanding the full Go→Rust FFI boundary.

Bypassing the Global SRS Cache

The GROTH_PARAM_MEMORY_CACHE in filecoin-proofs is a lazy_static Mutex<HashMap<String, Arc<Bls12GrothParams>>>—unbounded, never evicts, populated lazily on first proof call. Critically, the get_stacked_params() and get_post_params() functions that load parameters into this cache are pub(crate), meaning they cannot be called from cuzk-core. This forced a different approach for Phase 2: using SuprasealParameters::new(path) directly, bypassing the global cache entirely. This gives cuzk explicit control over SRS lifetime—a prerequisite for the tiered memory management that later phases will implement.

CID Commitment Parsing

Filecoin commitment CIDs use multibase base32lower encoding with a complex binary structure: version varint, codec varint, and a multihash containing hash type varint, digest length varint, and the 32-byte digest. The codec values distinguish sealed commitments (0xf102) from unsealed ones (0xf101), and the hash types distinguish Poseidon hashes from SHA-256. The cid crate v0.11 handles parsing, but understanding the exact byte layout was necessary to correctly extract CommR and CommD values from the CIDs used in SnapDeals proofs.

Knowledge Consolidation: The Message as Architectural Artifact

The most remarkable aspect of this chunk is not the code—it is the comprehensive knowledge consolidation document that is message 659 [1]. This 10,000+ word message, written at the natural transition point between Phase 2 completion and Phase 3 planning, captures nearly everything learned during the implementation of Phases 0 through 2. It serves multiple functions simultaneously:

• As a design document: The "Discoveries" section documents exact APIs, data formats, and architectural patterns. Someone reading it could reconstruct the entire implementation from scratch.
• As a knowledge base: Domain-specific knowledge about the Filecoin proving stack—exact param filename mappings (29 files with specific hashes), registered proof type enum values (FFI i32 to proofs-api mappings), CID parsing logic, vanilla proof serialization formats for all four proof types—that is scattered across multiple source files, Go packages, and Rust crates is consolidated into a single reference.
• As a project status report: The "Accomplished" section documents every git commit across all three completed phases, organized by phase, providing full traceability between the project plan and the implementation.
• As a performance baseline: The detailed timing breakdowns (synthesis=55s, GPU=35s, deserialization=172ms, SRS load=15s) serve as the "before" measurement for every optimization proposed in Phases 3–5.
• As a risk register: Untested paths are explicitly called out ("Pipeline implemented, needs E2E GPU test" for WinningPoSt, WindowPoSt, and SnapDeals), and known bottlenecks (the B_G2 CPU MSM, the max_num_circuits = 10 constant in groth16_srs.cuh) are documented for future work. This act of knowledge preservation is unusual in a conversational workflow—the assistant wrote it spontaneously, without being explicitly asked. It reflects a thinking process that values empirical grounding (every claim backed by specific measurements), forward-looking orientation (discoveries documented because they will matter in future phases), and risk awareness (untested paths explicitly called out).

What Remains: The Road Ahead

Phase 2 is complete, but several items remain before advancing fully:

1. E2E GPU validation for WinningPoSt, WindowPoSt, and SnapDeals: The pipeline functions for these proof types are implemented but have only been tested in CPU-only mode. GPU validation is needed to confirm they work correctly through the async overlap pipeline.
2. Phase 3: Cross-sector batching: The next major milestone, which will batch multiple sectors into a single GPU pass for 2–3x throughput improvement. This requires bumping the max_num_circuits constant in the CUDA kernel and implementing a BatchCollector that accumulates same-circuit-type requests.
3. Phase 4: Compute optimizations: SmallVec for LC evaluation, pinned memory for a/b/c vectors, parallel B_G2 MSMs.
4. Phase 5: Pre-compiled constraint evaluator: Replace circuit synthesis with a sparse MatVec operation, eliminating the synthesis bottleneck entirely. The project is in a clean, well-committed state (8 commits, branch feat/cuzk, 15 passing tests, zero warnings) and is ready to advance to Phase 3.

Conclusion

This chunk documents a pivotal moment in the cuzk project: the completion of the async overlap pipeline that forms the architectural heart of the proving engine. The 1.27x throughput improvement over sequential execution validates the approach, but the real value of this chunk lies in the knowledge it consolidates. From the JSON-within-JSON serialization format to the bellperson fork's public API surface, from the 6.6x slower per-partition finding to the exact param filename mappings, every discovery is documented with precision and context.

Message 659 [1] stands as a comprehensive reference that captures the full state of understanding about the cuzk system after three phases of implementation. It is not just a status update—it is an act of knowledge preservation, ensuring that the hard-won insights gained during months of implementation work are available to inform the optimizations yet to come.