The Art of the Minimal Fork: Exposing Bellperson's Hidden Pipeline for GPU Proving

Introduction

On a Linux workstation deep in the Filecoin ecosystem, an AI assistant executed a single git commit command that marked a quiet but decisive turning point in a months-long engineering project. The commit message, reproduced below, reads like a compressed design document — dense with architectural insight, memory budgets, and surgical code modifications:

feat(cuzk): Phase 2 — bellperson fork with split synthesis/GPU API

Fork bellperson 0.26.0 into extern/bellperson/ with minimal changes to
expose the synthesis/GPU split point for pipelined proving:

bellperson changes (3 files, ~130 lines changed):
- prover/mod.rs: Make ProvingAssignment struct and all fields pub
- prover/supraseal.rs: Make synthesize_circuits_batch() pub, add new
  prove_from_assignments() function (extracted GPU-phase code)
- groth16/mod.rs: Re-export ProvingAssignment, synthesize_circuits_batch,
  prove_from_assignments under cuda-supraseal feature

The internal two-phase architecture was already clean — synthesis runs
circuit.synthesize() on CPU (rayon parallel), producing ProvingAssignment
with a/b/c evaluation vectors + density trackers. GPU phase packs these
into raw pointer arrays and calls supraseal_c2::generate_groth16_proof().
We simply expose both phases as separate public functions.

cuzk workspace changes:
- Cargo.toml: Add [patch.crates-io] for bellperson fork, add bellperson
  as workspace dependency
- Cargo.lock: Updated to use local bellperson

Also includes cuzk-phase2-design.md with complete Phase 2 design:
- Per-partition pipeline strategy (13.6 GiB intermediate state instead of
  136 GiB for all 10 partitions)
- Memory budget analysis for 128 GiB vs 256 GiB machines
- SRS manager design using SuprasealParameters directly
- 7-step implementation plan
- Call chain comparison (Phase 1 monolithic vs Phase 2 pipelined)

All 8 existing cuzk tests pass. Zero new warnings from our changes.

This message is far more than a routine checkpoint. It represents the culmination of a deep architectural investigation into bellperson — the core Groth16 proving library underpinning Filecoin's proof-of-replication (PoRep) system — and the execution of a carefully considered strategy to unlock a pipelined proving architecture without rewriting the world. To understand why this message matters, one must understand the problem it solves, the constraints it navigates, and the assumptions it validates.

The Problem: Monolithic Proof Generation and 200 GiB Memory Spikes

The cuzk project (short for "CUDA Zero-Knowledge") was born from a practical observation: Filecoin's existing C2 proof generation pipeline, while functional, was architecturally monolithic. When a storage provider needed to generate a Groth16 proof for a 32 GiB sector, the system would load the entire circuit, synthesize all constraints across all 10 partitions simultaneously, and then hand the massive intermediate state to the GPU for the final proving step. The peak memory footprint hovered around 200 GiB — a figure that made multi-tenant operation on shared hardware impractical and forced operators to dedicate entire machines to single proof jobs.

Phase 0 and Phase 1 of cuzk had already built the scaffolding: a gRPC-based proving daemon with a priority scheduler, multi-GPU worker pool, support for all four Filecoin proof types (WinningPoSt, WindowPoSt, SnapDeals, and the original PoRep C2), and a benchmark suite validated against golden test data. But the fundamental architecture remained monolithic — each proof request triggered the full synthesis-to-GPU cycle as a single atomic operation.

The key insight that unlocked Phase 2 was discovered during a deep analysis of bellperson's internals (see [msg 400]). The assistant realized that bellperson already contained a clean internal separation between the CPU-bound synthesis phase — which runs circuit.synthesize() across all constraints using Rayon-based parallelization — and the GPU-bound proving phase, which packs the resulting evaluation vectors into raw pointer arrays and calls into supraseal_c2::generate_groth16_proof(). This separation existed as two internal functions: synthesize_circuits_batch() and the GPU code embedded within create_proof_batch_priority_inner. The problem was that both were private — locked inside the module, inaccessible to external consumers.

The Decision: Minimal Fork Over Major Rewrite

Faced with this situation, the assistant had a choice. One option was to rewrite the proving pipeline from scratch in cuzk-core, duplicating bellperson's logic and potentially introducing subtle correctness bugs. Another was to submit a pull request to upstream bellperson and wait for it to be reviewed, merged, and released — an unpredictable timeline incompatible with the project's delivery goals. A third was to fork bellperson with the absolute minimum changes necessary to expose the existing internal API.

The assistant chose the third path, and the reasoning is worth examining. The commit message's phrase "The internal two-phase architecture was already clean" is not merely descriptive — it is a justification. The assistant recognized that the synthesis/GPU split was not a missing feature that needed to be designed and implemented, but an existing capability that was simply invisible to external callers. The changes required were mechanical: changing struct ProvingAssignment to pub struct ProvingAssignment, changing fn synthesize_circuits_batch() to pub fn synthesize_circuits_batch(), and extracting approximately 30 lines of GPU-phase code into a new pub fn prove_from_assignments().

This is a philosophy of minimal intervention. The fork modifies exactly three source files and changes roughly 130 lines — less than 0.5% of bellperson's codebase. The assistant deliberately avoided the temptation to refactor, rename, or restructure. It left create_proof_batch_priority_inner untouched (still pub(super), visible only within the prover module), ensuring that existing callers within bellperson continued to work identically. The new prove_from_assignments function is a thin wrapper that extracts the GPU-phase logic from the monolithic function and exposes it as a standalone entry point.

The Design Document: Memory-Aware Pipeline Architecture

Before touching a single line of code, the assistant wrote a 791-line design document (cuzk-phase2-design.md) that laid out the complete Phase 2 architecture. This document was not a vague sketch — it contained concrete memory budget calculations, a per-partition pipeline strategy, an SRS manager design, and a 7-step implementation plan.

The most consequential decision documented in the design was the per-partition pipeline strategy. In the monolithic approach, all 10 partitions of a PoRep 32G proof are synthesized together, producing an intermediate state of approximately 136 GiB. By processing partitions one at a time — synthesizing partition 0, sending it to the GPU, discarding the intermediate state, then synthesizing partition 1 — the peak intermediate state drops to 13.6 GiB. This 10× reduction is the difference between needing a dedicated 256 GiB machine and being able to run multiple concurrent proof pipelines on commodity hardware.

The design document also analyzed the memory budget for 128 GiB versus 256 GiB machines, determining how many pre-synthesized partitions could be queued in each configuration. On a 256 GiB machine, the pipeline queue could hold at most one fully synthesized PoRep proof (all 10 partitions) — a finding that reinforced the wisdom of the per-partition approach, which allows finer-grained scheduling and better utilization of available memory.

The SRS Manager: Eliminating Redundant Loading

Another critical design decision concerned the Structured Reference String (SRS) parameters. In the monolithic pipeline, each proof generation loaded the SRS from disk — a 32 GiB file that takes significant time to read and parse. The Phase 2 design introduced an SRS manager that keeps the parameters resident in memory, using SuprasealParameters::new(path) directly (a public constructor) rather than going through the pub(crate) get_stacked_params() function that was inaccessible from outside the crate.

This design choice had been validated empirically in Phase 0, where the assistant measured a 20.5% speedup from SRS residency (see [msg 404]). The SRS manager extends this benefit to the pipelined architecture, ensuring that the parameters are loaded once at daemon startup and shared across all proof generations.

The Workspace Integration: Patches and Versioning

The commit also reveals the practical mechanics of integrating a local fork into an existing Rust workspace. The assistant initially set the fork's version to 0.26.0-cuzk.1, a pre-release semver tag, but discovered that Cargo's [patch.crates-io] mechanism requires the patched version to satisfy the dependency requirement exactly. Since the upstream dependency was bellperson = "0.26.0", the pre-release version 0.26.0-cuzk.1 did not match. The fix was to set the fork's version to 0.26.0 — identical to upstream — so that Cargo would accept it as a valid replacement (see [msg 422]).

This is a subtle but important detail. It means the fork intentionally uses the same version number as upstream, which could cause confusion if someone later tries to determine which version is installed. The assistant accepted this trade-off because the fork is a local development artifact, not a published crate. The version identity ensures seamless integration with the dependency resolver.

Verification and Confidence

The commit message closes with a verification statement: "All 8 existing cuzk tests pass. Zero new warnings from our changes." This is not boilerplate — it is evidence that the minimal fork strategy succeeded. The tests were run against the full workspace after the patch was applied, confirming that the fork compiles cleanly, that existing functionality is preserved, and that the new public API surfaces are compatible with the rest of the system.

The assistant also performed a manual review of visibility rules before committing. It checked that supraseal.rs's references to super::ProvingAssignment remained valid after the module was made public, and verified that create_proof_batch_priority_inner — still marked pub(super) — was not accidentally exposed to external callers. This attention to Rust's module visibility system prevented a class of compilation errors that could have derailed the implementation.

Assumptions and Their Validity

The entire Phase 2 approach rests on a critical assumption: that the synthesis phase and the GPU phase are truly separable — that the intermediate ProvingAssignment structure contains all the information needed by the GPU phase, and that no hidden state or side effect bridges the two phases. The assistant validated this assumption by reading the source code of create_proof_batch_priority_inner and confirming that it takes ProvingAssignment as input and performs no additional synthesis work. The commit message's description — "synthesis runs circuit.synthesize() on CPU (rayon parallel), producing ProvingAssignment with a/b/c evaluation vectors + density trackers. GPU phase packs these into raw pointer arrays and calls supraseal_c2::generate_groth16_proof()" — reflects this understanding.

A second assumption is that the per-partition pipeline strategy is correct — that partitions are independent and can be synthesized and proved one at a time without cross-partition dependencies. This is inherent in the Groth16 proof system's design for Filecoin PoRep, where each partition produces an independent proof that is aggregated at a higher level. The assistant verified this by examining the circuit construction in StackedCompound and confirming that each partition's circuit is self-contained.

A third assumption is that the SRS can be shared across partitions and across proof types. The design document's SRS manager assumes that SuprasealParameters is safe to hold in memory across multiple proof generations, and that concurrent access (from multiple GPU workers) is safe. The assistant did not explicitly verify thread safety in the commit, but the design document's mention of the SRS manager implies a plan to handle concurrent access, likely through reference counting or a read-only shared reference.

What This Message Creates

The commit produces two categories of output. First, it creates the bellperson fork itself — a concrete artifact at extern/bellperson/ that future phases of cuzk will depend on. The fork is not a one-time experiment; it is the foundation for the entire Phase 2 implementation. Every subsequent change to cuzk-core's prover module will call bellperson::groth16::synthesize_circuits_batch() and bellperson::groth16::prove_from_assignments().

Second, the commit creates the design document — a reference that encodes the architectural decisions, memory budgets, and implementation plan for the rest of the project. The document's 791 lines include call chain comparisons showing how the pipelined architecture differs from the monolithic one, memory calculations that justify the per-partition strategy, and a step-by-step implementation plan that the assistant will follow in subsequent rounds.

The Broader Significance

This commit represents a particular philosophy of software engineering: understand the existing code deeply, make the smallest possible change to unlock the desired capability, and verify that nothing is broken before moving forward. The assistant did not attempt to rewrite bellperson's proving logic, did not propose changes to the upstream project, and did not build a parallel implementation. It read the source, identified the seam between synthesis and GPU proving, and exposed that seam with three surgical edits.

The result is a fork that differs from upstream by approximately 130 lines — a delta small enough that it could theoretically be submitted as a pull request if the project later decides to upstream the changes. The fork is not a permanent divergence; it is a temporary scaffold that enables the pipelined architecture to be built and tested before any upstream coordination occurs.

For the cuzk project as a whole, this commit marks the transition from infrastructure (Phase 0: daemon, gRPC, scheduling; Phase 1: proof types, multi-GPU, test data) to the core architectural innovation (Phase 2: pipelined synthesis and proving). The remaining steps — implementing the per-partition pipeline in cuzk-core, integrating the SRS manager, and validating against golden test data — all depend on the API surface that this commit exposes. In the assistant's own task tracking, the "Fork bellperson and expose split API" item moved from "in_progress" to "completed," and the next items — implementing the pipelined prover in cuzk-core — became actionable.

Conclusion

The commit at f258e8c7 is a study in focused engineering. It solves a hard problem — how to decouple a tightly integrated proving pipeline — not by force but by leverage, exploiting an existing architectural seam that was already present in the code. The design document provides the strategic rationale, the fork provides the tactical capability, and the passing tests provide the confidence. For anyone studying how to approach a complex systems-engineering task, this message offers a template: understand deeply, change minimally, verify thoroughly, and document completely.