The Two-Phase Pivot: Completing Phase 1 and Architecting Phase 2 of the cuzk Proving Engine

Introduction

In the development of complex software systems, the most revealing moments are often not the grand architectural declarations but the quiet transitions between phases — the moment when one deliverable is sealed and the next begins to take shape. This article synthesizes a dense chunk of work from the cuzk proving engine project, a pipelined SNARK proving daemon for Filecoin's proof-of-replication ecosystem. The chunk spans two fundamentally different kinds of engineering work: the completion of Phase 1's final deliverable (the gen-vanilla command for generating vanilla proof test data) and the initiation of Phase 2 (a deep analysis of bellperson's internal architecture, culminating in a minimal fork that exposes the synthesis/GPU split APIs). These two halves — one about finishing, one about beginning — are connected by a common thread of disciplined engineering methodology: thorough research before implementation, minimal changes to upstream dependencies, and relentless verification at every step.

Part I: Closing Phase 1 — The gen-vanilla Command

The Last Deliverable

By the time the assistant reached this chunk, Phase 1 of the cuzk project was substantially complete. The gRPC daemon was running, all four Filecoin proof types (PoRep C2, WinningPoSt, WindowPoSt, SnapDeals) were implemented, the multi-GPU worker pool with priority scheduling was operational, and SRS residency was delivering a measured 20.5% speedup. Eight unit tests passed. But one deliverable remained: the gen-vanilla command, a utility to generate vanilla proof test data from sealed sector data on disk [3].

Vanilla proofs are the CPU-only Merkle inclusion proofs that precede the GPU-accelerated Groth16 SNARK proving phase. In Filecoin's proof architecture, the proving pipeline is split into two phases: a CPU-intensive "vanilla" phase that reads sector data from disk and constructs Merkle proofs, and a GPU-intensive "C2" phase that performs the elliptic curve operations for the SNARK. Without the gen-vanilla command, the team had golden PoRep C1 data but no way to test the WinningPoSt, WindowPoSt, or SnapDeals provers end-to-end [5].

The Research Phase

The assistant did not dive directly into implementation. Instead, it began with a strategic reconnaissance — reading four critical files to understand the current state of the codebase [4]. It examined cuzk-bench/src/main.rs to understand the CLI structure, cuzk-bench/Cargo.toml to understand the dependency boundary, the workspace root Cargo.toml for shared dependency declarations, and cuzk-core/src/prover.rs as a reference implementation for how filecoin-proofs-api functions were called.

This reading phase was followed by two parallel subagent tasks dispatched in a single round [5]. The first explored the filecoin-proofs-api function signatures for vanilla proof generation — discovering the exact signatures of generate_winning_post_sector_challenge, generate_fallback_sector_challenges, generate_single_vanilla_proof, and generate_partition_proofs. The second examined the golden test data at /data/32gbench/, revealing the 32 GiB sealed sector file, the cache directory structure, and the critical commdr.txt files containing Filecoin commitment CIDs.

The CID Parsing Challenge

One of the most instructive sub-problems in this chunk was the CID parsing decision. The golden test data stored commitment identifiers as Filecoin CID strings like bagboea4b5abcbx3jccdohrttzfneleehcpkmle4oltwuh4q3rcf5tpdodaoj6mtl — strings that encode 32-byte Merkle tree roots using a complex encoding involving multibase prefixes, varint codec identifiers, and base32 decoding [7]. The assistant needed to decode these into raw [u8; 32] commitment arrays for the upstream API functions.

The assistant initially leaned toward manual parsing to avoid adding a dependency [8]. It checked the Cargo.lock for available crates and found the cid crate absent [9]. But then came a pivotal reconsideration: "Wait — actually, re-reading the plan more carefully, I realize the cleanest approach is..." [10]. The assistant realized that filecoin-proofs-api was already going to be added as a dependency, and the cid crate was small and well-maintained. The marginal cost of adding it was negligible compared to the maintenance burden of hand-rolling CID parsing logic [11].

This decision exemplifies a mature engineering judgment: the ability to recognize when a "simpler" approach (manual parsing, no new dependencies) is actually more complex in the long run. The cid crate approach was not just simpler to implement; it was also more robust, more maintainable, and more consistent with the rest of the Filecoin ecosystem [12].

The Implementation

The implementation itself proceeded in layers. First, the dependencies were wired: filecoin-proofs-api and cid were added as optional dependencies behind a gen-vanilla feature flag in cuzk-bench/Cargo.toml [13]. This feature-gating was a deliberate architectural choice — the bench tool is primarily a lightweight gRPC client, and the heavy proving library should only be compiled when explicitly requested [14].

Then the gen_vanilla.rs module was created, containing three sub-subcommands (winning-post, window-post, snap-prove), each calling the corresponding CPU-only vanilla proof generation function from filecoin-proofs-api [15]. The module included CID parsing via the cid crate, commdr.txt file parsing, and JSON serialization of the resulting proofs as base64-encoded arrays.

The wiring into main.rs happened across three surgical edits [16][17]: adding the module import, adding the GenVanilla variant to the Commands enum, and adding the match-arm handler. Each edit was a single, focused change — a pattern that reduces risk and makes the diff history readable.

Verification and Validation

The assistant's verification discipline is worth examining in detail. After each edit, it ran cargo check to confirm compilation [18]. When a warning appeared about parse_randomness_hex being unused when the feature was disabled, the assistant fixed it immediately [19] — treating warnings as errors, not as noise to be deferred.

The feature-gated build revealed two issues: cid::Error does not implement std::error::Error (breaking anyhow::with_context usage), and there was an unused import of ChallengeSeed [22]. The assistant fixed both with a single targeted edit [23] and re-verified with a clean 0.43-second build [24].

The final validation ran the actual gen-vanilla commands against the golden test data at /data/32gbench/, producing correct outputs: a 164 KB WinningPoSt proof, a 25 KB WindowPoSt proof, and 12 MB of SnapDeals partition proofs [35]. Five new unit tests were added, and all 8 existing tests continued to pass. The milestone was committed as 9d8453c3 with the message "feat(cuzk): gen-vanilla — generate vanilla proof test data for PoSt/SnapDeals" [38].

Part II: Initiating Phase 2 — The Bellperson Fork

The Architecture of a Split

With Phase 1 complete, the assistant pivoted to Phase 2: the pipelined prover that would split Groth16 proof generation into separate CPU synthesis and GPU proving stages. The motivation was memory: the existing monolithic pipeline had a peak memory footprint of ~200 GiB, making it unsuitable for the heterogeneous cloud rental markets where Filecoin storage providers operate [46].

The assistant began by analyzing bellperson's internal architecture, discovering a critical insight: the synthesis/GPU split already existed internally [46]. Bellperson's create_proof_batch_priority_inner function already performed synthesis (circuit construction, which is CPU-bound and memory-intensive) and proving (MSM/NTT computation, which is GPU-accelerated) as separate internal steps. The problem was that these steps were not exposed as public APIs — they were hidden behind a monolithic create_proof function that did everything at once.

This discovery shaped the entire Phase 2 approach. Rather than rewriting bellperson's internals, the assistant designed a minimal fork that would expose the existing internal APIs [48]. The design document (cuzk-phase2-design.md, 791 lines) laid out a per-partition pipeline strategy that would reduce intermediate state from 136 GiB to 13.6 GiB by streaming partitions sequentially rather than processing them all at once [49][50].

The Minimal Fork Philosophy

The bellperson fork was created with approximately 130 lines of changes [58]. The modifications were surgical: making ProvingAssignment and its fields public, exposing synthesize_circuits_batch() as a public API, and adding prove_from_assignments() for the GPU phase. This is the essence of the minimal fork philosophy — change only what is necessary to expose the existing internal architecture, rather than rewriting or restructuring the upstream code [65].

The assistant read the bellperson source files carefully before making any changes [61][62], understanding the exact structure of ProvingAssignment, the flow of data through the proving pipeline, and the existing feature flags that controlled CUDA vs. CPU code paths. The fork was created at extern/bellperson/ in the cuzk workspace, and the workspace was patched via [patch.crates-io] to use the fork instead of the upstream crate [70].

The Semver Trap

One of the most instructive moments in the fork creation was the semver trap [72]. The assistant initially specified the fork's version as 0.22.1 (a patch bump from the upstream 0.22.0), but Cargo's dependency resolution rules treat version changes differently for workspace-patched crates. The fix was to keep the version at 0.22.0 and rely on the [patch] section to override — a subtle but critical detail of Cargo's dependency resolution that could have derailed the entire fork.

The fork compiled cleanly with all 8 existing tests passing, and was committed as f258e8c7 [77]. The project was now positioned to begin implementing the pipelined prover in cuzk-core, with the bellperson fork providing the necessary API surface for the synthesis/GPU split.

The Connecting Thread: Engineering Discipline

What unifies these two halves — the Phase 1 completion and the Phase 2 initiation — is a consistent engineering methodology. In both cases, the assistant:

1. Researched before implementing. Whether investigating CID parsing or bellperson internals, the assistant gathered precise information before writing code. The parallel subagent tasks for API discovery and test data examination [5] mirror the careful reading of bellperson source files before creating the fork [61].
2. Made minimal changes. The gen-vanilla implementation used feature-gated optional dependencies rather than restructuring the bench tool's architecture. The bellperson fork changed approximately 130 lines rather than rewriting the proving pipeline. Both decisions reflect a philosophy of surgical precision over wholesale restructuring.
3. Verified at every step. The assistant ran cargo check after every edit, fixed warnings immediately, and validated against golden test data. This discipline — the refusal to let even a single warning persist — is what separates robust engineering from hacking.
4. Documented decisions. The todo lists, the design document, the commit messages — each artifact captures a decision point for future reference. The Phase 2 design document at 791 lines is a particularly thorough example [49], but even the brief todo updates in messages like [6] serve as lightweight project management artifacts.

Conclusion

This chunk of the cuzk proving engine project captures a complete arc: from the final deliverable of one phase to the architectural foundation of the next. The gen-vanilla command, while seemingly a simple test utility, required navigating Filecoin's CID encoding, Rust's dependency management, feature-gating architecture, and careful validation against real 32 GiB sector data. The bellperson fork, while minimal in scope, required deep understanding of Groth16 proving internals, Cargo's patch mechanics, and the memory characteristics of the synthesis phase.

Together, these two efforts demonstrate that building production infrastructure for Filecoin proof generation is not about grand architectural gestures but about the cumulative effect of many small, correct decisions — each one researched, implemented, verified, and committed. The project now stands at the threshold of Phase 2 proper, with the bellperson fork providing the API surface needed to implement the pipelined prover that will reduce peak memory from 136 GiB to 13.6 GiB. The foundation has been laid; the pipeline awaits construction.## References

[1] "The Knowledge Synthesis: Understanding Message 351 in the cuzk Proving Engine Development" — Analysis of the comprehensive status document that preceded the gen-vanilla implementation.

[2] "The Weight of a Single Sentence: Authorization at the Crossroads of a Complex Engineering Project" — The user's authorization to proceed with Phase 1 completion.

[3] "The Pivot Point: How a Single Message Defined the Final Phase 1 Deliverable" — Message 353's declaration of the gen-vanilla command as the next step.

[4] "The Strategic Reconnaissance: Reading Four Files to Bridge Phase 1 and Phase 2" — The assistant's targeted file reads before implementation.

[5] "The Research Pivot: How Two Parallel Tasks Unlocked Phase 1's Final Deliverable" — The parallel subagent tasks for API discovery and test data examination.

[6] "The Pivot Point: How a Todo Update Captures the Transition from Research to Implementation" — The todo list update marking the transition to implementation.

[7] "Decoding Filecoin CIDs: The Critical Research Step That Made Phase 1 Possible" — The CID parsing investigation.

[8] "The Moment of Reconsideration: How a Dependency Decision Revealed the Assistant's Reasoning Process" — The initial consideration of manual CID parsing.

[9] "The Art of Dependency Management: A Case Study in CID Parsing for the cuzk Proving Daemon" — The lockfile check and dependency decision.

[10] "The Moment of Refinement: How a Single 'Wait' Reshaped the cuzk Proving Engine's Phase 1 Deliverable" — The pivot to using the cid crate.

[11] "The Dependency Decision: A Pivot Point in the cuzk Proving Engine" — The Cargo.toml edit adding optional dependencies.

[12] "The Pivot Point: From Research to Implementation in the cuzk Proving Engine" — The transition to writing the gen-vanilla module.

[13] "The Moment Phase 1 Closed: Writing the gen-vanilla Implementation" — Creation of the gen_vanilla.rs module.

[14] "The Art of the Glue: Wiring a New Subcommand into the cuzk Proving Engine" — The main.rs edits for CLI integration.

[15] "The Quiet Confirmation: How a Single 'Edit Applied Successfully' Marked a Phase Completion" — The edit confirmation for the gen-vanilla wiring.

[16] "The Smallest Commit: How a One-Line Enum Variant Completes a Feature" — Adding the GenVanilla variant to the Commands enum.

[17] "The Final Wire: How a Three-Line Edit Completed Phase 1 of the cuzk Proving Daemon" — Adding the match-arm handler.

[18] "The Compilation Check That Proves a Feature Gate Works" — The --no-default-features build verification.

[19] "The Discipline of Clean Builds: Why a Single Compiler Warning Reveals a Development Philosophy" — Fixing the unused function warning.

[20] "The Quiet Verification: A 0.30-Second Cargo Check That Closed Phase 1" — The clean build confirmation.

[21] "The Final Verification: A Build Check That Seals Phase 1" — The feature-gated build check.

[22] "The Two-Second Fix That Reveals Everything: A Case Study in Iterative Development" — Fixing the cid::Error and unused import issues.

[23] "The Quiet Fix: A Single-Edit Message in the cuzk Proving Engine" — The edit confirmation for the fixes.

[24] "The Verification That Closes the Loop: A Cargo Check as a Milestone" — The 0.43-second clean build after fixes.

[25] "The Checkpoint: When 'Clean Build, Zero Warnings' Becomes a Milestone" — The test suite run.

[26] "The Regression Check: A Pivotal Moment of Verification in the cuzk Proving Engine" — Running existing tests.

[27] "Verification as a Design Act: The gen-vanilla Help Output Check" — CLI help output validation.

[28] "The Final Validation: Verifying the gen-vanilla CLI in cuzk's Phase 1" — CLI validation.

[29] "The Final Help Check: Validating a CLI Before Moving to Phase 2" — Help output verification.

[30] "The Validation Threshold: A Milestone Transition in the cuzk Proving Engine" — The milestone transition.

[31] "The Release Build That Validated a Milestone: Compiling cuzk-bench's gen-vanilla for Production Testing" — Release build.

[32] "The Moment of Proof: Validating Phase 1 of the cuzk Proving Engine" — Running gen-vanilla against golden data.

[33] "Validation at the Threshold: Confirming Phase 1 of the cuzk Proving Engine" — Validation results.

[34] "The Final Validation: SnapDeals Vanilla Proof Generation" — SnapDeals validation.

[35] "Validation as Discovery: How a Single Bash Command Confirmed Phase 1 of the cuzk Proving Engine" — The validation command and its output.

[36] "A Milestone Confirmed: Validating Phase 1 of the cuzk Proving Engine" — Milestone confirmation.

[37] "The Commit That Closed a Phase: A Milestone Checkpoint in the cuzk Proving Engine" — The Phase 1 commit.

[38] "The Commit That Closed Phase 1: How a Single Git Message Captures a Milestone in SNARK Proving Infrastructure" — The commit message analysis.

[39] "The Todo That Closed a Chapter: How a Simple Status Update Marked the Transition Between Phases" — The todo update.

[40] "The Milestone Message: How a Simple Status Update Defines Engineering Progress" — The milestone announcement.

[41] "The Weight of 'Continue': How a Single Word Authorized a Phase Transition" — The authorization to proceed to Phase 2.

[42] "The Pivot Point: How a Three-Line Message Set the Stage for Phase 2" — The Phase 2 planning message.

[43] "The Blueprint Moment: Transitioning from Phase 1 to Phase 2 in the cuzk Proving Engine" — The transition.

[44] "The Transitional Artifact: How a Single Todo Update Marks the Boundary Between Phases" — The boundary marker.

[45] "The Pivot: How One Message Transformed a Proving Engine's Architecture" — The bellperson analysis begins.

[46] "The Critical Insight: Bellperson's Already-Internal Synthesis/GPU Split" — Discovery of the existing internal split.

[47] "The Pivot Point: From Analysis to Action in the cuzk Proving Engine" — Transition to fork creation.

[48] "The Split is Already Internal: A Pivotal Discovery in the cuzk Proving Engine" — Deep analysis of bellperson internals.

[49] "The Architecture of a Split: Designing Phase 2 of the cuzk Proving Engine" — The Phase 2 design document.

[50] "The Moment of Crystallization: How a Single Analysis Message Defined Phase 2" — Design crystallization.

[51] "The Pivot Point: From Analysis to Design in the cuzk Proving Engine" — The design pivot.

[52] "The Preparatory Glance: How a Single Read Command Anchored a Complex Design Document" — Reading bellperson source.

[53] "The Pivot Point: From Analysis to Architecture in the cuzk Proving Engine" — Architecture definition.

[54] "Precision in Engineering: Verifying Memory Budget Assumptions in the cuzk Phase 2 Design" — Memory budget analysis.

[55] "The Memory Constraint That Shaped a Pipeline Architecture" — Memory-driven design.

[56] "Message 406" — Brief message.

[57] "The Pivot Point: From Design to Implementation in the cuzk Proving Engine" — Transition to fork implementation.

[58] "The Pivot from Design to Implementation: Creating the Bellperson Fork" — Fork creation.

[59] "The Directory Listing That Preceded a Fork: Reconnaissance Before Action" — Directory examination.

[60] "The Fork That Unlocks Pipelined Proving: Bellperson at the Crossroads" — The fork's significance.

[61] "The Moment Before the Fork: Reading Before Editing in the cuzk Phase 2 Pipeline" — Reading bellperson source.

[62] "Reading Bellperson Prover Files: The Moment Before the Fork" — Reading prover files.

[63] "The Moment of Execution: Forking Bellperson for Pipelined Proving" — Fork execution.

[64] "The Pivot Point: A Single Edit That Unlocks Pipelined Groth16 Proving" — The key edit.

[65] "The Art of the Minimal Fork: Exposing Bellperson's Hidden Pipeline" — Minimal fork philosophy.

[66] "The Moment the Bellperson Fork Took Shape: Exposing Internal APIs" — Fork shaping.

[67] "The Quietest Line of Code: How a Single Edit Unlocked Pipelined SNARK Proving" — The critical line.

[68] "The Capstone Edit: Re-exporting the Bellperson Fork API" — API re-export.

[69] "The Moment of Transition: Wiring a Minimal Fork into a Proving Pipeline" — Wiring the fork.

[70] "The Pivot Point: Patching Bellperson into the cuzk Workspace" — Workspace patching.

[71] "The Version That Wasn't: A Cargo Patch Pitfall in the cuzk Proving Engine" — The version issue.

[72] "The Semver Trap: How a Version Number Derailed a Bellperson Fork" — Semver resolution.

[73] "The Moment the Fork Compiled: Validating a Minimal Bellperson Fork" — Compilation validation.

[74] "The Validation Milestone: When a Bellperson Fork Proves Its Worth" — Fork validation.

[75] "The Checkpoint That Validated a Fork: Bellperson's Minimal API Exposure" — Checkpoint.

[76] "The Threshold Moment: Committing the Bellperson Fork" — Fork commit.

[77] "The Commit That Unlocks the Pipeline: Forking Bellperson for the cuzk Proving Engine" — The fork commit.

[78] "The Art of the Minimal Fork: Exposing Bellperson's Hidden Pipeline for GPU Proving" — Minimal fork analysis.

[79] "The Silent Acknowledgment: How an Empty Message Marked a Pivotal Milestone" — The milestone acknowledgment.