The Two-Phase Pivot: Closing Phase 1 and Architecting Phase 2 of the cuzk Proving Engine

Introduction

In the lifecycle of any ambitious software project, the transition between phases is the moment of greatest risk and greatest opportunity. It is the point where momentum can stall or accelerate, where architectural decisions made in one phase either enable or constrain the next. This article examines a pivotal segment of work from the cuzk proving engine project — a pipelined SNARK proving daemon for Filecoin's proof-of-replication (PoRep) ecosystem — that captures exactly such a transition. In a single sustained session, the assistant completed the final deliverable of Phase 1 (the gen-vanilla command for generating vanilla proof test data) and laid the architectural foundation for Phase 2 (a minimal bellperson fork exposing the synthesis/GPU split APIs, backed by a comprehensive 791-line design document).

What makes this segment particularly instructive is not the volume of code produced — the gen-vanilla command was a few hundred lines, and the bellperson fork was approximately 130 lines of changes — but the engineering methodology that guided each decision. The assistant did not rush to implementation. Instead, it researched thoroughly before writing code, made minimal changes to upstream dependencies, verified at every step with relentless discipline, and documented decisions for future reference. This article synthesizes both halves of the segment, examining the technical details, the decision-making process, and the connecting thread of disciplined engineering that unifies them.

Part I: Closing Phase 1 — The gen-vanilla Command

The Final Deliverable

By the time the assistant reached this segment, Phase 1 of the cuzk project was substantially complete. The gRPC daemon was operational, all four Filecoin proof types (PoRep C2, WinningPoSt, WindowPoSt, SnapDeals) were implemented, the multi-GPU worker pool with priority scheduling was running, and SRS (Structured Reference String) residency was delivering a measured 20.5% speedup. Eight unit tests passed. But one deliverable remained on the checklist: the gen-vanilla command, a utility to generate vanilla proof test data from sealed sector data on disk.

Vanilla proofs are the CPU-only Merkle inclusion proofs that precede the GPU-accelerated Groth16 SNARK proving phase. In Filecoin's proof architecture, the proving pipeline is split into two conceptual stages: a CPU-intensive "vanilla" phase that reads sector data from disk and constructs Merkle proofs, and a GPU-intensive "C2" phase that performs the elliptic curve operations for the SNARK. Without the gen-vanilla command, the team had golden PoRep C1 data but no way to test the WinningPoSt, WindowPoSt, or SnapDeals provers end-to-end. The command was the missing link in the test data generation pipeline.

Strategic Reconnaissance

The assistant did not dive directly into implementation. Instead, it began with a deliberate research phase, reading four critical files to understand the current state of the codebase. It examined cuzk-bench/src/main.rs to understand the CLI structure, cuzk-bench/Cargo.toml to understand the dependency boundary, the workspace root Cargo.toml for shared dependency declarations, and cuzk-core/src/prover.rs as a reference implementation for how filecoin-proofs-api functions were called. Each file read was targeted and purposeful — not random browsing but focused reconnaissance.

This reading phase was followed by two parallel subagent tasks dispatched in a single round. The first explored the filecoin-proofs-api function signatures for vanilla proof generation, discovering the exact signatures of generate_winning_post_sector_challenge, generate_fallback_sector_challenges, generate_single_vanilla_proof, and generate_partition_proofs. The second examined the golden test data at /data/32gbench/, revealing the 32 GiB sealed sector file, the cache directory structure, and the critical commdr.txt files containing Filecoin commitment CIDs. These parallel investigations allowed the assistant to gather all the information needed for implementation in a single round, demonstrating an efficient use of the subagent model.

The CID Parsing Decision

One of the most instructive sub-problems in this segment was the CID parsing decision. The golden test data stored commitment identifiers as Filecoin CID strings — opaque strings like bagboea4b5abcbx3jccdohrttzfneleehcpkmle4oltwuh4q3rcf5tpdodaoj6mtl that encode 32-byte Merkle tree roots using a complex encoding involving multibase prefixes, varint codec identifiers, and base32 decoding. The assistant needed to decode these into raw [u8; 32] commitment arrays for the upstream API functions.

The assistant initially leaned toward manual parsing to avoid adding a dependency. It checked the Cargo.lock for available crates and found the cid crate absent. But then came a pivotal reconsideration: the assistant realized that filecoin-proofs-api was already going to be added as a dependency, and the cid crate was small, well-maintained, and purpose-built for exactly this task. The marginal cost of adding it was negligible compared to the maintenance burden of hand-rolling CID parsing logic.

This decision exemplifies a mature engineering judgment: the ability to recognize when a "simpler" approach (manual parsing, no new dependencies) is actually more complex in the long run. The cid crate approach was not just simpler to implement; it was also more robust, more maintainable, and more consistent with the rest of the Filecoin ecosystem. The assistant's willingness to reconsider an initial instinct in favor of a better solution is a hallmark of experienced engineering.

Layered Implementation

The implementation itself proceeded in layers, each building on the previous one. First, the dependencies were wired: filecoin-proofs-api and cid were added as optional dependencies behind a gen-vanilla feature flag in cuzk-bench/Cargo.toml. This feature-gating was a deliberate architectural choice — the bench tool is primarily a lightweight gRPC client, and the heavy proving library should only be compiled when explicitly requested. The feature flag ensures that developers who don't need vanilla proof generation don't pay the compilation cost.

Then the gen_vanilla.rs module was created, containing three sub-subcommands (winning-post, window-post, snap-prove), each calling the corresponding CPU-only vanilla proof generation function from filecoin-proofs-api. The module included CID parsing via the cid crate, commdr.txt file parsing, and JSON serialization of the resulting proofs as base64-encoded arrays. Each sub-subcommand followed the same pattern: parse the commitment from the CID file, read the sector data, call the API function, and serialize the result.

The wiring into main.rs happened across three surgical edits: adding the module import, adding the GenVanilla variant to the Commands enum, and adding the match-arm handler. Each edit was a single, focused change — a pattern that reduces risk and makes the diff history readable. The assistant never made sweeping changes; every modification was targeted and verifiable.

Verification Discipline

The assistant's verification discipline is worth examining in detail. After each edit, it ran cargo check to confirm compilation. When a warning appeared about parse_randomness_hex being unused when the feature was disabled, the assistant fixed it immediately — treating warnings as errors, not as noise to be deferred. This is a philosophy that pays dividends: a warning today is a bug tomorrow.

The feature-gated build revealed two issues: cid::Error does not implement std::error::Error (breaking anyhow::with_context usage), and there was an unused import of ChallengeSeed. The assistant fixed both with a single targeted edit and re-verified with a clean 0.43-second build. The speed of these fixes is notable — each was identified and resolved within seconds of the build completing, demonstrating a tight feedback loop between compilation and correction.

The final validation ran the actual gen-vanilla commands against the golden test data at /data/32gbench/, producing correct outputs: a 164 KB WinningPoSt proof, a 25 KB WindowPoSt proof, and 12 MB of SnapDeals partition proofs. Five new unit tests were added, and all eight existing tests continued to pass. The milestone was committed as 9d8453c3 with the message "feat(cuzk): gen-vanilla — generate vanilla proof test data for PoSt/SnapDeals." Phase 1 was complete.## Part II: Initiating Phase 2 — The Bellperson Fork

The Architecture of a Split

With Phase 1 complete, the assistant pivoted to Phase 2: the pipelined prover that would split Groth16 proof generation into separate CPU synthesis and GPU proving stages. The motivation was memory. The existing monolithic pipeline had a peak memory footprint of approximately 200 GiB, making it unsuitable for the heterogeneous cloud rental markets where Filecoin storage providers operate. Storage providers need to rent GPU instances from cloud providers, and the cost of a machine with 200+ GiB of RAM is substantially higher than one with 32 GiB. Reducing peak memory was not just an engineering optimization — it was a business requirement for the economic viability of Filecoin storage.

The assistant began by analyzing bellperson's internal architecture, and the discovery was transformative: the synthesis/GPU split already existed internally. Bellperson's create_proof_batch_priority_inner function already performed synthesis (circuit construction, which is CPU-bound and memory-intensive) and proving (MSM/NTT computation, which is GPU-accelerated) as separate internal steps. The problem was that these steps were not exposed as public APIs — they were hidden behind a monolithic create_proof function that did everything at once, allocating all intermediate state before releasing any of it.

This discovery shaped the entire Phase 2 approach. Rather than rewriting bellperson's internals — a massive undertaking that would require deep understanding of Groth16 proving, elliptic curve cryptography, and GPU kernel programming — the assistant designed a minimal fork that would expose the existing internal APIs. The approach was surgical: make the internal split externally visible without changing how the internals worked.

The Phase 2 Design Document

Before touching any code, the assistant wrote a comprehensive design document (cuzk-phase2-design.md, 791 lines). This document laid out the per-partition pipeline strategy, which is the core architectural insight of Phase 2. The key idea is simple but powerful: instead of synthesizing all partitions at once (which requires holding all intermediate state simultaneously, consuming ~136 GiB), the pipeline processes partitions sequentially, reducing intermediate state to approximately 13.6 GiB — a 10x reduction.

The design document covered several critical areas:

Memory Budget Analysis: The assistant calculated the exact memory consumption of each stage in the pipeline. The SRS (Structured Reference String) consumes approximately 1.5 GiB per partition. The synthesized circuit assignments consume approximately 12 GiB per partition. By processing partitions one at a time, the peak memory drops from 136 GiB (all partitions simultaneously) to 13.6 GiB (one partition at a time plus SRS overhead).

SRS Manager Design: The SRS manager is responsible for loading and caching the Structured Reference String, which is the largest single data structure in the proving pipeline. The design document specified how the SRS would be loaded once and shared across partitions, how it would be pinned in GPU memory, and how it would be evicted when no longer needed.

Per-Partition Pipeline Strategy: Each partition goes through four stages: (1) CPU synthesis via synthesize_circuits_batch(), (2) assignment serialization and transfer to GPU, (3) GPU proving via MSM and NTT operations, and (4) proof aggregation. By streaming partitions through these stages sequentially, the pipeline minimizes memory pressure while maximizing GPU utilization.

7-Step Implementation Plan: The design document broke Phase 2 into seven concrete implementation steps, each with clear deliverables and success criteria. This plan served as the roadmap for the entire phase, ensuring that the team always knew what to build next and how to verify it.

The Minimal Fork Philosophy

The bellperson fork was created with approximately 130 lines of changes. The modifications were surgical and precise:

Making ProvingAssignment and its fields public: The ProvingAssignment struct is the data structure that carries the synthesized circuit from the CPU phase to the GPU phase. By making it public, the assistant enabled external code to inspect and serialize assignments without modifying bellperson's internals.
Exposing synthesize_circuits_batch() as a public API: This function performs the CPU-bound circuit synthesis. By exposing it, the assistant enabled the Phase 2 pipeline to run synthesis as a separate step, capturing the assignments for later GPU processing.
Adding prove_from_assignments() for the GPU phase: This function takes pre-synthesized assignments and performs the GPU-accelerated proving phase. By separating this from synthesis, the pipeline can run synthesis on one machine (or at one time) and proving on another (or at another time). This is the essence of the minimal fork philosophy: change only what is necessary to expose the existing internal architecture, rather than rewriting or restructuring the upstream code. The assistant read the bellperson source files carefully before making any changes, understanding the exact structure of ProvingAssignment, the flow of data through the proving pipeline, and the existing feature flags that controlled CUDA vs. CPU code paths. Every change was informed by deep understanding, not guesswork.

The Semver Trap and Workspace Patching

One of the most instructive moments in the fork creation was the semver trap. The assistant initially specified the fork's version as 0.22.1 (a patch bump from the upstream 0.22.0), but Cargo's dependency resolution rules treat version changes differently for workspace-patched crates. When a crate is patched via [patch.crates-io], Cargo uses the patched version regardless of the version number — but if the version number differs from what other crates expect, dependency resolution can fail in subtle ways.

The fix was to keep the version at 0.22.0 and rely on the [patch] section to override. This is a subtle but critical detail of Cargo's dependency resolution that could have derailed the entire fork. The assistant's ability to recognize and correct this issue demonstrates deep familiarity with Rust's build system — knowledge that comes from experience, not from documentation.

The workspace was patched via [patch.crates-io] in the root Cargo.toml, directing all crates in the workspace to use the fork at extern/bellperson/ instead of the upstream crate. This approach has several advantages: it requires no changes to any crate's Cargo.toml, it works transparently across the entire workspace, and it can be easily reverted when the upstream crate incorporates the necessary changes.

Compilation and Validation

The fork compiled cleanly with all eight existing tests passing. The assistant ran cargo check across the entire workspace, confirming that no crate was broken by the fork. The tests were run to ensure that the fork didn't introduce any regressions — a critical validation step that separates a working fork from a broken one.

The fork was committed as f258e8c7, and the project was now positioned to begin implementing the pipelined prover in cuzk-core. The bellperson fork provided the necessary API surface for the synthesis/GPU split, and the design document provided the roadmap for implementation. Phase 2 was ready to begin.

The Connecting Thread: Engineering Discipline

What unifies these two halves — the Phase 1 completion and the Phase 2 initiation — is a consistent engineering methodology that manifests in several patterns:

Research Before Implementation

In both cases, the assistant gathered precise information before writing code. For the gen-vanilla command, this meant reading four critical files and dispatching two parallel subagent tasks for API discovery and test data examination. For the bellperson fork, this meant reading the bellperson source files to understand the internal architecture before making any changes. In neither case did the assistant start coding with incomplete information.

This pattern is deceptively simple but remarkably rare in practice. The temptation to start coding immediately — to feel productive by producing output — is strong. The assistant resisted this temptation, investing time in understanding before acting. The result was fewer false starts, fewer bugs, and a cleaner final implementation.

Minimal Changes

Both deliverables were achieved with remarkably few lines of changed code. The gen-vanilla implementation used feature-gated optional dependencies rather than restructuring the bench tool's architecture. The bellperson fork changed approximately 130 lines rather than rewriting the proving pipeline. Both decisions reflect a philosophy of surgical precision over wholesale restructuring.

This philosophy has several benefits. Minimal changes are easier to review, easier to revert, and easier to merge with upstream changes. They reduce the risk of introducing bugs in unrelated code paths. And they make the intent of each change clear — a 130-line diff tells a story that a 1,300-line diff cannot.

Verification at Every Step

The assistant ran cargo check after every edit, fixed warnings immediately, and validated against golden test data. This discipline — the refusal to let even a single warning persist — is what separates robust engineering from hacking. Each verification step was a checkpoint that confirmed the system was still in a known good state.

The tight feedback loop between compilation and correction is particularly noteworthy. When a warning appeared, it was fixed within seconds. When a build error occurred, it was diagnosed and resolved in the same round. This speed comes from experience — knowing exactly where to look for the cause of a build failure — but the discipline to verify at every step is a choice that any engineer can make.

Documentation of Decisions

The todo lists, the design document, the commit messages — each artifact captures a decision point for future reference. The Phase 2 design document at 791 lines is a particularly thorough example, but even the brief todo updates serve as lightweight project management artifacts. When the assistant wrote "Phase 1 complete. Moving to Phase 2," it was not just a status update — it was a commitment to a new direction, recorded for anyone who reads the conversation history.

This documentation discipline is especially valuable in a project like cuzk, where the engineering team may change over time. A new engineer joining the project can read the design document and understand not just what was built, but why it was built that way. The commit messages tell the story of the project's evolution. The todo lists show the priorities at each stage.

Conclusion

This segment of the cuzk proving engine project captures a complete arc: from the final deliverable of one phase to the architectural foundation of the next. The gen-vanilla command, while seemingly a simple test utility, required navigating Filecoin's CID encoding, Rust's dependency management, feature-gating architecture, and careful validation against real 32 GiB sector data. The bellperson fork, while minimal in scope, required deep understanding of Groth16 proving internals, Cargo's patch mechanics, and the memory characteristics of the synthesis phase.

Together, these two efforts demonstrate that building production infrastructure for Filecoin proof generation is not about grand architectural gestures but about the cumulative effect of many small, correct decisions — each one researched, implemented, verified, and committed. The project now stands at the threshold of Phase 2 proper, with the bellperson fork providing the API surface needed to implement the pipelined prover that will reduce peak memory from 136 GiB to 13.6 GiB. The foundation has been laid; the pipeline awaits construction.

The engineering methodology that guided this segment — research before implementation, minimal changes, relentless verification, and thorough documentation — is not specific to SNARK proving or Filecoin. It is a universal approach to building reliable software systems. Whether the domain is cryptography, distributed systems, or web applications, the same principles apply. The cuzk project is a case study in how these principles play out in practice, and this segment is a particularly clear example of their power.

References

[1] "The Two-Phase Pivot: Completing Phase 1 and Architecting Phase 2 of the cuzk Proving Engine" — The chunk article that this synthesis builds upon, providing detailed analysis of the gen-vanilla implementation and bellperson fork creation.