The Split is Already Internal: A Pivotal Discovery in the cuzk Proving Engine

Introduction

In the development of complex software systems, there are moments when a deep investigation into existing code reveals that the architecture you were planning to build already exists—hidden behind visibility modifiers, waiting to be exposed. Message 398 of this opencode session captures exactly such a moment. The assistant, having just completed Phase 1 of the cuzk pipelined SNARK proving engine, turns its attention to Phase 2: splitting the monolithic proof generation pipeline into separate synthesis (CPU) and GPU computation phases. What follows is a masterclass in rigorous upstream code analysis, minimal-fork philosophy, and architectural decision-making.

The Message in Context

To understand the significance of message 398, we must first appreciate what came before. The cuzk project is an ambitious endeavor: building a persistent GPU-resident SNARK proving engine for Filecoin's proof-of-replication (PoRep) system—analogous to how vLLM or TensorRT serve inference models. Phase 0 established the gRPC daemon with basic PoRep C2 proving. Phase 1 added all four Filecoin proof types (WinningPoSt, WindowPoSt, SnapDeals, and the original PoRep C2) along with multi-GPU worker pools, priority scheduling, observability (tracing, Prometheus metrics), and a gen-vanilla command for generating test data. Phase 1 was just committed as 9d8453c3.

Now, with Phase 1 complete, the assistant turns to Phase 2. The project plan (cuzk-project.md) calls for splitting the monolithic create_proof_batch_priority function into two phases: CPU-side circuit synthesis and GPU-side computation (NTT, MSM, proof assembly). This split would enable the pipelined architecture where synthesis for one proof can run while GPU computation for another is in flight, dramatically improving throughput and reducing peak memory.

The assistant begins by reading the project plan ([msg 393]), then launches two parallel investigation tasks. The first task ([msg 395]) analyzes bellperson's prover internals at a high level. The second ([msg 396]) reads the complete source code of the supraseal prover and related files. By message 397, the assistant has "a complete picture" and is ready to synthesize findings.

Message 398: The Analysis and the Plan

Here is the message in full:

[assistant] The analysis is clear. Here's what I found and the plan:

## Phase 2 Analysis: The Split is Already Internal

The key insight is that bellperson's supraseal prover already has a clean two-phase architecture internally:

1. synthesize_circuits_batch() (private fn, ~60 lines) — CPU-only. Runs circuit.synthesize() for each circuit in parallel via rayon. Returns: - Vec<ProvingAssignment<Scalar>> — containing a, b, c evaluation vectors + density trackers - Vec<Arc<Vec<Scalar>>> — input assignments (moved out of ProvingAssignment) - Vec<Arc<Vec<Scalar>>> — aux assignments (moved out of ProvingAssignment)

2. GPU phase (~70 lines after synthesis) — Packs raw pointers, calls supraseal_c2::generate_groth16_proof() which does NTT + MSM + proof assembly on CUDA.

The boundary is clean. The required fork changes are minimal: - Make ProvingAssignment<Scalar> public (currently crate-private struct) - Make synthesize_circuits_batch() public - Add a prove_from_assignments() public function that takes the synthesis output and runs only the GPU phase - Expose DensityTracker (from ec-gpu-gen dependency, already pub there)

The changes are ~30 lines of code in bellperson. No logic changes, just visibility.

Before I start making the fork, let me check one more thing — how does filecoin-proofs call into bellperson? The split API needs to work at the filecoin-proofs-api level too, or we bypass that layer entirely. [task] ...

The message then launches a third investigation task to trace the call chain from filecoin-proofs-api down to bellperson, to understand whether the split should be introduced at the bellperson level or at a higher layer.

The Reasoning and Motivation

Why was this message written? The assistant had just completed two intensive investigation tasks ([msg 395] and [msg 396]) that together consumed thousands of lines of analysis. Message 398 serves as the synthesis point—the moment where raw data is transformed into actionable knowledge.

The motivation is deeply practical. The cuzk project plan calls for a pipelined proving engine, but the original assumption was that this would require significant restructuring of bellperson's internals. The assistant's investigation reveals that the restructuring is unnecessary: the pipeline boundary already exists as a clean function boundary within the existing code. This is a classic "discovery" moment in software engineering—the realization that the system you need to build is already latent in the system you have.

The assistant's reasoning proceeds through several layers:

Confirmation of the two-phase architecture: The investigation confirms that synthesize_circuits_batch() is a standalone function that performs only CPU work (circuit synthesis via rayon-parallelized circuit.synthesize() calls), and the GPU phase is a separate block of code that takes the synthesis output and performs NTT, MSM, and proof assembly on CUDA.
Assessment of the boundary cleanliness: The assistant evaluates whether the data structures flowing across the boundary are well-defined. The synthesis function returns ProvingAssignment objects containing a, b, c evaluation vectors and density trackers, plus input and auxiliary assignments. These are precisely the data needed by the GPU phase.
Estimation of fork scope: The assistant estimates the required changes at ~30 lines of code, all visibility changes. This is a critical engineering judgment—it means the fork is low-risk, low-maintenance, and unlikely to diverge from upstream.
Identification of the remaining unknown: Before committing to the fork approach, the assistant identifies a key question: does the split need to happen at the bellperson level, or should it be at the filecoin-proofs-api level? This drives the third investigation task.

Decisions Made

Message 398 contains several implicit and explicit decisions:

Decision 1: Fork bellperson rather than rewrite it. The assistant could have chosen to implement the synthesis/GPU split by rewriting the prover from scratch, or by adding a wrapper layer outside bellperson. Instead, the decision is to create a minimal fork that only changes visibility modifiers and adds a single new public function (prove_from_assignments). This is the "minimal-fork philosophy" in action: make the smallest possible change to achieve the goal, minimizing maintenance burden and divergence risk.

Decision 2: Expose existing internals rather than refactor them. The synthesize_circuits_batch() function already works correctly. It has been tested through the existing create_proof_batch_priority path. Rather than rewriting it or creating a new synthesis pathway, the assistant decides to simply make it public. This leverages years of existing testing and production use.

Decision 3: Investigate the full call chain before implementing. The assistant explicitly pauses before making the fork to trace how filecoin-proofs-api calls into bellperson. This is a wise precaution: if the split needs to happen at a higher level (e.g., within filecoin-proofs-api itself), then modifying bellperson might be unnecessary or insufficient. The decision to investigate first rather than assume demonstrates intellectual rigor.

Decision 4: The split boundary is at ProvingAssignment. By choosing to expose ProvingAssignment and synthesize_circuits_batch(), the assistant implicitly defines the pipeline boundary as the point where synthesis output is ready for GPU processing. This is a natural boundary: synthesis is CPU-bound and memory-intensive (generating the constraint system evaluation), while the GPU phase is compute-bound and GPU-memory-intensive (NTT and MSM). Splitting at this point allows the CPU to work on the next proof's synthesis while the GPU processes the current proof's assignments.

Assumptions Made

The assistant makes several assumptions, most of which are well-justified:

Assumption 1: The synthesis function is truly CPU-only. The assistant assumes that synthesize_circuits_batch() contains no GPU calls. This is supported by reading the source code, which shows only rayon-parallelized circuit.synthesize() calls and data structure manipulation. However, the assumption should be validated by tracing any indirect dependencies.

Assumption 2: The GPU phase can be cleanly separated. The assistant assumes that the ~70 lines of code after the synthesis call constitute a self-contained GPU phase that can be extracted into prove_from_assignments(). This is supported by the code structure, but the assistant hasn't yet verified that there are no hidden dependencies on synthesis-time state.

Assumption 3: Visibility changes are sufficient. The assistant assumes that making ProvingAssignment public and synthesize_circuits_batch() public will not break any invariants. ProvingAssignment is a data container—making its fields public should be safe as long as no internal invariants depend on encapsulation.

Assumption 4: The fork will compile without issues. The assistant estimates ~30 lines of change but hasn't yet verified compilation. This is a reasonable estimate based on the scope of changes, but the actual effort may vary depending on transitive visibility constraints and Rust's module system.

Assumption 5: The filecoin-proofs-api layer can be bypassed. The assistant's question about whether the split API needs to work at the filecoin-proofs-api level reveals an assumption that cuzk might call bellperson directly rather than through the higher-level API. This is likely correct—cuzk's architecture already bypasses filecoin-proofs-api for the GPU proving path (it calls supraseal_c2 directly via FFI), so bypassing it for synthesis too is consistent.

Input Knowledge Required

To fully understand message 398, one needs knowledge of:

Groth16 proof generation: Understanding that Groth16 proving consists of circuit synthesis (translating the computation into a constraint system and evaluating it) followed by cryptographic computations (NTT for polynomial operations, MSM for multi-scalar multiplication). The split between these phases is what the message is about.
Bellperson's architecture: Bellperson is a Rust library for Groth16 proving, used in Filecoin's proof system. It has a supraseal prover backend that uses C++/CUDA code for GPU acceleration. The message references specific internal types (ProvingAssignment, DensityTracker) and functions (synthesize_circuits_batch, create_proof_batch_priority).
Filecoin proof types: The message references PoRep (proof-of-replication), WinningPoSt, WindowPoSt, and SnapDeals—the four proof types that the cuzk engine supports. Understanding the distinction between "vanilla" proofs (Merkle inclusion proofs, CPU-only) and "C2" proofs (Groth16 SNARK proofs, GPU-accelerated) is helpful context.
The cuzk project architecture: The message builds on the cuzk project plan, which envisions a pipelined proving engine with SRS (Structured Reference String) residency, priority scheduling, and multi-GPU support. Phase 2 specifically aims to split synthesis from GPU computation to enable pipelining.
Rust visibility and module system: The message discusses making crate-private types and functions public, which requires understanding Rust's pub visibility modifiers and how they interact with module boundaries.
Rayon parallelism: The synthesis function uses rayon for parallel circuit synthesis, which is relevant to understanding the CPU-side performance characteristics.

Output Knowledge Created

Message 398 creates several forms of knowledge:

A confirmed architectural finding: The key insight—that bellperson already has a clean internal split between synthesis and GPU computation—is now documented. This finding will drive the entire Phase 2 implementation.
A concrete fork specification: The message specifies exactly what changes are needed: make ProvingAssignment public, make synthesize_circuits_batch() public, add prove_from_assignments(), and expose DensityTracker. This is essentially a design document for the fork.
A scope estimate: The ~30 lines estimate provides a concrete sense of the fork's size and risk. This is valuable for project planning and stakeholder communication.
An identified open question: The message explicitly identifies the need to trace the filecoin-proofs-api to bellperson call chain before proceeding. This prevents premature implementation and ensures the fork is designed correctly.
A methodological precedent: The message establishes a pattern for how the cuzk project approaches upstream modifications: investigate thoroughly, find the minimal change, validate assumptions, and only then implement. This pattern will likely be repeated for future phases.

The Thinking Process Visible in Reasoning

The message reveals a structured thinking process that moves through several stages:

Stage 1: Synthesis of findings. The assistant begins with "The analysis is clear," signaling that it has integrated the results of two parallel investigation tasks. This is not raw data regurgitation—it's synthesized knowledge.

Stage 2: Statement of the key insight. "The Split is Already Internal" is the headline finding. The assistant emphasizes this with bold formatting and the word "already," which carries the connotation of pleasant surprise—the architecture they were planning to build already exists.

Stage 3: Evidence presentation. The assistant lists the two phases with their inputs/outputs, providing concrete evidence for the claim. The level of detail (function line counts, return types, data structures) demonstrates thorough investigation.

Stage 4: Assessment of boundary cleanliness. The assistant explicitly evaluates the boundary as "clean," which is a qualitative judgment based on code reading. This judgment is critical because a messy boundary would require refactoring rather than simple visibility changes.

Stage 5: Fork scope estimation. The assistant enumerates the four specific changes needed and estimates ~30 lines. The emphasis on "No logic changes, just visibility" is reassuring—it means the fork is low-risk.

Stage 6: Identification of remaining unknowns. Rather than diving into implementation, the assistant pauses to investigate the filecoin-proofs-api call chain. This demonstrates intellectual humility and thoroughness—the willingness to question one's own assumptions before acting.

The thinking process is notable for its discipline. The assistant could have started implementing the fork immediately after the investigation tasks completed. Instead, it takes the time to synthesize findings, document the plan, and identify open questions. This discipline is characteristic of experienced engineers who have learned that the cost of asking one more question before implementation is far lower than the cost of implementing the wrong thing.

Mistakes and Incorrect Assumptions

While the message is generally well-reasoned, there are potential issues worth examining:

Potential issue 1: The "clean boundary" assessment may be overly optimistic. The assistant describes the boundary as clean based on reading the source code, but there may be implicit coupling that isn't visible at the function level. For example, the GPU phase might depend on global state initialized during synthesis, or the ProvingAssignment struct might contain fields that are only valid during synthesis. The ~30 lines estimate assumes no such hidden coupling exists.

Potential issue 2: The assumption that visibility changes are sufficient may underestimate Rust's module system constraints. In Rust, making a struct public requires all its field types to also be public (or at least reachable). If ProvingAssignment contains fields of types that are themselves crate-private, the visibility changes could cascade. The assistant's estimate of ~30 lines may need revision if such cascading visibility issues arise.

Potential issue 3: The fork may need to track upstream changes. Even a minimal fork creates a maintenance burden. If bellperson's internal architecture changes in a future release (e.g., if synthesize_circuits_batch() is refactored or renamed), the fork will need corresponding updates. The assistant doesn't discuss this maintenance cost.

Potential issue 4: The split may not be sufficient for pipelining. The assistant assumes that exposing the synthesis/GPU boundary is sufficient for the pipelined architecture envisioned in the project plan. However, true pipelining may require additional infrastructure (e.g., async task queues, memory management for intermediate state, synchronization between CPU and GPU work). The fork is necessary but may not be sufficient.

These potential issues are not so much mistakes as they are risks that the assistant has implicitly accepted. The decision to investigate the filecoin-proofs-api call chain before implementing shows awareness that there may be additional complexities.

Conclusion

Message 398 represents a pivotal moment in the cuzk project's Phase 2 development. It is the moment when investigation transforms into design, when raw data becomes actionable knowledge, and when the assistant commits to a specific technical approach. The message is notable for its intellectual rigor: the assistant synthesizes findings from multiple investigation tasks, evaluates the boundary cleanliness, estimates fork scope, and identifies remaining unknowns before proceeding.

The key insight—that bellperson's supraseal prover already has a clean internal split between synthesis and GPU computation—is a classic example of the value of deep code reading. Rather than building the pipeline boundary from scratch, the assistant discovers that it already exists, hidden behind Rust's visibility modifiers. The minimal-fork philosophy (change only visibility, add one function, ~30 lines total) reflects engineering wisdom: the best modification is often the smallest one that achieves the goal.

This message also demonstrates a pattern that will likely characterize the rest of the cuzk project: thorough investigation, minimal intervention, and disciplined decision-making. It is a small but significant step toward the larger goal of building a persistent, memory-efficient, pipelined SNARK proving engine for Filecoin's proof-of-replication system.