From Globs to GPU Kernels: A Methodical Deep-Dive into Bellperson's Groth16 Prover

Introduction

In the world of zero-knowledge proof systems, few challenges are as daunting as understanding a production-grade Groth16 prover with a ~200 GiB peak memory footprint. This article chronicles a remarkable opencode coding session in which an AI assistant systematically dissected the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep) protocol. Over the course of dozens of messages, the assistant traced the call chain from Curio's Go task layer, through Rust FFI in the bellperson crate, into the C++/CUDA kernels of the supraseal-c2 library, producing four comprehensive documents and three composable optimization proposals. The session is a masterclass in methodical codebase archaeology, revealing how an AI agent navigates an unfamiliar environment, recovers from tool failures, and synthesizes dispersed information into actionable architectural knowledge.

The Opening Gambit: When Globs Fail

The investigation began with what seemed like a straightforward task. The user's request ([msg 0]) was precise: read five specific files from the bellperson crate's Groth16 prover and map the exact boundary between CPU synthesis and GPU computation. The goal was to split the monolithic create_proof_batch_priority function into two independently callable phases — CPU-bound witness synthesis and GPU-accelerated NTT/MSM computation.

The assistant's first move was logical: use glob patterns to locate the files. As [12] describes in "The Opening Gambit That Failed," the assistant dispatched four glob commands targeting paths like /home/theuser/.cargo/registry/src/index.crates.io-*/bellperson-0.26.0/src/groth16/prover/mod.rs. Every single one returned "No files found."

This failure was not a dead end — it was a data point. The glob tool in this environment apparently did not expand wildcards in directory path components the way a standard Unix shell would. The assistant could have spun its wheels debugging the tool, but instead it pivoted to a more fundamental exploration technique: a raw ls of the registry parent directory. This revealed that the system had two Cargo registry indices — index.crates.io-1949cf8c6b5b557f and index.crates.io-6f17d22bba15001f — each containing a copy of bellperson-0.26.0.

As [4] ("When Globs Fail") and [11] ("The Discovery Step") document, this pivot from glob to ls exemplifies a critical debugging pattern: when a high-level search tool fails, fall back to lower-level directory listing. The assistant demonstrated resilience, systematically narrowing the search space until the files were located.

The Diff That Unlocked a Pipeline

With two copies of the same crate discovered, the assistant faced an ambiguity: which copy was authoritative? Rather than arbitrarily picking one, it ran a diff command comparing the mod.rs files from both indices. As [5] ("The Diff That Unlocked a Pipeline") explains, this single diff operation was a methodological crossroads. The assistant chose verification over speed, establishing a foundation of trust for all subsequent analysis.

The diff confirmed the files were identical, and the assistant proceeded to read the five requested files. But it didn't stop there. In what [17] ("The Proactive Analyst") describes as a pattern of proactive code comprehension, the assistant recognized that the user's deeper question — how to design a split API — required understanding not just the prover internals but also the public API surface, the parameter types, and the proof structure. It proactively read four additional files: ext_supraseal.rs, supraseal_params.rs, params.rs, and proof.rs.

This decision reveals a sophisticated understanding of software architecture. The assistant was not mechanically fulfilling a checklist; it was building a mental model of the entire call chain from the outermost public API to the innermost GPU kernel invocation.

Crossing the FFI Boundary

The bellperson prover's supraseal variant delegates GPU computation to the supraseal-c2 crate — a thin Rust wrapper around Supranational's C++ CUDA library. Understanding this FFI boundary was essential for the split API design. The assistant's journey to read the supraseal-c2 source was itself an odyssey of tool adaptation.

As [7] ("The Glob That Failed") recounts, the assistant first tried to glob for the crate's source files using **/*.rs patterns. Both attempts returned "No files found." Rather than giving up, the assistant switched to the Unix find command in what [13] ("The Pivot Point") calls a moment of adaptation. The find command succeeded immediately, revealing the crate's minimal Rust surface area — just a single lib.rs, a build.rs, and a test file.

Reading the supraseal-c2 source in [16] ("Peering into the FFI Boundary") revealed the SRS type as an opaque pointer wrapper (#[repr(C)] pub struct SRS { ptr: *const core::ffi::c_void }), confirming that the ~200 GiB Structured Reference String lives entirely on the C++ side. The Default impl producing a null pointer hinted at an unloaded state, and the sppark::cuda_error!() macro invocation revealed CUDA error propagation. Most importantly, the crate was confirmed to be a minimal wrapper — the heavy logic lives in the compiled C++ library.

Tracing the Data Flow: DensityTrackers and Linear Combinations

With the structural map in place, the assistant turned to understanding the data flow that bridges synthesis and GPU computation. The DensityTracker type, imported in the prover module from ec_gpu_gen::multiexp_cpu, kept appearing in the code. As [14] ("The DensityTracker Signal") describes, the assistant executed a targeted grep for DensityTracker across the ec-gpu-gen crate, locating its definition at line 119 of multiexp_cpu.rs.

Reading the struct definition in [10] ("Tracing the DensityTracker") revealed a simple but powerful type:

#[derive(Clone, PartialEq, Eq, Debug, Default)]
pub struct DensityTracker {
    pub bv: BitVec,
    pub total_density: usize,
}

This bitvector tracks which variables in the constraint system are "dense" (used in many constraints) versus "sparse" (unused), enabling the MSM algorithm to skip zero-weight contributions. It is the precise data structure that carries sparsity information from the CPU synthesis phase to the GPU proving phase.

The assistant then traced the eval_with_trackers function in lc.rs ([3] "Peeling the Onion"), which evaluates linear combinations while populating density trackers. This function sits at the exact seam where synthesis produces the data that GPU computation consumes. The optional density trackers (Option<&mut DensityTracker>) suggested a possible optimization: if density tracking can be deferred or skipped during synthesis, the intermediate state could be reduced.

The Moment of Synthesis

The culmination of the exploration phase was message 16, which [6] ("The Moment of Synthesis") calls the architectural keystone of the entire investigation. In this message, the assistant synthesized information from approximately 1,500 lines of source code across eight files into a coherent map of the Groth16 proving pipeline.

The critical discovery was that the separation the user wanted to design already existed internally. The synthesize_circuits_batch() function is a standalone private function that returns all the intermediate state needed for the GPU phase. The boundary is exactly at the return of synthesize_circuits_batch(). The intermediate state consists of:

Vec<ProvingAssignment<Scalar>> containing a, b, c vectors and three density trackers
Vec<Arc<Vec<Scalar>>> — input assignments (public witness)
Vec<Arc<Vec<Scalar>>> — aux assignments (private witness) The assistant also noted a critical detail: after synthesis, the ProvingAssignment has had its input_assignment and aux_assignment moved out via std::mem::take, leaving only the constraint evaluations and density trackers. The intermediate state is already partially decomposed.

From Analysis to Architecture: The Phase 2 Design

With the bellperson internals fully mapped, the investigation transitioned from analysis to design. The assistant created a minimal bellperson fork at extern/bellperson/ with approximately 130 lines of changes: making ProvingAssignment and its fields public, exposing synthesize_circuits_batch(), and adding prove_from_assignments() for the GPU phase. The workspace was patched via [patch.crates-io], and the fork compiled cleanly with all eight existing tests passing.

A detailed Phase 2 design document (cuzk-phase2-design.md, 791 lines) was written, covering the per-partition pipeline strategy, memory budget analysis, SRS manager design, and a seven-step implementation plan. The per-partition pipeline strategy reduces intermediate state from ~136 GiB to ~13.6 GiB by streaming partitions sequentially through the GPU rather than holding all partitions in memory simultaneously.

The Three Optimization Proposals

The analysis produced three composable optimization proposals, each targeting a different bottleneck in the pipeline:

Proposal 1: Sequential Partition Synthesis. By breaking the all-10-partitions-in-parallel model, peak memory can be reduced from ~200 GiB to ~64-103 GiB. Partitions are streamed one-at-a-time through the GPU, dramatically reducing the memory pressure that makes the current pipeline expensive to run on cloud instances.

Proposal 2: Persistent Prover Daemon. The current pipeline loads the ~48 GiB SRS from disk for every proof, costing approximately 60 seconds per proof in I/O overhead. By keeping the proving process alive across proofs, this overhead is eliminated entirely. The daemon maintains the SRS in GPU memory and accepts synthesis artifacts for continuous proving.

Proposal 3: Cross-Sector Batching. With the memory freed by Sequential Partition Synthesis, multiple sectors' circuits can be batched into single GPU invocations. This improves GPU utilization and amortizes kernel launch overhead, projecting 2-3× throughput per GPU and 5-6× reduction in cost per proof when combined with the other proposals.

Micro-Optimization Analysis

A final round of micro-optimization analysis examined CPU synthesis hotpaths (the enforce() loop, SHA-256 bit manipulation, Fr field arithmetic), GPU NTT/MSM compute characteristics, H-to-D transfer patterns, and the feasibility of recomputing a/b/c vectors on-the-fly to avoid materialization entirely. This analysis revealed that the CPU synthesis phase is dominated by SHA-256 bit manipulation (from the Filecoin proof-of-replication circuit's Merkle proof verification) and Fr field arithmetic, while the GPU phase is memory-bandwidth-bound on NTT operations.

The Overarching Achievement

What makes this session remarkable is not just the depth of the analysis but the shift in perspective it represents. The investigation began with a narrow question — "how does the current prover work?" — and ended with a comprehensive architectural vision: a continuous, memory-efficient proving pipeline optimized for heterogeneous cloud rental markets where RAM cost dominates.

The assistant's methodical approach — recovering from tool failures, tracing dependencies across crate boundaries, verifying assumptions with diffs, and proactively expanding the scope of investigation — demonstrates a level of systematic reasoning that is essential for understanding complex cryptographic software. The session produced not just knowledge but actionable designs: a bellperson fork with exposed APIs, a detailed implementation plan, and three optimization proposals that together promise to transform Filecoin's proving infrastructure.

In the end, this session is a testament to the power of methodical codebase archaeology. The path from failed globs to GPU kernel analysis is not a straight line — it is a winding journey of adaptation, verification, and synthesis. But for those willing to follow the data, it leads to deep understanding and transformative architectural insight.