From 200 GiB to Streamlined: Deconstructing and Reimagining the SUPRASEAL_C2 Groth16 Proving Pipeline

Introduction

In the world of decentralized storage, proof generation is the invisible engine that powers trust. Filecoin's Proof-of-Replication (PoRep) requires storage miners to periodically generate Groth16 zero-knowledge proofs demonstrating that they are faithfully storing the data they claim to hold. These proofs are not cheap: a single SUPRASEAL_C2 proof generation consumes approximately 200 GiB of peak memory and pushes modern hardware—spanning Go, Rust, C++, and CUDA—to its absolute limits. This article synthesizes a deep-dive investigation into that pipeline, tracing the full call chain from Curio's Go task orchestration layer through Rust FFI into C++/CUDA kernels, accounting for every gigabyte of that 200 GiB footprint, identifying nine structural bottlenecks, and proposing three composable optimization strategies that collectively could reduce memory by 50–68% and improve throughput by 2–6×.

The work proceeded in two major phases. First, a comprehensive mapping of the pipeline architecture produced a detailed background reference document with file:line references and memory accounting showing exactly where each GiB goes: ten parallel partition circuits at ~16 GiB each, plus ~48 GiB of Structured Reference String (SRS) data in pinned GPU memory. Second, after a challenge to "think bigger," the analysis expanded to encompass the Curio orchestration model, circuit value distribution statistics (~99% boolean aux_assignment), and computational hotpath characterization at the instruction level. The result is a shift in perspective—from optimizing individual proof generation toward architecting a continuous, memory-efficient proving pipeline optimized for heterogeneous cloud rental markets where RAM cost dominates.

The Pipeline: A Multi-Layered Beast

The SUPRASEAL_C2 proving pipeline is a study in layered complexity. At the top, Curio—a Go-based Filecoin storage mining implementation—orchestrates proof generation tasks. When a PoRep proof is needed, Curio's task layer invokes a Rust FFI function that bridges into the bellperson library (a fork of the original bellman library for Groth16 proofs). Bellperson, in turn, calls into supraseal-c2, a C++ library that orchestrates GPU compute kernels via CUDA.

The proof generation process itself follows the Groth16 protocol, which requires constructing a rank-1 constraint system (R1CS) circuit representing the computation being proved, then performing a multi-step proving procedure involving number-theoretic transforms (NTTs) and multi-scalar multiplications (MSMs) on the GPU. The critical architectural decision that drives the 200 GiB memory footprint is that SUPRASEAL_C2 processes ten partitions in parallel. Each partition corresponds to a portion of the circuit, and processing them simultaneously requires holding all ten in memory at once. At ~16 GiB per partition, that accounts for ~160 GiB. Add ~48 GiB of SRS data loaded into pinned GPU memory, and the 200 GiB total is accounted for.

The investigation mapped this call chain in exhaustive detail, producing a background reference document that traces the path from curio/go/tasks/seal.go through extern/bellperson/src/groth16/prover/supraseal.rs into extern/supraseal-c2/src/ C++ files and finally into CUDA kernels. Each file:line reference was captured, and each memory allocation was attributed to its source. This document became the foundation for all subsequent optimization analysis.

Nine Structural Bottlenecks

The pipeline mapping revealed nine structural bottlenecks, each representing an opportunity for optimization:

  1. Ten parallel partition circuits — The dominant memory consumer at ~160 GiB. Processing partitions sequentially would dramatically reduce peak memory.
  2. SRS loading overhead — Loading ~48 GiB of SRS data from disk into pinned GPU memory takes ~60 seconds per proof and blocks the proving process.
  3. Partition circuit materialization — Each partition circuit is fully materialized in memory before proving begins, consuming ~16 GiB per partition.
  4. A/B/C vector materialization — The linear combinations A, B, and C are fully computed and stored before being consumed by the GPU.
  5. CPU-side synthesis hotpath — The enforce() closure pattern in bellperson's constraint system creates and discards thousands of temporary LinearCombination allocations.
  6. GPU NTT memory bandwidth — The NTT kernels are memory-bandwidth-bound, not compute-bound.
  7. GPU MSM characteristics — Multi-scalar multiplications are both memory and compute intensive, with limited parallelism.
  8. Host-to-device transfer overhead — Moving data from CPU to GPU memory consumes PCIe bandwidth and adds latency.
  9. Proof serialization and verification — The final proof must be serialized and verified, adding overhead at the end of the pipeline. Each bottleneck was documented with its memory contribution, its location in the codebase, and the conditions under which it manifests. This systematic catalog enabled the team to prioritize optimization efforts by impact.

The Subagent Investigation: Tracing LC Closure Patterns

While the pipeline mapping addressed the macro-level architecture, a parallel investigation zoomed in on a micro-level hotspot: the enforce closure pattern in bellperson's constraint system. Profiling had identified Boolean::lc() at 6.51% of runtime and UInt32::addmany at 6.82% as the hottest functions during circuit synthesis. A subagent was dispatched with a focused mission: categorize how every enforce closure across the gadget codebase uses (or ignores) the lc parameter—a LinearCombination::zero() passed into each closure that could potentially be recycled to eliminate allocations.

The subagent's investigation unfolded across seven articles, each capturing a critical step in the process:

Opening Moves: Systematic File Discovery ([7]) — The subagent began by issuing three parallel glob searches to locate all relevant gadget files across the bellperson and bellpepper-core crates. The searches revealed seven gadget files in bellperson and, notably, returned "No files found" for storage-proofs circuits, confirming the investigation could focus entirely on the bellperson ecosystem.

The Discovery of Bellpepper-Core ([2]) — A critical pivot occurred when the subagent searched for boolean.rs in bellperson's gadgets directory and found nothing. By reading the module index file gadgets.rs, it discovered that Boolean is not defined in bellperson at all—it is re-exported from bellpepper-core, a separate crate. This architectural insight was essential: the definition of Boolean::lc() lives in extern/bellpepper-core/src/gadgets/boolean.rs, not in bellperson itself.

Tracing the Re-Export ([6]) — Following the re-export trail, the subagent located boolean.rs in bellpepper-core and simultaneously mapped the entire bellpepper-core source tree, discovering lc.rs, constraint_system.rs, and num.rs—all essential for understanding the full picture of LinearCombination allocation.

The Parallel Read ([4]) — With all file paths identified, the subagent issued eight parallel read commands to load the source code of every key gadget file: boolean.rs, uint32.rs, multieq.rs, sha256.rs, multipack.rs, lookup.rs, blake2s.rs, and num.rs. This parallel strategy minimized round trips and gathered all the raw data needed for analysis in a single message.

Tracing the LC Closure Pattern ([1]) — A pivotal read of constraint_system.rs revealed the enforce signature, confirming that each closure receives a LinearCombination::zero(). This foundational knowledge enabled the subagent to interpret the closure patterns correctly.

Anatomy of a Performance Investigation ([5]) — The subagent produced a comprehensive report answering all five of the user's questions. The critical finding: every branch of Boolean::lc() starts with LinearCombination::<Scalar>::zero(), meaning every call allocates a fresh Vec on the heap. The report also delivered a systematic census of every cs.enforce() call across the gadget codebase, revealing that ~84% of closures USE the lc parameter (building on it with |lc| lc + variable) while ~16% IGNORE it (constructing fresh LCs from scratch).

Deconstructing the enforce Closure ([3]) — The investigation concluded with a detailed analysis connecting the closure patterns back to the optimization strategy. The 84% majority pattern meant that a recycling pool or in-place mutation approach (like add_to_lc/sub_from_lc methods) could eliminate millions of temporary allocations. The 16% minority pattern required a different approach—perhaps avoiding the allocation of the passed-in lc entirely when it won't be used.

This subagent investigation exemplifies the power of systematic, hypothesis-driven code analysis. It transformed vague profiling data ("these functions are hot") into precise, actionable understanding ("these functions are hot because they allocate fresh LCs on every call, and here are all the places where that happens").

Thinking Bigger: Three Optimization Proposals

With the pipeline mapped and the bottlenecks identified, the investigation shifted to designing solutions. The user challenged the team to "think bigger," and the response was three composable optimization proposals that together could transform the proving pipeline:

Proposal 1: Sequential Partition Synthesis

The core insight: the ten parallel partitions are the dominant memory consumer at ~160 GiB. By streaming partitions sequentially through the GPU—processing one partition at a time, reusing the same GPU memory for each—peak memory drops from ~200 GiB to approximately 64–103 GiB. This is a 50–68% reduction. The trade-off is increased latency: sequential processing takes longer per proof. But in a continuous proving pipeline where proofs are generated back-to-back, the throughput impact can be mitigated by overlapping partition processing with SRS loading for the next partition.

Proposal 2: Persistent Prover Daemon

The SRS loading overhead of ~60 seconds per proof is pure waste when proofs are generated in sequence. By keeping the proving process alive across multiple proofs—a persistent daemon that maintains the GPU state, SRS data in pinned memory, and the CUDA context—the SRS loading cost is paid once and amortized across hundreds or thousands of proofs. This eliminates the ~60s per-proof overhead entirely. Combined with Sequential Partition Synthesis, the daemon can also pre-load the next partition's data while the current partition is being processed, hiding latency.

Proposal 3: Cross-Sector Batching

With the memory freed by Sequential Partition Synthesis (from ~200 GiB down to ~64–103 GiB), there is headroom to process multiple sectors' circuits in a single GPU invocation. Instead of proving one sector at a time, the system can batch 2–3 sectors' circuits into the same NTT and MSM operations, amortizing GPU kernel launch overhead and improving utilization. Projections suggest 2–3× throughput per GPU and, when combined with the other proposals, 5–6× reduction in cost per proof.

These proposals are composable: Sequential Partition Synthesis reduces memory, the Persistent Prover Daemon eliminates SRS overhead, and Cross-Sector Batching exploits the freed memory to improve throughput. Together, they represent a fundamental re-architecting of the proving pipeline—from a batch-oriented, memory-hungry process to a continuous, memory-efficient stream.

Micro-Optimization Analysis

The final phase of the investigation examined the computational hotpaths at the instruction level, identifying opportunities for micro-optimization across CPU and GPU:

CPU Synthesis Hotpaths: The enforce() loop in bellperson's constraint system was analyzed in detail. The SHA-256 bit manipulation operations (rotation, XOR, AND) in the circuit synthesis were found to be dominated by LinearCombination allocation overhead. The add_to_lc/sub_from_lc optimization directly addresses this by allowing in-place mutation of existing LCs rather than creating new ones. Fr field arithmetic (the prime field used in BLS12-381) was also examined, revealing that addition operations are cheap but multiplication (which requires reduction) is significantly more expensive.

GPU NTT/MSM Characteristics: The NTT kernels were confirmed to be memory-bandwidth-bound—they spend more time waiting for data from GPU memory than computing. This means optimization efforts should focus on data layout and access patterns rather than arithmetic intensity. The MSM operations are both memory and compute intensive, with the bottleneck shifting depending on the number of points being multiplied.

Host-to-Device Transfer Patterns: The investigation analyzed how data flows from CPU to GPU memory. The partition circuits, once synthesized on the CPU, are transferred to the GPU via PCIe. At ~16 GiB per partition, this transfer takes significant time and consumes bandwidth that could be used for other operations. The Sequential Partition Synthesis proposal mitigates this by reducing the total data that needs to be in GPU memory at any given time.

Recomputing A/B/C Vectors On-the-Fly: A provocative optimization was evaluated: instead of materializing the A, B, and C linear combinations as full vectors in memory, could they be recomputed on-the-fly during the GPU proving phase? The analysis concluded that this is feasible for some circuit structures but not all. For circuits where the constraints follow regular patterns (like SHA-256), the A/B/C vectors can be regenerated from the circuit description with minimal overhead. For irregular circuits, the cost of recomputation may exceed the memory savings. This remains an open research direction.

From Optimization to Architecture

The overarching achievement of this investigation is a shift in perspective. The initial framing was about optimizing individual proof generation—reducing memory, improving throughput, cutting costs. But the deeper insight is that the proving pipeline should be re-architected as a continuous, memory-efficient stream rather than a batch-oriented, memory-hungry process.

The three proposals—Sequential Partition Synthesis, Persistent Prover Daemon, Cross-Sector Batching—are not independent optimizations. They are a coherent architectural vision: a proving pipeline that:

Conclusion

The SUPRASEAL_C2 Groth16 proof generation pipeline is a marvel of cross-language engineering, spanning Go, Rust, C++, and CUDA to produce zero-knowledge proofs for Filecoin's decentralized storage network. But its ~200 GiB peak memory footprint is a critical liability, limiting deployment options and driving up costs. This investigation mapped the pipeline in exhaustive detail, accounting for every gigabyte, identifying nine structural bottlenecks, and proposing three composable optimization strategies that together could reduce memory by 50–68% and improve throughput by 2–6×.

The work also demonstrated the value of systematic, hypothesis-driven code analysis at multiple scales—from the macro-level pipeline mapping to the micro-level investigation of LinearCombination allocation patterns in enforce closures. The subagent's census of closure patterns, revealing that ~84% of closures use the lc parameter, provided the data needed to design targeted optimizations like add_to_lc/sub_from_lc methods.

Ultimately, the investigation's greatest contribution is a new architectural vision: a continuous, memory-efficient proving pipeline optimized for the realities of cloud computing, where RAM is expensive and utilization is paramount. This vision, captured in three composable proposals, charts a path from 200 GiB to streamlined—and from optimizing individual proofs to architecting a sustainable proving infrastructure for Filecoin's future.