Taming the 200 GiB Beast: A Deep-Dive into Filecoin's SUPRASEAL_C2 Groth16 Proof Pipeline

Introduction

In the world of zero-knowledge proofs, the difference between a system that works and one that is economically viable often comes down to memory. When Filecoin's storage providers generate Groth16 proofs for Proof-of-Replication (PoRep), they contend with a peak memory footprint of approximately 200 GiB—a staggering figure that constrains hardware choices, increases cloud rental costs, and limits decentralization. The SUPRASEAL_C2 pipeline, which orchestrates this proof generation from Curio's Go task layer through Rust FFI into C++/CUDA kernels, has been the subject of intense optimization efforts. But understanding the problem requires more than surface-level profiling—it demands a deep, systematic mapping of every layer in the call chain, every data structure that consumes memory, and every computational hotpath that determines throughput.

This article synthesizes a comprehensive investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep [1]. The work proceeded in two major phases: first, a comprehensive mapping of the entire call chain with detailed memory accounting showing exactly where each GiB goes; second, after the user challenged the investigation to "think bigger," an expansion that included the Curio orchestration model, circuit value distribution statistics, and computational hotpath characterization at the instruction level. The result is a set of four documents: a background reference capturing the full pipeline architecture with nine identified structural bottlenecks, plus three composable optimization proposals that collectively promise to dramatically reduce memory footprint and improve throughput.

The Architecture: From Curio to CUDA

The SUPRASEAL_C2 pipeline is a marvel of layered engineering. At the top sits Curio, Filecoin's Go-based storage mining orchestrator. Curio manages the lifecycle of storage proofs—selecting sectors, managing challenges, and dispatching proving work. When a PoRep proof needs to be generated, Curio calls into a Rust FFI layer that bridges the Go runtime with the proving library.

The Rust layer is built on bellperson, a fork of the Bellman zk-SNARKs library adapted for Filecoin's needs. Bellperson provides the Groth16 prover infrastructure: the ProvingAssignment that constructs the A, B, and C linear combination evaluations, the DensityTracker that records which auxiliary variables appear in each constraint, and the multi-exponentiation (MSM) engine that performs the heavy elliptic curve arithmetic. Below bellperson sits bellpepper-core, which provides the generic ConstraintSystem trait and basic gadgets like AllocatedNum, AllocatedBit, and the critical to_bits_le function that decomposes field elements into boolean bits.

The actual heavy lifting happens in supraseal-c2, a C++/CUDA library that implements GPU-accelerated Groth16 proving. This is where the NTT (Number Theoretic Transform) and MSM operations are executed on NVIDIA GPUs, processing the large R1CS matrices that define the PoRep circuit. The CUDA kernels are designed to handle the full circuit for a single partition, with the prover synthesizing all 10 partitions in parallel to maximize GPU utilization—but at a tremendous memory cost.

The Memory Problem: Where 200 GiB Goes

The investigation produced a detailed memory accounting that explains the ~200 GiB peak footprint. The root cause is the parallel synthesis of all 10 partitions simultaneously. Each partition's circuit contains approximately 107 million constraints, requiring:

The Nine Bottlenecks

The background reference document identified nine structural bottlenecks in the pipeline, each representing an opportunity for optimization:

  1. Parallel partition synthesis memory: The simultaneous synthesis of all 10 partitions is the dominant memory cost. Each partition's a/b/c vectors consume ~12.8 GiB, and holding all 10 in memory simultaneously is the primary driver of the 200 GiB peak.
  2. SRS loading overhead: The ~48 GiB SRS must be loaded from disk and pinned in GPU memory before any proving can begin. This loading takes approximately 60 seconds per proof, during which the GPU is idle.
  3. CPU-GPU transfer bandwidth: The synthesized circuit data (a/b/c vectors, aux_assignment) must be transferred from CPU memory to GPU memory over PCIe. With ~200 GiB of data per proof, this transfer is a significant fraction of total proving time.
  4. SHA-256 labeling constraint density: The labeling step, which uses SHA-256 to derive node labels from parent values, accounts for approximately 98.5% of all constraints in the circuit. This dominance makes SHA-256 optimization the single highest-leverage target for constraint reduction.
  5. Boolean decomposition overhead: Each to_bits_le() call decomposes a field element into 255 boolean bits, each becoming a separate auxiliary variable. With thousands of such decompositions per challenge, this generates a massive number of boolean witness variables—approximately 340,000 per challenge, or 99.3% of all aux_assignment values.
  6. Poseidon hash constraint accumulation: While Poseidon hashing accounts for only ~1.5% of total constraints, the repeated hashing across Merkle proofs, column hashing, and encoding operations still contributes ~87,000 constraints per challenge.
  7. DensityTracker overhead in MSM: The DensityTracker bitvectors that track which auxiliary variables appear in A and B queries add complexity to the MSM kernel dispatch. The a_aux_density is estimated at 70-90%, while b_aux_density is 30-50%, requiring different MSM strategies for each.
  8. FFT domain padding: The a/b/c vectors must be padded to the next power of two (2^27 = 134M) for the NTT, even though the actual constraint count per partition is ~107M. This padding wastes ~20% of the memory allocated for these vectors.
  9. Per-proof initialization overhead: Each proof generation requires initializing the GPU context, loading kernels, and setting up memory allocations. This overhead is incurred per-proof, even when processing multiple sectors sequentially.

The Three Optimization Proposals

The investigation produced three composable optimization proposals, each targeting different aspects of the memory and throughput problem.

Proposal 1: Sequential Partition Synthesis

The core insight behind Sequential Partition Synthesis is that the current architecture synthesizes all 10 partitions in parallel, holding all a/b/c vectors in memory simultaneously. But the NTT and MSM operations process partitions independently—there is no cross-partition dependency. By synthesizing partitions one at a time, streaming each through the GPU and freeing its memory before synthesizing the next, the peak memory can be dramatically reduced.

Memory impact: Peak memory drops from ~200 GiB to approximately 64-103 GiB, depending on implementation details. The a/b/c vectors go from 10 × 12.8 GiB = 128 GiB to just 1 × 12.8 GiB = 12.8 GiB. The aux_assignment similarly drops from ~2 GiB to ~197 MiB. The SRS (~48 GiB) remains, and overhead is reduced proportionally.

Throughput impact: Sequential processing increases total proving time slightly due to loss of parallelism, but the freed memory headroom enables other optimizations (see Proposal 3) that more than compensate.

Implementation approach: Modify the Curio orchestration layer to dispatch partitions sequentially rather than in parallel. Each partition is synthesized, transferred to GPU, proved, and then the memory is freed before the next partition begins. This requires changes to the Go-Rust FFI interface but no changes to the CUDA kernels themselves.

Proposal 2: Persistent Prover Daemon

The SRS loading overhead of ~60 seconds per proof is a significant inefficiency, especially when proving multiple sectors in sequence. The Persistent Prover Daemon proposal eliminates this overhead by keeping the proving process alive across proofs, maintaining the SRS in GPU memory and the GPU context initialized.

Memory impact: Negligible change to peak memory, but eliminates the ~60s per-proof SRS loading time.

Throughput impact: For batch processing of multiple sectors, this eliminates a fixed per-proof overhead. For a single proof, the daemon can be started once and kept alive for subsequent proofs, amortizing the initialization cost.

Implementation approach: Create a long-lived proving daemon process that accepts proof requests via IPC (e.g., Unix sockets or shared memory). The daemon loads the SRS once at startup and keeps it pinned in GPU memory. When a proof request arrives, the daemon receives the synthesized circuit data, performs the GPU proving steps, and returns the proof. The daemon can also maintain a pool of GPU memory allocations to avoid per-proof allocation overhead.

Proposal 3: Cross-Sector Batching

With the memory freed by Sequential Partition Synthesis, there is headroom to batch multiple sectors' circuits into single GPU invocations. This improves GPU utilization by increasing the work per kernel launch, reducing launch overhead and improving throughput.

Memory impact: Peak memory increases relative to Sequential Partition Synthesis alone, but stays below the original 200 GiB. The exact memory depends on the batch size.

Throughput impact: Projected 2-3× throughput per GPU and 5-6× reduction in cost per proof when combined with Sequential Partition Synthesis. The improvement comes from amortizing kernel launch overhead, improving GPU occupancy, and reducing idle time between partitions.

Implementation approach: Modify the proving pipeline to accept multiple sectors' circuits in a single invocation. The CUDA kernels are already designed to handle large circuits—batching simply means concatenating the constraint systems and processing them as a single larger proof. This requires changes to the partition scheduling logic in Curio and the FFI interface.

The Micro-Optimization Analysis

Beyond the three architectural proposals, the investigation conducted a detailed micro-optimization analysis of the CPU and GPU hotpaths.

CPU Synthesis Hotpaths

The CPU-side circuit synthesis was analyzed at the instruction level. The dominant hotpaths are:

GPU Compute Characteristics

The GPU hotpaths were characterized for the NTT and MSM operations:

Recomputing a/b/c Vectors On-the-Fly

A particularly intriguing micro-optimization considered was whether the a/b/c vectors could be recomputed on-the-fly rather than materialized in memory. The idea is that since the constraint system is deterministic (given the witness values and the circuit structure), the a/b/c evaluations could be recomputed during the NTT rather than stored. However, the analysis concluded that this is not feasible for the current architecture because:

  1. The a/b/c vectors are needed for multiple passes of the NTT algorithm.
  2. The linear combination evaluation requires access to all witness values, which would need to be kept in memory anyway.
  3. The computational cost of re-evaluating all constraints during NTT would exceed the memory savings. This analysis informed the decision to focus on sequential partition processing rather than on-the-fly recomputation.

The Boolean Density Discovery

One of the most striking findings of the investigation was the quantification of boolean density in the aux_assignment vector. Through systematic tracing of the circuit synthesis code, the analysis determined that approximately 99.3% of all aux_assignment values are boolean (0 or 1). This finding has profound implications for optimization:

The Shift in Perspective

Perhaps the most important outcome of this investigation is not any single optimization proposal, but the shift in perspective it represents. The initial framing was about optimizing individual proof generation—reducing memory, speeding up MSM, improving kernel occupancy. But as the investigation deepened, it became clear that the real opportunity was architectural: moving from a batch-oriented, per-proof proving model to a continuous, memory-efficient proving pipeline optimized for heterogeneous cloud rental markets.

In this new vision, the Persistent Prover Daemon maintains a long-lived proving process that can handle proofs from multiple sectors without reloading the SRS. Sequential Partition Synthesis ensures that memory usage stays within the limits of available GPU RAM, enabling deployment on a wider range of hardware. Cross-Sector Batching exploits the freed memory headroom to improve throughput, making each GPU more productive.

This architectural shift is particularly well-suited to the economics of cloud GPU rentals, where RAM cost dominates and per-hour pricing favors workloads that can utilize the full GPU memory without exceeding it. By reducing peak memory from ~200 GiB to ~64-103 GiB, the proposals enable deployment on smaller, cheaper GPU instances. By improving throughput 2-3× per GPU, they reduce the cost per proof. And by eliminating the SRS loading overhead, they make batch processing economically viable.

Conclusion

The deep-dive investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline has produced a comprehensive understanding of the ~200 GiB peak memory footprint and its root causes. The work mapped the entire call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels, identified nine structural bottlenecks, and developed three composable optimization proposals that collectively promise to reduce memory, eliminate overhead, and improve throughput.

The key findings—that 99.3% of aux_assignment values are boolean, that SHA-256 labeling accounts for 98.5% of constraints, and that the a/b/c vectors are ~65× larger than aux_assignment—provide a quantitative foundation for prioritizing optimization efforts. The three proposals (Sequential Partition Synthesis, Persistent Prover Daemon, Cross-Sector Batching) are not independent optimizations but a coherent strategy for transforming the proving pipeline from a memory-hungry batch processor into a continuous, efficient proving system.

For anyone working on Groth16 proof generation at scale, this investigation provides a template for systematic optimization: understand the full call chain, quantify every memory consumer, identify structural bottlenecks, and propose architectural changes that address root causes rather than symptoms. The 200 GiB memory problem is not solved by this analysis alone, but it is now much better understood—and understanding is the first and most important step toward a solution.## References

[1] "Probing the Density of PoRep Circuits: A Strategic Inquiry into Groth16 Memory Optimization" — The opening message that framed the investigation, asking six precise questions about circuit density, witness values, and the relative sizes of a/b/c vectors versus aux_assignment.

[2] "The Moment of Synthesis: Why a Simple Cleanup Command Marks the Pivot from Exploration to Analysis" — Documents the transition from data collection to synthesis, when the assistant declared it had gathered sufficient information.

[3] "Breaking Through the Filesystem Barrier: How a Subagent Secured Source Code for PoRep Circuit Density Analysis" — Chronicles the discovery that cp commands could bypass tool restrictions that blocked direct file reads.

[4] "Quantifying Circuit Density: The Search for Constraint Counts in PoRep's R1CS Pipeline" — The first attempt to quantify constraint costs, searching for to_bits_le and num_constraints in bellperson.

[5] "The Boolean Decomposition Hunt: Tracing Circuit Density Through to_bits_le" — Traces the to_bits_le function through the codebase to understand boolean witness creation.

[6] "Reading the Prover's Blueprint: How DensityTracker and the A/B/C Query Construction Shape PoRep Memory Optimization" — Examines the bellperson prover infrastructure, including DensityTracker and the native/supraseal prover split.

[7] "Peering into the Poseidon Hash Circuit: A Micro-Investigation of Constraint Density in Filecoin's PoRep Pipeline" — Searches for Poseidon constraint counting logic and round parameters in the Neptune library.

[8] "Diving into Circuit Density: The Search for Constraint Counts in Filecoin's PoRep" — A pivotal pivot from structural mapping to quantitative analysis, searching for SHA-256 and Poseidon circuit details.

[9] "Reading the Circuit: How an Assistant Navigated Code Access Restrictions to Analyze PoRep Synthesis" — Documents the breakthrough moment when grep -n "" was discovered as a workaround for reading blocked source files.

[10] "The Bellpepper Discovery: Tracing to_bits_le to Unlock Circuit Density Analysis" — Corrects the search for to_bits_le from bellperson to bellpepper-core, locating the actual implementation.

[11] "Quantifying Circuit Density: The Search for Key Parameters in PoRep's Groth16 Pipeline" — Searches for BASE_DEGREE, EXP_DEGREE, and other critical parameters in the vanilla implementation.

[12] "The Anatomy of a Billion-Constraint Circuit: How Boolean Density Unlocks Memory Optimization in Filecoin's PoRep Prover" — The comprehensive synthesis document (msg 39) that quantifies the 99.3% boolean density finding and its implications.

[13] "Peering into the Poseidon Hash Circuit: A Micro-Investigation of Constraint Density in Filecoin's PoRep Pipeline" (msg 32) — Extracts Poseidon round parameters and constraint counting logic.

[14] "The Bit That Broke the Prover: How a Single Grep Quantified 99.3% Boolean Density in Filecoin's Groth16 Pipeline" (msg 36) — The final verification step that confirmed the boolean density finding.

[15] "Field Into Allocated Bits Le: How field_into_allocated_bits_le Reveals the Root of Circuit Density in Filecoin's PoRep" (msg 37) — Traces the field_into_allocated_bits_le function to confirm the boolean decomposition mechanism.

[16] "The Final Check: How One Bash Command Anchored a Deep-Dive into Filecoin's PoRep Circuit Density" (msg 35) — The SHA-256 constraint verification that anchored the density analysis.

[17] "Unearthing the Constants: How a Single Grep Command Anchored the PoRep Circuit Density Analysis" (msg 26) — Discovery of key circuit constants and parameters.

[18] "The Density Discovery: How a Simple Grep Uncovered the Skeleton of Supraseal's Memory Optimization" (msg 28) — Discovery of density-related parameters in the C2 library's Rust FFI bindings.

[19] "Tracing the Poseidon Round Numbers: A Detective's Journey Through Filecoin's Groth16 Circuit Density" (msg 33) — Detailed extraction of Poseidon round number parameters.

[20] "The Round Numbers File: A Microcosm of Cryptographic Circuit Analysis" (msg 34) — Analysis of the Poseidon round numbers file and its constraint implications.