The SUPRASEAL_C2 Investigation: Decoding Filecoin's Groth16 Pipeline from Go to CUDA

Introduction

In the world of zero-knowledge proofs, few systems push hardware boundaries as aggressively as Filecoin's Proof-of-Replication (PoRep) pipeline. When a single 32 GiB sector requires approximately 200 GiB of peak memory to generate a Groth16 proof — a ratio of 6:1 — every byte demands explanation. This article synthesizes a deep-dive investigation into the SUPRASEAL_C2 pipeline, the GPU-accelerated Groth16 prover at the heart of Filecoin's storage verification system. The investigation traced the full call chain from Curio's Go orchestration layer through Rust FFI bindings into C++/CUDA kernels, mapping memory allocation, identifying nine structural bottlenecks, and producing three composable optimization proposals. At the center of this work was a foundational question: what is the exact relationship between num_constraints and num_aux_variables in the 32 GiB PoRep circuit, and how does that relationship explain the ~200 GiB memory footprint?

The Architecture: A Multi-Language Beast

The SUPRASEAL_C2 pipeline is a study in heterogeneous system design. At the top level, Curio — Filecoin's storage mining orchestrator — manages sealing tasks in Go. When a sector needs a Groth16 proof, Curio calls into supraffi, a Go FFI bridge that invokes Rust code in the supraseal library. The Rust layer, in turn, calls into C++ code that launches CUDA kernels on NVIDIA GPUs. This three-language, two-runtime architecture means that understanding the memory footprint requires tracing data structures across Go heap allocations, Rust's ownership model, C++'s manual memory management, and CUDA's device memory.

The investigation began with a specific question posed by the user: verify the exact relationship between num_constraints and num_aux_variables for a 32 GiB PoRep partition circuit, noting that these "could be VERY different numbers" [35]. This question was not academic — it directly determined the sizes of the largest data structures in the prover: the a/b/c evaluation vectors (each num_constraints elements long), the L polynomial's MSM (num_aux elements), the H polynomial's NTT (domain_size elements), and the A/B polynomial MSMs (density-weighted subsets of variables).

The Evidence Trail: From Go Comments to CUDA Assertions

The investigation proceeded through a systematic traversal of the codebase, reading more than two dozen source files across all three language layers. The trail began in Curio's Go code, where a comment in post_vproof_types.go revealed that the 32 GiB PoRep circuit has approximately 125,279,217 constraints [44]. Another comment in types.go described "10x 130M constraints" for PoRep and "16x 130M constraints" for Snap proofs [44]. These numbers were tantalizing but imprecise — the investigation needed exact values.

The partition count was confirmed in lib/proofsvc/common/types.go, where porepPartitions := 10 is hardcoded [44]. Each partition is an independent Groth16 circuit, meaning a single 32 GiB sector requires 10 separate proofs. This immediately explained a factor of 10 in the memory equation: if each partition's circuit requires ~16 GiB of working memory, 10 parallel partitions would consume ~160 GiB before any shared overhead.

The search then moved to the Rust dependency layer. By examining Cargo.lock, the investigation confirmed that bellperson version 0.26.0 is the proving library, depending on bellpepper-core for constraint system synthesis [15]. This established that the circuit dimensions are not hardcoded in the supraseal codebase but are computed dynamically by the Rust circuit synthesis in storage-proofs-porep. The exact values would require either running the synthesis or parsing the SRS file.

The Rosetta Stone: The SRS File Format

The breakthrough came from reading groth16_srs.cuh, the CUDA header that defines the SRS (Structured Reference String) file format [3]. The SRS file is the proving key for the Groth16 system, containing all the elliptic curve point sets needed for proof generation. The file format, documented at lines 30-57 of the header, reveals the complete structure:

The CUDA Assertions: Formalizing the Relationships

The most definitive evidence came from the CUDA kernel code itself. In groth16_cuda.cu, lines 129-134, a series of assertions establishes the exact relationships between circuit dimensions and SRS point counts [58]:

assert(points_l.size() == p.aux_assignment_size);       // l_count == num_aux
assert(points_a.size() == p.inp_assignment_size + p.a_aux_popcount);
assert(points_b_g1.size() == p.b_inp_popcount + p.b_aux_popcount);
assert(p.a_aux_bit_len == p.aux_assignment_size);
assert(p.b_aux_bit_len == p.aux_assignment_size);
assert(p.b_inp_bit_len == p.inp_assignment_size);

These assertions are runtime checks that would fire if the assumptions were violated. They confirm that l_count equals aux_assignment_size (the number of auxiliary variables) exactly, and that the density bitmasks for the A and B polynomials each cover the full set of auxiliary variables.

The NTT kernel in groth16_ntt_h.cu provides the complementary relationship for constraints [46]:

size_t actual_size = input.abc_size;           // = num_constraints
size_t npoints = points_h.size();              // = h_count from SRS
size_t lg_domain_size = lg2(npoints - 1) + 1;  // smallest k such that 2^k >= npoints
size_t domain_size = (size_t)1 << lg_domain_size;

This tells us that h_count = domain_size - 1, where domain_size is the smallest power of two greater than or equal to num_constraints. The NTT operates on a domain of size domain_size, and the H points are the SRS elements corresponding to the domain_size - 1 non-trivial roots of unity.

The Synthesis: num_constraints ≠ num_aux

After assembling all the evidence, the investigation arrived at a clear conclusion: num_constraints and num_aux_variables are distinct quantities [58]. The codebase treats them separately throughout:

Memory Accounting: Where the 200 GiB Comes From

With the circuit dimensions established, the investigation could perform precise memory accounting. Each of the 10 parallel partitions requires:

The Nine Bottlenecks

The investigation identified nine structural bottlenecks in the pipeline, documented in the background reference [analyzer_summary]:

  1. SRS loading overhead: ~60 seconds per proof spent loading the ~48 GiB SRS file from disk into GPU pinned memory
  2. 10-way partition parallelism: Peak memory scales linearly with partition count; all 10 partitions are processed simultaneously
  3. Full materialization of a/b/c vectors: Each vector is num_constraints elements (32 bytes each), totaling ~12.5 GiB per partition when fully materialized
  4. Density bitmask storage: The A and B density bitmasks each cover aux_assignment_size bits (~130M bits ≈ 16 MiB per bitmask), stored in host memory
  5. GPU kernel launch overhead: Multiple kernel launches per partition (NTT, MSM for H, L, A, B_G1, B_G2) with synchronization barriers
  6. Host-to-device transfer: Large data transfers from pinned host memory to GPU device memory for each partition
  7. FFT domain padding: The domain size is 2^27 = 134,217,728, while num_constraints is ~130,902,761, meaning ~3.3 million unused slots in the FFT buffer (~100 MiB wasted per partition)
  8. SRS file format inefficiency: The B_G2 points are stored at 192 bytes each (G2 affine) while B_G1 points are 96 bytes (G1 affine), doubling the memory for the B_G2 set
  9. Per-partition proving key duplication: Each partition loads its own copy of the SRS points, even though the points are identical across partitions of the same sector size

Three Optimization Proposals

Building on the bottleneck analysis, the investigation produced three composable optimization proposals [analyzer_summary]:

Proposal 1: Sequential Partition Synthesis

The most impactful optimization breaks the all-10-partitions-in-parallel model. Instead of processing all partitions simultaneously, the GPU processes partitions one at a time, reusing the same working buffers. This reduces peak memory from ~200 GiB to approximately 64-103 GiB — a 50-68% reduction. The trade-off is increased total proof time (since partitions are serialized), but the reduction in memory pressure allows the use of cheaper GPU instances with less RAM, potentially lowering the cost per proof.

The key insight is that the 10 partitions are independent Groth16 proofs — there is no cryptographic reason to process them in parallel. The parallelism was an artifact of the Curio orchestration model, which launches all partition proofs concurrently for throughput. By streaming partitions sequentially through the GPU, the peak memory drops to the per-partition requirement plus shared SRS data.

Proposal 2: Persistent Prover Daemon

The ~60 seconds per proof spent loading the SRS file is pure overhead that can be eliminated. The Persistent Prover Daemon keeps the proving process alive across proofs, maintaining the SRS data in GPU pinned memory. When a new proof request arrives, the daemon skips the SRS loading step entirely. For a workload generating proofs for multiple sectors, this eliminates the ~60s overhead per proof, translating directly to throughput improvement.

The daemon architecture also enables pre-allocation of GPU working buffers, avoiding the allocation/deallocation cycle that contributes to memory fragmentation and latency.

Proposal 3: Cross-Sector Batching

The memory freed by Sequential Partition Synthesis (Proposal 1) can be repurposed for batching. Instead of processing one partition at a time, the GPU can process multiple sectors' circuits in a single invocation, amortizing kernel launch overhead and improving GPU utilization. The investigation projects 2-3× throughput per GPU and 5-6× reduction in cost per proof when combined with the other proposals.

Cross-sector batching exploits the fact that the SRS is shared across all sectors of the same size. Once the SRS is loaded (and kept resident by the Persistent Prover Daemon), adding more sectors' circuits to a batch increases the computation proportionally but not the SRS memory overhead.

Micro-Optimization Analysis

A final round of micro-optimization analysis examined the CPU and GPU hotpaths at the instruction level [analyzer_summary]:

CPU synthesis hotpaths: The enforce() loop in bellperson's ProvingAssignment is the dominant CPU cost during circuit synthesis. Each enforce() call produces one element in each of the a, b, and c vectors, requiring Fr field arithmetic and SHA-256 bit manipulation for the density bitmasks. The analysis found that the SHA-256 operations (used for the Fiat-Shamir transform in the constraint system) account for a significant fraction of CPU time, and that Fr addition in the BLS12-381 scalar field is memory-bound due to the 256-bit operand size.

GPU NTT/MSM characteristics: The NTT for the H polynomial is bandwidth-bound on current NVIDIA GPUs (A100, H100), achieving approximately 60-70% of peak memory bandwidth. The MSM operations for L, A, and B polynomials are compute-bound, limited by the number of elliptic curve point additions per second. The analysis found that the L MSM (size num_aux) and the H NTT (size domain_size) have similar computational costs, confirming that neither dominates the GPU time.

H-to-D transfer patterns: The transfer of SRS points from pinned host memory to GPU device memory is a significant contributor to wall-clock time. The analysis found that the transfer is bandwidth-limited on PCIe Gen4 (approximately 25 GB/s), and that overlapping transfers with computation (via CUDA streams) could hide some of this latency.

On-the-fly a/b/c recomputation: The investigation evaluated whether the a, b, and c vectors could be recomputed on-the-fly rather than materialized in memory. The analysis concluded that recomputation is feasible for the a and b vectors (which are sparse, with density determined by the bitmasks), but the c vector is dense and would require full materialization. The trade-off is between memory (saving ~12.5 GiB per partition) and compute (re-running the constraint evaluation). For memory-constrained scenarios, partial recomputation of a and b could be worthwhile.

The Broader Implications

The SUPRASEAL_C2 investigation represents a shift from optimizing individual proof generation toward architecting a continuous, memory-efficient proving pipeline. The three proposals are designed to be composable — Sequential Partition Synthesis reduces peak memory, the Persistent Prover Daemon eliminates SRS loading overhead, and Cross-Sector Batching improves throughput by exploiting the freed memory headroom. Together, they project a 5-6× reduction in cost per proof on heterogeneous cloud rental markets where RAM cost dominates.

The investigation also produced a comprehensive background reference document mapping the entire call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels, with file:line references for every major component. This document serves as a guide for future optimization work and as a case study in multi-language systems analysis.

Conclusion

The SUPRASEAL_C2 investigation demonstrates that understanding a complex system's memory footprint requires tracing data structures across language boundaries, from configuration constants to runtime assertions. The foundational question — the relationship between num_constraints and num_aux_variables — was answered not by finding a single documented constant but by synthesizing evidence from a dozen source files across Go, Rust, C++, and CUDA. The answer — that they are distinct quantities with separate computational roles and separate SRS arrays — unlocked the memory accounting that explained the ~200 GiB peak and enabled the three optimization proposals.

The investigation's methodology — systematic traversal of the call chain, cross-referencing multiple sources of truth, and explicit acknowledgment of uncertainty — is a template for reverse-engineering any complex system. In a domain where a single wrong assumption can lead to months of wasted effort optimizing the wrong bottleneck, this rigor is not just good practice — it is essential.

References

[1] The Hunt for Circuit Dimensions: Tracing a Grep Through Filecoin's PoRep Pipeline [3] Reading the SRS File: Uncovering the Circuit Dimensions of Filecoin's Groth16 Proving Key [5] Tracing the C2 Call Chain: A Strategic Pivot in Circuit Dimension Analysis [15] Peering into the Proving System: How a Single Grep Command Uncovered bellperson's Role in Filecoin's Groth16 Pipeline [35] The Circuit Dimensions Question: A Microcosm of Reverse-Engineering a Zero-Knowledge Proof Pipeline [44] The Moment of Synthesis: Connecting Partitions to Constraints in the SUPRASEAL_C2 Pipeline [46] The Pivot Point: Tracing the NTT Computation Path in SUPRASEAL_C2's Groth16 Pipeline [54] The Pivotal Synthesis: Decoding the Constraint-to-Variable Relationship in Supraseal's Groth16 Pipeline [58] Disambiguating Constraints and Auxiliary Variables in the SUPRASEAL_C2 Groth16 Pipeline [66] Reading the SRS File: Uncovering the Circuit Dimensions of Filecoin's Groth16 Proving Key