The Architecture of Memory: Deconstructing the SUPRASEAL_C2 Groth16 Pipeline for Filecoin PoRep

Introduction

In the world of zero-knowledge proof generation, few workloads are as demanding as Filecoin's Proof-of-Replication (PoRep) pipeline. The Groth16 proving system, when applied to storage verification at scale, requires orchestrating hundreds of gigabytes of data across CPU synthesis and GPU acceleration, all while maintaining tight latency and throughput constraints. The SUPRASEAL_C2 pipeline—the C2 (commit) phase of PoRep proof generation—is a marvel of systems engineering, but it carries a staggering ~200 GiB peak memory footprint that makes it economically challenging to deploy in cloud environments where RAM is the dominant cost factor.

This article examines a deep-dive investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline, a session that produced four comprehensive documents mapping the entire call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels. The work identified nine structural bottlenecks, proposed three composable optimization strategies, and conducted micro-optimization analysis of both CPU and GPU hotpaths. The overarching achievement is a fundamental shift in perspective: from optimizing individual proof generation as a batch job, toward architecting a continuous, memory-efficient proving pipeline optimized for heterogeneous cloud rental markets.

The Scale of the Problem

To appreciate the scope of this investigation, one must first understand what the SUPRASEAL_C2 pipeline does. Filecoin storage providers must periodically prove that they are storing the data they have committed to store. This proof takes the form of a Groth16 zk-SNARK, generated over a Rank-1 Constraint System (R1CS) that encodes the storage verification logic. For a 32 GiB sector, the circuit has millions of constraints and tens of millions of variables.

The current pipeline handles this by running ten partition circuits in parallel. Each partition circuit requires approximately 16 GiB of memory during synthesis, and the Structured Reference String (SRS)—the proving parameters—occupies another ~48 GiB in pinned GPU memory. Add in the witness assignments, the linear combination evaluations, and the intermediate buffers, and the total peaks at approximately 200 GiB. This is not merely a large number—it is a structural constraint that determines what hardware the pipeline can run on, how much it costs to operate, and how many proofs can be generated per unit time.

The investigation mapped this memory consumption with surgical precision. The ten parallel partitions each hold their own ProvingAssignment instances, each containing a, b, and c scalar vectors (one per constraint), plus input_assignment and aux_assignment witness vectors. The density trackers—BitVec-backed structures that record which variables are "dense" (non-zero) in each constraint matrix—add further memory. The SRS, loaded from disk via the supraseal-c2 FFI layer, occupies pinned GPU memory that cannot be swapped or paged. Every gigabyte was accounted for, and every gigabyte became a target for optimization.

Mapping the Call Chain: From Curio to CUDA

The first major achievement of this investigation was a comprehensive mapping of the entire call chain, documented with file:line references that trace the path from Curio's Go task orchestration layer down to the CUDA kernels that perform Number Theoretic Transforms (NTT) and Multi-Scalar Multiplications (MSM).

At the top level, Curio's Go daemon manages the proof lifecycle: it receives vanilla proofs (the initial PoRep outputs), deserializes them, and dispatches them to the cuzk proving engine via a Rust FFI bridge. The cuzk engine, implemented in cuzk-core/src/engine.rs, owns the scheduler, GPU workers, and SRS manager. It routes synthesis requests to the appropriate pipeline function based on proof type (PoRep C2, WinningPoSt, WindowPoSt, SnapDeals).

The synthesis phase begins when Circuit::synthesize() is called for each circuit. This invokes the ProvingAssignment::enforce() method (in bellperson/src/groth16/prover/mod.rs), which builds three LinearCombination objects (A, B, C) for each constraint, evaluates them against the witness assignments via eval_with_trackers() (in bellperson/src/lc.rs), and pushes the resulting scalars into prover.a[], prover.b[], and prover.c[]. During evaluation, DensityTracker instances (from ec_gpu_gen::multiexp_cpu) record which variables are dense, producing bitvectors that the GPU phase uses to optimize MSM computation.

After synthesis, the SynthesizedProof struct (in cuzk-core/src/pipeline.rs) carries the provers, assignments, and random values to the GPU phase. The function prove_from_assignments() validates uniformity across all circuits in a batch—they must have identical constraint counts and density profiles—and builds FFI pointer arrays. Finally, supraseal_c2::generate_groth16_proof() is called with raw pointers, crossing the Rust-to-C++ boundary into the CUDA kernels that perform NTT and MSM.

This call chain, documented across dozens of source files spanning five crates, represents the most detailed architectural map of the SUPRASEAL_C2 pipeline ever produced. It reveals not just what the code does, but where every byte of memory is allocated, how data flows between phases, and where the bottlenecks live.

The Nine Bottlenecks

The investigation identified nine structural bottlenecks in the current pipeline, each contributing to the ~200 GiB peak memory footprint and the ~77-second total proof time (after Phase 4 optimizations).

1. Parallel Partition Materialization. The most glaring bottleneck is that all ten partition circuits are synthesized in parallel, each holding its own ~16 GiB of synthesis data. This is the single largest contributor to peak memory. The partitions are independent—they could theoretically be processed sequentially—but the current architecture materializes them all simultaneously because the GPU phase consumes them as a batch.

2. SRS Loading Overhead. The ~48 GiB SRS is loaded from disk into pinned GPU memory for every proof generation. This loading takes approximately 60 seconds and involves reading large .params files, parsing them, and transferring the data to GPU memory. If the proving process were persistent across proofs, this overhead would be paid only once.

3. Density Tracker Materialization. The DensityTracker structs, backed by BitVec bitvectors, are materialized during synthesis and passed to the GPU phase. While the bitvectors themselves are compact, the memory overhead of the density tracking infrastructure—including the BitVec allocations and the popcount computations—adds measurable pressure during the peak memory window.

4. Witness Assignment Duplication. The input_assignment and aux_assignment vectors are stored in the ProvingAssignment during synthesis and then extracted via std::mem::take() for the GPU phase. During the transition, both the synthesis and GPU representations exist simultaneously, doubling the memory for these vectors.

5. Linear Combination Buffer Recycling. The LcVecPool mechanism recycles Vec buffers for linear combination temporaries, eliminating approximately 780 million malloc/free calls per batch. However, the recycling pool itself holds onto memory that could be freed earlier, contributing to peak pressure.

6. Uniformity Validation Overhead. The requirement that all circuits in a batch have identical constraint counts and density profiles means that the system cannot handle heterogeneous circuits efficiently. If one circuit is smaller, the batch must still allocate for the largest circuit, wasting memory.

7. CPU-GPU Transfer Synchronization. The H-to-D (Host-to-Device) transfer of synthesis outputs to GPU memory is a blocking operation that stalls the pipeline. The current architecture does not overlap transfer with computation, leaving the GPU idle during data loading.

8. Background Deallocation Latency. The async deallocation of synthesis data (~130 GB for 10 circuits) happens on background threads, but the memory is not reclaimed until the deallocation completes. During the transition between synthesis and GPU proving, both old and new data coexist.

9. Single-Proof-at-a-Time Throughput. The pipeline is designed to generate one proof at a time per process. There is no mechanism to batch multiple sectors' circuits into a single GPU invocation, leaving GPU utilization below capacity.

The Optimization Proposals

With the bottlenecks identified, the investigation produced three composable optimization proposals that together could transform the pipeline's economics.

Proposal 1: Sequential Partition Synthesis

The most impactful optimization is to break the all-ten-partitions-in-parallel model. Instead of synthesizing all partitions simultaneously, the system would stream partitions one-at-a-time through the GPU. The key insight is that the GPU phase does not need all partitions at once—it can process them sequentially, freeing memory after each partition completes.

Under this proposal, peak memory drops from ~200 GiB to approximately 64-103 GiB, depending on whether the SRS is kept in pinned memory or reloaded per partition. The savings come from eliminating the nine extra partition circuits that were previously held in memory simultaneously. The trade-off is a potential increase in total proof time if the GPU cannot keep up with the sequential stream, but the memory savings are dramatic enough to enable deployment on hardware with 128 GiB or even 96 GiB of RAM.

The implementation would modify synthesize_circuits_batch_with_hint() to yield partitions one at a time, rather than collecting all results before proceeding. The prove_from_assignments() function would need to accept a streaming interface, and the GPU phase would need to handle partial proofs. This is a significant engineering effort, but the memory reduction alone justifies it.

Proposal 2: Persistent Prover Daemon

The second proposal addresses the ~60-second SRS loading overhead. Instead of loading the SRS from disk for every proof, the proving process would be kept alive across proofs. The cuzk daemon would maintain a pool of persistent prover processes, each holding the SRS in pinned GPU memory. When a new proof request arrives, it is routed to an idle prover, which already has the SRS loaded.

This eliminates the SRS loading overhead entirely, reducing per-proof latency by approximately 60 seconds. It also enables the daemon to pre-load SRS for multiple circuit types, reducing the latency of switching between PoRep C2, WinningPoSt, and WindowPoSt proofs.

The implementation requires changes to the process lifecycle management in cuzk-daemon and the addition of a prover pool abstraction. The SRS manager (srs_manager.rs) already caches loaded parameters as Arc<SuprasealParameters>, so the infrastructure partially exists. The missing piece is keeping the GPU context alive across proofs.

Proposal 3: Cross-Sector Batching

The third proposal exploits the memory headroom freed by Sequential Partition Synthesis to batch multiple sectors' circuits into single GPU invocations. Instead of generating one proof per GPU invocation, the system would combine circuits from multiple sectors into a larger batch, amortizing the GPU kernel launch overhead and improving utilization.

The projection is 2-3× throughput per GPU and 5-6× reduction in cost per proof when combined with the other proposals. The key insight is that the GPU's MSM and NTT kernels are most efficient when operating on large batches. By feeding them more work per invocation, the system achieves higher arithmetic intensity and better occupancy.

This proposal requires changes to the circuit batching logic in synthesize_porep_c2_multi() and the uniformity validation in prove_from_assignments(). The uniformity requirement—all circuits must have identical constraint counts—must be relaxed or worked around by padding smaller circuits.

Micro-Optimization Analysis

Beyond the structural proposals, the investigation conducted a detailed micro-optimization analysis of the hotpaths at the instruction level.

CPU Synthesis Hotpaths. The enforce() loop, which evaluates linear combinations for each constraint, was characterized in terms of its instruction mix. The dominant operations are SHA-256 bit manipulation (from the Filecoin circuit's Merkle proof verification), Fr field arithmetic (the BLS12-381 scalar field), and memory loads for the witness assignment vectors. The SHA-256 operations are particularly expensive because they involve bit-level permutations that are difficult to vectorize. The Fr addition and multiplication, while implemented in optimized assembly, still account for a significant fraction of the ~50-second synthesis time.

GPU NTT/MSM Characteristics. The GPU phase is dominated by NTT and MSM operations. The NTT is a bandwidth-bound operation that benefits from the high memory bandwidth of NVIDIA GPUs (up to 2 TB/s on A100). The MSM is a compute-bound operation that scales with the number of points and the density of the constraint matrices. The density trackers produced during synthesis directly affect MSM performance: dense variables require full multi-exponentiation, while sparse variables can use a bucket-based approach.

H-to-D Transfer Patterns. The transfer of synthesis outputs from host memory to device memory was analyzed for bandwidth utilization and latency. The current architecture performs a single large transfer after all synthesis completes, which maximizes bandwidth but introduces a latency bubble. The Sequential Partition Synthesis proposal would replace this with a streaming transfer pattern, potentially overlapping transfer with computation.

Recomputing a/b/c Vectors On-the-Fly. One of the more radical ideas explored was whether the a/b/c scalar vectors could be recomputed on the GPU rather than materialized during CPU synthesis. If the constraint matrices are sparse and the witness assignments are known, the inner products A·w, B·w, C·w could be computed on the GPU directly, eliminating the need to store the a/b/c vectors in host memory. This would reduce peak memory by approximately 30 GiB (the size of the scalar vectors for ten partitions) but would increase GPU compute time. The trade-off analysis showed that this is feasible but requires careful engineering to avoid exceeding GPU memory capacity.

The Shift in Perspective

The most important outcome of this investigation is not any single optimization proposal, but the fundamental shift in perspective that underlies all of them. The current pipeline treats proof generation as a batch job: load everything, compute everything, free everything. This model is simple but wasteful. The ~200 GiB peak memory footprint is not a fundamental requirement of the Groth16 protocol—it is an artifact of the batch-oriented architecture.

The proposed alternative is a continuous, memory-efficient proving pipeline. Instead of materializing all data before processing, the pipeline streams data through the GPU, processing partitions sequentially and freeing memory as soon as each partition completes. Instead of loading the SRS for every proof, the pipeline keeps the prover alive across proofs, paying the loading overhead only once. Instead of generating one proof per invocation, the pipeline batches multiple sectors to amortize GPU overhead.

This shift has profound economic implications. In cloud rental markets, RAM is the dominant cost factor for GPU instances. A machine with 128 GiB of RAM costs significantly less than one with 256 GiB or 384 GiB. By reducing peak memory from ~200 GiB to ~64-103 GiB, the Sequential Partition Synthesis proposal alone could enable deployment on cheaper instances, reducing the cost per proof by 40-60%. Combined with the Persistent Prover Daemon and Cross-Sector Batching, the projected 5-6× reduction in cost per proof would make Filecoin storage proving economically viable at a scale that was previously unattainable.

Conclusion

The deep-dive investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline represents a landmark achievement in systems-level understanding of zero-knowledge proof infrastructure. By mapping the entire call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels, accounting for every gigabyte of the ~200 GiB peak memory footprint, and identifying nine structural bottlenecks, the work provides the foundation for a fundamental re-architecture of the proving pipeline.

The three composable optimization proposals—Sequential Partition Synthesis, Persistent Prover Daemon, and Cross-Sector Batching—offer a path from the current batch-oriented, memory-intensive architecture to a continuous, memory-efficient pipeline optimized for the economic realities of cloud GPU rental. The micro-optimization analysis of CPU synthesis hotpaths, GPU NTT/MSM characteristics, and H-to-D transfer patterns provides the granular understanding needed to implement these proposals without regressions.

The overarching achievement is a shift in perspective: from optimizing individual proof generation toward architecting a continuous proving pipeline. This is not merely an engineering improvement—it is a rethinking of what the pipeline is and what it can become. In an industry where every gigabyte of RAM and every second of latency translates directly into cost, this shift has the potential to transform the economics of Filecoin storage proving.