From 200 GiB to Pipeline: The SUPRASEAL_C2 Optimization Journey

Introduction

In the world of zero-knowledge proofs, few computations are as memory-intensive as Filecoin's Proof-of-Replication (PoRep) Groth16 proof generation. The SUPRASEAL_C2 pipeline, a CUDA-accelerated proving system powering Filecoin's storage verification, routinely consumes approximately 200 GiB of peak memory for a single proof. For operators renting cloud GPU instances—where RAM is billed by the gigabyte—this memory footprint translates directly into operational cost. Reducing it is not merely an engineering exercise; it is an economic imperative.

This article synthesizes a deep-dive investigation into the SUPRASEAL_C2 pipeline, covering two major phases of work: the completion of Phase 3 (End-to-End GPU validation of cross-sector batching) and the launch of Phase 4 (compute-level optimizations). The investigation mapped the full call chain from Curio's Go task layer through Rust FFI (bellperson) into C++/CUDA kernels (supraseal-c2), identified nine structural bottlenecks, produced three composable optimization proposals, and began implementing concrete code changes. Along the way, a critical dependency ambiguity was resolved—revealing that two seemingly identical local source directories were entirely unused by the Rust build—and an initial benchmark of the Phase 4 changes showed a regression that prompted immediate instrumentation and refinement.

The overarching achievement of this work is a conceptual shift: from optimizing individual proof generation as an isolated event, toward architecting a continuous, memory-efficient proving pipeline optimized for the heterogeneous cloud rental markets where RAM cost dominates.

Phase 3: Validating Cross-Sector Batching

The Phase 3 campaign was a systematic End-to-End (E2E) GPU testing effort on an RTX 5070 Ti with real 32 GiB PoRep data. The goal was to validate the cross-sector batching architecture—a proposal that exploits the observation that multiple sectors' circuits can be synthesized together into a single GPU invocation, amortizing the fixed costs of SRS loading and kernel launch overhead across multiple proofs.

Baseline Characterization

Before testing the batching feature, the team first characterized the baseline single-proof behavior. The daemon showed a peak RSS of 202.9 GiB during synthesis, dropping to ~45 GiB idle with the SRS (Structured Reference String) resident in pinned GPU memory. This baseline established the memory accounting: approximately 10 parallel partition circuits at ~16 GiB each, plus ~48 GiB of SRS data in pinned memory, plus overhead for the a/b/c vectors and intermediate allocations.

Four Systematic Tests

Four tests were executed against the daemon running with max_batch_size=2:

Timeout flush: Verified that the BatchCollector correctly flushes a single proof after the configured max_batch_wait_ms of 30,000ms. The actual flush occurred at 30,258ms—within tolerance—confirming the timeout mechanism works for single-proof scenarios.
Batch=2: Two concurrent proofs were submitted and synthesized together as 20 circuits (10 per proof) in 55.3s—identical to the time required for 10 circuits in a single proof. This is the headline result: synthesis cost is fully amortized across sectors. The GPU time scaled linearly (34s per sector), but the CPU-bound synthesis—which dominated the single-proof runtime—was shared.
3-proof overflow: Three proofs were submitted, demonstrating correct batch-of-2 + overflow behavior. The first two proofs were batched together, and the third was deferred to the next batch window. Pipeline overlap between the synthesis and GPU phases was confirmed.
WinningPoSt bypass: A non-batchable proof type (WinningPoSt) was submitted and confirmed to skip the BatchCollector entirely, completing in 0.8s total. This validated the architecture's ability to handle heterogeneous workloads.

Quantitative Results

The numbers speak for themselves. Batch=2 achieves 1.42× throughput improvement (62.7s/proof amortized vs 89s baseline), with synthesis cost fully shared while GPU time scales linearly. Memory peaks at ~360 GiB for batch=2 (vs 203 GiB single), closely matching the predicted ~408 GiB estimate when accounting for SRS overhead. All proof outputs were valid 1920-byte Groth16 proofs, and the daemon logs confirmed every stage of the Phase 3 architecture—synth_batch_full, synthesize_porep_c2_multi{num_sectors=2}, total_circuits=20, and split_batched_proofs producing correct per-sector boundaries.

The results were compiled into cuzk-project.md and committed as 353e4c2a, marking Phase 3 as fully validated and documented.

The Dependency Revelation

Before Phase 4 could begin in earnest, a critical ambiguity needed resolution. The Curio repository contained two directories—extern/supra_seal/c2/ and extern/supraseal/c2/—both appearing to define the supraseal-c2 crate with identical Cargo.toml files. A subagent was dispatched to determine which one the Rust build actually used.

The investigation, spanning messages 0 through 8 of a subagent session, systematically traced the dependency chain from cuzk-core through bellperson to supraseal-c2. The key discovery came from the Cargo.lock file: the supraseal-c2 entry showed source = "registry+https://github.com/rust-lang/crates.io-index" and version 0.1.2, while both local directories declared version 0.1.0. In Cargo's lock file format, a source field of "registry+..." means the package was fetched from crates.io—not from a local path. Local-path dependencies have no source field at all.

The assistant methodically ruled out every alternative mechanism: no [patch.crates-io] entry for supraseal-c2 existed in the workspace Cargo.toml, no .cargo/config.toml overrides were present, and bellperson's Cargo.toml declared supraseal-c2 by version only ("0.1.0"), not by path. The conclusion was definitive: neither local directory is used by the Rust build. Both are vestigial—likely remnants from an earlier development phase or artifacts used by the Go/CGO FFI build of the main Curio binary.

This finding had immediate practical consequences for the optimization work. Any modifications to supraseal-c2 would require either adding a [patch.crates-io] entry to redirect to a local fork, or working with the published crate version 0.1.2. The team proceeded to create extern/bellpepper-core/ and extern/supraseal-c2/ directories, patching them into the workspace via [patch.crates-io].

Phase 4: Compute-Level Optimizations

With the dependency chain clarified, Phase 4 began implementing the highest-impact items from the optimization proposal document. Four optimizations were selected for initial implementation:

A1: SmallVec for LC Indexer

The Linear Combination (LC) Indexer in bellpepper-core was using Vec<(usize, Scalar)> to store constraint indices and coefficients. Each partition's synthesis involves approximately 780 million such entries, each requiring a heap allocation. The optimization replaced Vec with SmallVec<[(usize, Scalar); 4]>, which stores small arrays inline (up to 4 elements) without heap allocation, spilling to the heap only for larger arrays. This eliminates the vast majority of the ~780M heap allocations per partition, reducing allocation overhead and memory fragmentation.

A2: Pre-sizing ProvingAssignment

The ProvingAssignment data structure—which holds the a/b/c vectors during synthesis—was growing dynamically via repeated push operations, causing approximately 32 GiB of reallocation copies across all partitions. A new_with_capacity constructor was added, allowing the caller to pre-allocate the exact capacity needed. This eliminates the reallocation overhead entirely.

A4: Parallelize B_G2 CPU MSMs

The B_G2 multi-scalar multiplication (MSM) is a CPU-bound computation that runs on the G2 curve (which is slower than G1). The original implementation processed the MSMs sequentially. The optimization changed this to use groth16_pool.par_map, parallelizing across available CPU cores. Since G2 operations are compute-bound rather than memory-bound, this parallelization directly reduces wall-clock time.

B1: Pin a/b/c Vectors

The a/b/c vectors—each approximately 4 GiB—are allocated by Rust and passed to the CUDA kernel. Under normal conditions, these buffers are pageable, meaning the GPU driver must copy them through a temporary pinned buffer (a D-to-D copy via a staging buffer). By adding cudaHostRegister/cudaHostUnregister around the Rust-provided arrays, the buffers are pinned in physical memory, allowing the GPU to access them directly via DMA. This eliminates the staging copy.

D4: Per-MSM Window Tuning

The original code used a single msm_t instance with fixed window sizes for all MSM operations. The optimization splits this into three msm_t instances, each tuned for the specific popcount characteristics of the L, A, and B_G1 MSMs. Different window sizes optimize for different sparsity patterns, reducing the number of point additions.

The Regression and Recovery

An initial E2E single-proof benchmark on the RTX 5070 Ti with real 32 GiB PoRep data revealed a regression: 106s total vs the 89s baseline from Phase 3. The breakdown was instructive:

Synthesis: Rose from 54.7s to 61.6s. The culprit was A2's pre-sizing: the new_with_capacity constructor allocated 328 GiB upfront (all partitions simultaneously), triggering page-fault storms as the kernel faulted in all that memory at once. The dynamic push approach, while causing reallocation copies, had the advantage of lazy allocation—only faulting pages as they were actually written.
GPU time: Rose from 34s to 44.2s. The B1 pinning optimization (cudaHostRegister) was called 30 times × 4 GiB each, and each call has non-trivial overhead for pinning/unpinning memory. The cumulative cost of these operations exceeded the savings from eliminating staging copies. The team responded immediately: A2's hint usage was reverted in the synthesis call sites (keeping the API available for future use), and detailed phase-level timing instrumentation was added to the CUDA code using std::chrono. The instrumentation measures prep_msm, NTT+MSM_H, batch_addition, tail MSMs, B_G2, proof assembly, and pin/unpin separately. This granular timing data will enable precise A/B testing to isolate each optimization's true impact and identify which changes need refinement before the final Phase 4 validation.

The Broader Architecture: Three Optimization Proposals

Beyond the implemented code changes, the investigation produced three composable optimization proposals that together represent a fundamental rethinking of the proving pipeline:

Proposal 1: Sequential Partition Synthesis

The current architecture processes all 10 partition circuits in parallel, each consuming ~16 GiB. Sequential Partition Synthesis breaks this model by streaming partitions one-at-a-time through the GPU, reusing the same memory for each partition. This reduces peak memory from ~200 GiB to approximately 64-103 GiB—a 50-68% reduction—with only a modest increase in wall-clock time (since the GPU is the bottleneck, not the CPU).

Proposal 2: Persistent Prover Daemon

Each proof generation currently loads the SRS (~48 GiB) from disk into pinned GPU memory, taking approximately 60 seconds per proof. The Persistent Prover Daemon keeps the proving process alive across proofs, maintaining the SRS in GPU memory and eliminating the reloading overhead. For high-throughput environments processing many proofs, this effectively removes the 60s fixed cost from the per-proof budget.

Proposal 3: Cross-Sector Batching

The Phase 3-validated batching architecture exploits the freed memory headroom from Sequential Partition Synthesis to batch multiple sectors' circuits into single GPU invocations. Projections indicate 2-3× throughput per GPU and 5-6× reduction in $/proof when combined with the other proposals. The key insight is that synthesis is CPU-bound and scales sublinearly with circuit count, while GPU operations scale linearly—making batching a natural fit for the heterogeneous CPU+GPU architecture.

Micro-Optimization Analysis

A final round of analysis examined the computational hotpaths at the instruction level:

CPU synthesis hotpaths: The enforce() loop dominates, performing SHA-256 bit manipulation for constraint index hashing and Fr field arithmetic for linear combination evaluation. The SHA-256 operations are particularly expensive because they operate on individual bits rather than bytes, and the Fr arithmetic involves modular reduction after each addition.
GPU NTT/MSM characteristics: The Number Theoretic Transform (NTT) is memory-bandwidth-bound, while the MSM is compute-bound with significant point addition overhead. The per-MSM window tuning (D4) directly addresses the MSM compute characteristics.
H-to-D transfer patterns: The host-to-device transfers for the a/b/c vectors are bandwidth-bound on the PCIe link. The pinning optimization (B1) was intended to reduce overhead here, but the cudaHostRegister calls themselves introduced latency that outweighed the savings.
Recomputing a/b/c vectors on-the-fly: An analysis was conducted on the feasibility of recomputing the a/b/c vectors from the constraint system rather than materializing them in memory. While this would eliminate the ~12 GiB of vector storage entirely, the computational cost of recomputation (re-running the synthesis for each vector) was deemed prohibitive for the current architecture.

Conclusion

The SUPRASEAL_C2 investigation represents a comprehensive effort to understand and optimize one of the most memory-intensive computations in the Filecoin ecosystem. The work spans from Go orchestration layers through Rust FFI boundaries into C++/CUDA kernels, from dependency resolution in Cargo's lock file to instruction-level analysis of SHA-256 bit manipulation.

The Phase 3 results validated the cross-sector batching architecture with a 1.42× throughput improvement, while Phase 4's initial implementation revealed the complexity of optimizing a system at this scale—where a well-intentioned pre-sizing optimization can trigger page-fault storms, and a pinning optimization can introduce overhead that exceeds its savings. The addition of detailed phase-level timing instrumentation ensures that subsequent iterations will be guided by data rather than intuition.

The three optimization proposals—Sequential Partition Synthesis, Persistent Prover Daemon, and Cross-Sector Batching—form a coherent vision for a continuous, memory-efficient proving pipeline. Together, they project a 5-6× reduction in cost per proof, achieved not through any single breakthrough but through the compounding effect of multiple architectural changes, each addressing a specific bottleneck identified in the pipeline analysis.

The overarching achievement of this work is the conceptual shift it represents: from treating proof generation as an isolated, monolithic event to be optimized in isolation, toward architecting a continuous pipeline where memory is streamed, resources are shared across sectors, and the proving process itself persists across proofs. In the cloud rental markets where Filecoin storage providers operate, this distinction—between an event and a pipeline—is the difference between paying for 200 GiB of RAM per proof and paying for a fraction of that.