The SUPRASEAL_C2 Groth16 Optimization Journey: From 200 GiB Peak Memory to a Continuous Proving Pipeline

Introduction

Filecoin's Proof-of-Replication (PoRep) mechanism is the cryptographic foundation that ensures storage providers are honestly storing the data they claim. But this trust comes at a staggering computational price. A single 32 GiB sector's C2 proof—the second and most computationally intensive phase of PoRep—consumes nearly 200 GiB of RAM, demands a high-end GPU for minutes of compute, and forces operators to provision machines with 256 GiB or more of memory. In cloud rental markets where RAM is the dominant cost factor, this memory wall is the primary constraint on profitability.

This article synthesizes a deep-dive investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline—a journey that began with a single, dense user prompt and culminated in four comprehensive documents, three composable optimization proposals, and a fundamental rethinking of how Filecoin proofs should be generated. The investigation spanned the full call chain from Curio's Go task layer through Rust FFI (bellperson) into C++/CUDA kernels (supraseal-c2), accounting for every GiB of the ~200 GiB peak memory, identifying nine structural bottlenecks, and proposing a continuous, memory-efficient proving pipeline architecture optimized for heterogeneous cloud rental markets.

The Seed of Investigation

The journey began with a single message from a user ([msg 0]): "Dive into rust fil proofs / supraseal code — Investigate SUPRASEAL_C2 code path, especially understand proof synthesis step (cpu before GPU used), the memory usage (nearly 200GB peak for PoRep 10 partition Snark) and look for opportunities to reduce data / pipeline harder / merge/parallelise work. Mostly understand the current code deeply. Entry is extern/filecoin-ffi Commit2."

This 38-word prompt was a masterclass in expert-level problem specification. It identified a concrete pain point (200 GiB memory), scoped the investigation to a specific code path (SUPRASEAL_C2 via filecoin-ffi Commit2), isolated a suspected leverage point (CPU synthesis before GPU), and established a clear priority (understand first, optimize second). The assumptions embedded in this prompt—about the target, the entry point, and the nature of the problem—were sophisticated enough to guide a multi-week investigation, yet contained the seeds of the investigation's most important discovery: that the parallelism model, not synthesis itself, was the root cause of the memory problem.

The user's framing was particularly notable for what it did not say. It did not prescribe a solution. It did not jump to conclusions about what was wrong. It identified a symptom (200 GiB memory) and a suspected area (CPU synthesis), but left the diagnosis open. This restraint was critical: it allowed the investigation to follow the evidence wherever it led, rather than confirming a pre-existing hypothesis.

The Parallel Exploration Strategy

The assistant's response ([msg 1]) was not a linear code reading exercise but a carefully orchestrated burst of parallel exploration. Faced with a system spanning four programming languages (Go, Rust, C++, CUDA), multiple FFI boundaries, and a complex feature flag propagation chain, the assistant decomposed the investigation into four concurrent threads:

  1. FFI Entry Point: Tracing the Go-side SealCommitPhase2 function through CGO bindings into the Rust libfilcrypto.a library
  2. Supraseal C2 Rust Code: Analyzing the Rust-side implementation in supraseal-c2, including SRS loading and the generate_groth16_proof function
  3. Proof Synthesis Details: Investigating the CPU-intensive circuit synthesis step in bellperson's supraseal.rs
  4. Build System: Mapping how supraseal is built and integrated into the Curio/filecoin-ffi build system This parallel exploration strategy was a methodological necessity. The system was too large and too interconnected for any single sequential read to capture. By decomposing the investigation into four orthogonal axes, the assistant ensured that each thread could focus deeply on its domain while the results could be synthesized into a unified understanding. The assistant's approach also demonstrated a sophisticated understanding of how to explore unfamiliar codebases. Rather than reading every file from top to bottom, it followed the call chain from the known entry point outward, tracing function calls, data structures, and memory allocations as they appeared. This "breadcrumb trail" approach—following the data from allocation to deallocation—is far more efficient than exhaustive reading, especially when the goal is to understand a specific phenomenon like memory usage.

The Anatomy of a 200 GiB Memory Problem

The synthesis of the four exploration threads produced a watershed moment ([msg 5]): a comprehensive analysis that identified the root cause of the memory problem and provided a complete memory budget. The executive summary was devastatingly clear: "Peak memory is driven by 10 partition circuits being fully synthesized in parallel on CPU, each consuming ~16 GiB, totaling ~160 GiB of host RAM just for the ProvingAssignment data. The SRS (proving key) adds another ~30-70 GiB of pinned host memory. This explains the ~200 GiB peak."

The detailed memory budget broke down every significant consumer:

| Component | Memory | Location | Lifetime | |-----------|--------|----------|----------| | 10 × ProvingAssignment (a,b,c) | ~120 GiB | Rust heap | Synthesis → end of C++ call | | 10 × aux_assignment | ~40 GiB | Rust heap (Arc) | Synthesis → end of C++ call | | 10 × DensityTrackers | ~0.5 GiB | Rust heap (bitvec) | Synthesis → end of C++ call | | SRS pinned host memory | ~48 GiB | cudaHostAlloc | Cached (persistent) | | split_vectors bitmaps | ~2 GiB | C++ heap | Duration of C++ fn | | tail_msm bases + scalars | ~20 GiB | C++ heap | Duration of C++ fn | | GPU: NTT workspace | 8-12 GiB | GPU VRAM | Per-circuit NTT | | GPU: batch add temp | ~2 GiB | GPU VRAM | Per batch add call | | GPU: MSM temp | ~2-4 GiB | GPU VRAM | Per MSM invocation | | Estimated host peak | ~200-230 GiB | | | | Estimated GPU peak | ~12-16 GiB | | |

The critical finding was the collect() barrier: all 10 circuits must finish synthesis before any GPU work begins, due to the .collect() barrier in synthesize_circuits_batch(). This is the root cause of the peak memory. If circuits could be pipelined to the GPU one-by-one, peak memory would drop dramatically.

The message also mapped the complete call chain from Curio's Go task layer down to CUDA kernels, providing a navigable map of a system that spans four languages and multiple FFI boundaries. It identified nine structural bottlenecks and ranked eight optimization opportunities by memory savings, performance impact, and difficulty.

The Nine Structural Bottlenecks

The investigation identified nine distinct bottlenecks in the current pipeline:

  1. Parallel synthesis memory wall: All 10 partitions synthesized simultaneously on CPU, consuming ~160 GiB before any GPU work begins
  2. SRS loading per proof: ~48 GiB loaded from disk and deserialized for every single proof, taking 30-90 seconds
  3. No GPU/CPU overlap: GPU sits idle during the entire ~60-120s CPU synthesis phase
  4. Duplicate a/b/c storage: The a/b/c vectors (~12 GiB per partition) are stored in Rust heap and then re-allocated in C++ heap for GPU transfer
  5. Full aux_assignment retention: All 10 aux_assignments (~40 GiB total) held in memory until proof completion, even though each is only needed during its partition's MSM phase
  6. DensityTracker overhead: Bit vectors tracking variable density consume ~0.5 GiB and are recomputed rather than streamed
  7. Single-sector batching limit: The batch size is fixed at 10 partitions (one sector), preventing amortization of GPU fixed costs across multiple sectors
  8. Process-per-proof model: Each C2 proof spawns a child process, preventing any cross-proof optimization or state reuse
  9. No recomputation vs. storage tradeoff: The a/b/c vectors are always materialized in full, even though partial recomputation might reduce peak memory Each bottleneck was documented with its location in the codebase, its contribution to peak memory or latency, and a proposed approach for mitigation. This systematic cataloging transformed the investigation from a single finding ("memory is high") into a prioritized action plan.

The "Think Bigger" Pivot

The user's response to this analysis was transformative. Rather than accepting the initial findings as complete, the user challenged the assistant to "think bigger" ([msg 6]), asking for 2-3 high-impact improvements that consider the whole picture of a constant proving pipeline optimized for maximum proof throughput per dollar of system cost.

This pivot shifted the investigation from understanding a specific memory problem to architecting a fundamentally different proving pipeline. The assistant responded by asking about the target deployment model ([msg 9]), and the user answered: "Proofshare marketplace"—a heterogeneous fleet where all dimensions matter, with RAM as the top constraint.

A critical verification task ([msg 8]) corrected a major misunderstanding: the aux_assignment vector contained approximately 130 million elements per partition circuit—roughly 4 GiB—not the ~6 million initially assumed. This changed the memory picture dramatically and informed everything that followed. The correction was possible because the investigation had traced the actual data structures rather than relying on documentation or assumptions.

The "think bigger" pivot was a turning point in the investigation. It forced the analysis to move beyond the question "how do we reduce memory?" to the more fundamental question "what should the architecture of a continuous proving pipeline look like?" This reframing opened up a much larger design space, including changes to the process model, the scheduling model, and the batch composition strategy.

The Three Optimization Proposals

The synthesis of these insights produced three composable optimization proposals ([msg 10]), each building on the previous to form a coherent architectural vision.

Proposal 1: Sequential Partition Synthesis

The first and most foundational proposal breaks the all-at-once batch model. Instead of synthesizing all 10 partitions in parallel and sending them to the GPU together, the assistant proposed processing them one at a time in a pipeline:

Partition 0: Synthesize → NTT+H on GPU → extract aux → free a,b,c
Partition 1: Synthesize → NTT+H on GPU → extract aux → free a,b,c  
...
Partition 9: Synthesize → NTT+H on GPU → extract aux → free a,b,c
Then: Run all MSMs (L, A, B) using stored aux assignments
Then: Assemble 10 proofs

The memory savings are dramatic. Instead of 10 partitions' a,b,c vectors coexisting (~120 GiB), only one partition's vectors exist at a time (~12 GiB). The aux_assignments must still be accumulated (~40 GiB for all 10), but the peak drops from ~200 GiB to ~64-103 GiB—fitting comfortably in a 128 GiB machine.

The key insight enabling this proposal is that the 10 partition circuits are identical in structure (same R1CS, same constraint system) with different witness values. The current code synthesizes all 10 in parallel, but there is no mathematical reason they must all exist simultaneously. The decision to batch them was a design choice for GPU efficiency, but the memory cost of that choice is enormous.

The pipeline overlap also improves GPU utilization. Today, the GPU is idle for the entire ~60-120s synthesis phase. With sequential processing, the GPU can start working on partition 0's NTT+H as soon as it finishes synthesis, while the CPU moves on to synthesizing partition 1. The GPU is never idle after the first partition.

The implementation challenge is primarily in the Rust/C++ interface. Currently, synthesize_circuits_batch() returns all 10 ProvingAssignments at once. To implement sequential processing, the synthesis function would need to yield assignments one at a time, or the C++ code would need to accept individual assignments and manage the pipeline state. The assistant estimated 2-3 weeks of engineering effort for this proposal, with the bulk of the work being in refactoring the batch interface.

Proposal 2: Persistent Prover Daemon

The second proposal addresses a different inefficiency: the per-proof SRS loading overhead. Today, each C2 proof spawns a child process (curio ffi) that loads the ~48 GiB SRS from disk, deserializes it into pinned memory, computes one proof, and exits. For a proofshare node processing 50+ proofs per day, this means loading 48 GiB hundreds of times.

The proposal is to replace the child-process-per-proof model with a long-lived proving daemon that:

Proposal 3: Cross-Sector Proof Batching

The third proposal exploits the freed memory headroom from Proposals 1 and 2 to batch partitions across sectors. The supraseal generate_groth16_proofs_c() already handles batches of circuits—today it batches the 10 partitions of one sector. But there's nothing stopping it from batching partitions across sectors.

If three sectors are waiting for C2, instead of processing them sequentially, the system could do a mega-batch of 30 partitions across 3 sectors. The GPU's MSM throughput scales sub-linearly with batch size—more circuits amortize fixed costs (SRS transfer, kernel launch overhead, memory allocation).

The combined impact of all three proposals is a projected 5-6× reduction in $/proof, enabling the use of cheaper 128 GiB machines and a 2.4-3.6× increase in throughput per GPU.

The cross-sector batching proposal also opens up interesting scheduling questions. Should the system wait for a full batch of sectors before starting, or should it start processing as soon as any sector is ready? The answer depends on the latency requirements of the marketplace. For a proofshare marketplace where proofs are submitted asynchronously, waiting for a full batch might be acceptable. For a system with strict per-sector latency requirements, a hybrid approach—start processing immediately but allow late-arriving sectors to join the batch if they arrive before the GPU work begins—might be more appropriate.

The Documentation Threshold

The user's response to the proposals was concise but transformative: "Write down c2-improvement-background.md with all relevant insights into the realm of possible optimizations and pointers; write down c2-optimization-proposal-X.md with deeper details of each option" ([msg 11]).

This request marked the transition from analysis to artifact—the moment when investigative findings and architectural proposals were formalized into permanent, shareable, actionable documents. The assistant spent the next several messages writing these documents, creating four files totaling thousands of lines of analysis, code references, implementation plans, and performance projections.

The background document (c2-improvement-background.md) captured the full pipeline architecture with file:line references, memory accounting, and nine identified structural bottlenecks. The three proposal documents (c2-optimization-proposal-1.md, c2-optimization-proposal-2.md, c2-optimization-proposal-3.md) provided detailed implementation plans, estimated timelines (2-3 weeks per proposal), and quantitative projections of impact.

The delivery message ([msg 21]) summarized everything in a structured, digestible form. It curated the key findings: memory reduction from 200 GiB to 64 GiB, elimination of ~60s SRS loading per proof, 2-3× throughput improvement, and 5-6× reduction in $/proof. It reinforced the composability argument—that these proposals are not independent optimizations but a layered stack where each layer enables the next.

The documentation phase was crucial for several reasons. First, it forced the analysis to be precise: every claim about memory usage had to be backed by a code reference, every optimization had to include an implementation plan. Second, it created artifacts that could be shared with other engineers, reviewed, and iterated upon. Third, it served as a forcing function for completeness—the act of writing revealed gaps in understanding that had to be filled.

The Micro-Optimization Deep Dive

The user's final challenge was to go even deeper: "Look further for more big ideas in compute optimization, improving everything like using more advanced avx/blas/better cpu/gpu cache use/less memory copy/paging in sequantial paths/etc" ([msg 22]).

The assistant responded by launching six parallel deep-dive investigations ([msg 23]), each targeting a different layer of the performance stack:

  1. Synthesis CPU hotpath analysis: Understanding the CPU-level performance of the circuit synthesis phase, including the enforce() loop called ~130 million times per partition, SHA-256 bit manipulation, and Fr field arithmetic
  2. GPU NTT performance analysis: Characterizing the Number Theoretic Transform implementation, kernel launch configurations, memory bandwidth utilization, and occupancy
  3. blst Fr field arithmetic analysis: Understanding BLS12-381 scalar field arithmetic performance, including the assembly-level implementation of multiplication and reduction
  4. Memory access pattern analysis: Characterizing cache behavior during CPU synthesis and the C++ prep_msm phase
  5. Host-to-device transfer pattern analysis: Documenting transfer sizes and patterns for each phase of GPU computation
  6. Recomputation vs. storage analysis: Exploring whether the ~12 GiB a/b/c vectors can be recomputed on-the-fly rather than stored The synthesis of these investigations ([msg 24]) produced a comprehensive micro-optimization analysis that examined CPU synthesis hotpaths at the instruction level, GPU NTT/MSM compute characteristics, H-to-D transfer patterns, and the feasibility of recomputing a/b/c vectors on-the-fly to avoid materialization entirely.

CPU Synthesis Hotpaths

The CPU synthesis phase is dominated by the enforce() function, which is called approximately 130 million times per partition. Each call performs:

GPU NTT and MSM Characteristics

The GPU NTT implementation was found to be memory-bandwidth-bound rather than compute-bound. The kernel achieves approximately 60-70% of peak memory bandwidth on an A100, suggesting that further optimization would require reducing memory traffic rather than increasing compute throughput. The MSM (Multi-Scalar Multiplication) phase, by contrast, is compute-bound, with kernel occupancy limited by register pressure.

Host-to-Device Transfer Patterns

The analysis documented the transfer sizes for each phase of GPU computation. The largest transfers are the a/b/c vectors (~12 GiB per partition) and the SRS data (~48 GiB). The transfers are currently performed synchronously, meaning the GPU is idle during the transfer. Overlapping transfers with computation could reduce the per-partition latency.

Recomputation vs. Storage

The most provocative finding was that the a/b/c vectors could potentially be recomputed on-the-fly rather than stored. Since the circuit structure is known and the witness values are deterministic, the a/b/c vectors for each constraint could be recomputed during the GPU phase, eliminating the need to store them in host memory. This would save ~12 GiB per partition, but at the cost of increased GPU compute time. The analysis concluded that this tradeoff is unlikely to be beneficial for the current architecture, but could become attractive if the memory savings enable a cheaper machine class.

The Overarching Achievement

The journey documented in this segment represents a fundamental shift in how to think about Groth16 proof generation for Filecoin PoRep. It moves from optimizing individual proof generation—tweaking parameters, reducing constant factors, improving cache behavior—toward architecting a continuous, memory-efficient proving pipeline optimized for heterogeneous cloud rental markets where RAM cost dominates.

The key achievements are:

  1. Complete call chain mapping: From Curio's Go task layer through Rust FFI (bellperson) into C++/CUDA kernels (supraseal-c2), with file:line references for every step
  2. Comprehensive memory accounting: Breaking down the ~200 GiB peak into its constituent parts, identifying exactly where each GiB goes
  3. Nine structural bottlenecks: Documented with clear descriptions, proposed solutions, and difficulty ratings
  4. Three composable optimization proposals: Sequential Partition Synthesis (memory reduction), Persistent Prover Daemon (SRS loading elimination), Cross-Sector Batching (throughput improvement)
  5. Micro-optimization analysis: Examining CPU and GPU hotpaths at the instruction level, identifying opportunities for cache optimization, memory transfer reduction, and recomputation strategies The overarching insight is that the current architecture's memory problem is not a performance bug—it is a design choice made for simplicity. Challenging that choice—and proposing a streaming, pipelined alternative—is the kind of architectural thinking that leads to transformative improvements. The proposals do not just optimize the existing design; they imagine a better one.

Lessons for Engineering Practice

Beyond the specific technical findings, this investigation offers several lessons for engineering practice:

Start with understanding, not optimizing. The investigation's first priority was to understand the current code deeply. This prevented premature optimization and ensured that the proposals addressed root causes rather than symptoms.

Trace the data. Following memory allocations from creation to deallocation revealed the true memory picture. Documentation and comments often lie; the code and the data do not.

Quantify everything. Every claim about memory usage was backed by a calculation. Every optimization proposal included a quantitative projection of its impact. This discipline prevented vague recommendations and forced precision.

Think in systems, not components. The proposals are not independent optimizations but a layered stack where each layer enables the next. This systems-level thinking is what separates architectural improvements from micro-optimizations.

Write it down. The transition from analysis to artifact—from understanding to documentation—was crucial. The documents created during this investigation are not just records; they are tools for communication, review, and iteration.

Conclusion

The SUPRASEAL_C2 investigation demonstrates the power of systematic, multi-layered analysis in tackling complex engineering problems. Starting from a single user prompt about a ~200 GiB memory footprint, the investigation traced the complete call chain across four programming languages, identified the root cause in the parallelism model, produced comprehensive documentation, and proposed a fundamental re-architecture of the proving pipeline.

The journey from "understand the current code deeply" to "architect a continuous proving pipeline" represents a shift in perspective that is applicable far beyond the specific context of Filecoin PoRep. It demonstrates that the biggest gains often come not from micro-optimizing individual components but from rethinking the architecture—challenging assumptions about what must be done in parallel, what must be stored in memory, and what can be streamed, pipelined, or recomputed.

For anyone working on Groth16 proof systems, GPU-accelerated cryptography, or memory-constrained high-performance computing, this investigation offers a case study in how to analyze a complex system, identify bottlenecks, and propose targeted optimizations. It demonstrates the value of systematic exploration, careful memory accounting, and engineering judgment in prioritizing improvements—and it shows that the first step to solving a hard problem is often not to solve it, but to map it.

References

[1] From 200 GiB to Pipeline Architecture: The Complete SUPRASEAL_C2 Groth16 Optimization Journey — The chunk article covering the full investigation journey, including the parallel exploration strategy, memory budget analysis, and the three optimization proposals.