Architecting a Memory-Efficient Proving Pipeline: Deep-Dive into SUPRASEAL_C2 Groth16 Optimization

Introduction

In the world of decentralized storage, Filecoin's Proof-of-Replication (PoRep) mechanism stands as a critical component, ensuring that storage providers are honestly storing the data they claim to hold. But beneath this high-level protocol lies an extraordinary computational challenge: generating Groth16 proofs for the SUPRASEAL_C2 pipeline consumes approximately 200 GiB of peak memory per proof. This staggering memory footprint translates directly into cloud rental costs, making each proof an expensive proposition for storage providers operating in heterogeneous cloud markets where RAM is a primary cost driver.

This article synthesizes a deep-dive investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline, tracing the full call chain from Curio's Go task layer through Rust FFI (bellperson) into C++ and CUDA kernels (supraseal-c2). The investigation produced four major documents: a comprehensive background reference mapping the entire pipeline architecture with file:line references and nine identified structural bottlenecks, plus three composable optimization proposals. A final round of micro-optimization analysis examined CPU synthesis hotpaths, GPU NTT/MSM characteristics, and the feasibility of recomputing a/b/c vectors on-the-fly. The earlier articles in this series documented the benchmarking infrastructure investigation [1], the assistant's initial reconnaissance [2], the comprehensive analysis of the cuzk-bench tool's e2e capabilities [3], and the handling of tool output truncation during code exploration [4].

The overarching achievement of this work is a paradigm shift: from optimizing individual proof generation toward architecting a continuous, memory-efficient proving pipeline optimized for the realities of cloud rental markets where memory cost dominates compute cost.

The Problem: Where Does 200 GiB Go?

The investigation began with a fundamental question: where exactly does ~200 GiB of memory go during SUPRASEAL_C2 proof generation? The answer required tracing the entire call chain from Curio's task orchestration layer down to individual CUDA kernel invocations.

The Architecture at a Glance

The SUPRASEAL_C2 pipeline involves four major layers:

  1. Curio (Go) — Task orchestration layer that manages proving jobs, schedules them across available workers, and handles the lifecycle of each proof request.
  2. bellperson (Rust) — The Rust FFI layer that bridges Curio's Go world to the C++/CUDA proving engine. This is where the circuit is constructed and the proof synthesis begins.
  3. supraseal-c2 (C++) — The core proving engine that implements the Groth16 prover, including the heavy computational kernels for Number Theoretic Transforms (NTT) and Multi-Scalar Multiplication (MSM).
  4. CUDA kernels — GPU-accelerated implementations of NTT and MSM that run on NVIDIA GPUs, handling the bulk of the floating-point and integer arithmetic.

Memory Accounting: The 200 GiB Breakdown

The investigation's memory accounting revealed a clear picture. The peak memory footprint is driven by two primary factors:

Phase 1: Mapping the Call Chain and Identifying Bottlenecks

The first phase of the investigation produced a comprehensive background reference document that mapped the entire call chain with precise file:line references. This document served as the foundation for all subsequent analysis.

The Call Chain from Curio to CUDA

The investigation traced the following path:

  1. Curio task layer (curio/tasks/seal.go) — A Go routine monitors the proving queue, picks up a sector to prove, and calls into the Rust FFI.
  2. bellperson FFI boundary (extern/supraseal/c2/supraseal-c2/src/ffi.rs) — The Go-to-Rust transition point. Parameters are marshaled across the FFI boundary, and the Rust side begins constructing the circuit.
  3. Circuit construction (extern/supraseal/c2/supraseal-c2/src/circuit/) — The circuit is built from the public inputs and the witness data. This is where the 10 partitions are created.
  4. Synthesis (extern/supraseal/c2/supraseal-c2/src/synthesis/) — The proof synthesis phase, which runs on the CPU and generates the a/b/c vectors that feed into the GPU proving phase.
  5. GPU proving (extern/supraseal/c2/supraseal-c2/src/gpu/) — The synthesized proof data is transferred to the GPU, where NTT and MSM kernels execute the heavy linear algebra.
  6. CUDA kernels (extern/supraseal/c2/supraseal-c2/cuda/) — The lowest layer, where NTT butterfly operations and MSM point additions run on thousands of GPU cores.

The Nine Structural Bottlenecks

The investigation identified nine structural bottlenecks in the pipeline, each representing an opportunity for optimization:

  1. All-10-partitions-in-parallel memory blowup: The current architecture synthesizes all 10 partitions simultaneously, maximizing peak memory.
  2. SRS loading per proof: The 48 GiB SRS is loaded from disk for every proof, taking ~60 seconds.
  3. CPU synthesis time exceeds GPU time: Synthesis takes ~38 seconds while GPU proving takes ~26 seconds, leaving the GPU idle for ~12 seconds per proof.
  4. No inter-proof overlap in partitioned mode: Unlike the standard pipeline, the partitioned path blocks the synthesis task for the entire proof duration.
  5. SHA-256 bit manipulation in circuit synthesis: The enforce() loop performs extensive bit manipulation for SHA-256, which is CPU-intensive.
  6. Fr field arithmetic overhead: Field element addition and multiplication in the BLS12-381 scalar field (Fr) adds overhead during synthesis.
  7. NTT memory bandwidth limitations: GPU NTT kernels are memory-bandwidth-bound, limiting throughput.
  8. MSM point arithmetic cost: Multi-scalar multiplication on the GPU involves significant point addition and doubling.
  9. H-to-D transfer latency: Transferring synthesized data from host (CPU) to device (GPU) over PCIe adds latency.

Phase 2: Three Composable Optimization Proposals

With the pipeline mapped and bottlenecks identified, the investigation produced three optimization proposals. Each proposal targets a different dimension of the problem, and they are designed to be composable — they can be deployed together for multiplicative benefit.

Proposal 1: Sequential Partition Synthesis

The core insight behind Sequential Partition Synthesis is simple: do not synthesize all 10 partitions at once. Instead, stream partitions one-at-a-time through the GPU, synthesizing each partition, transferring it to the GPU, proving it, and then freeing the CPU-side memory before starting the next partition.

Memory impact: Reduces peak memory from ~200 GiB to approximately 64–103 GiB, depending on the implementation details. The 10× parallel partition memory (~160 GiB) collapses to 1× partition memory (~16 GiB) plus the SRS (~48 GiB).

Trade-off: Sequential processing increases total proof time because partitions cannot overlap. However, the GPU is already the bottleneck in the current pipeline, and the sequential approach ensures the GPU is fed continuously rather than sitting idle while all partitions are synthesized upfront.

Implementation approach: Modify the synthesis loop in supraseal-c2 to process partitions one at a time, using a producer-consumer pattern where the synthesis thread produces one partition while the GPU thread consumes the previous one. This requires careful synchronization but avoids the all-at-once memory allocation.

Proposal 2: Persistent Prover Daemon

The Persistent Prover Daemon proposal addresses the SRS loading overhead. Currently, the proving process is ephemeral: it starts, loads the 48 GiB SRS (~60 seconds), generates one proof, and exits. The next proof pays the same 60-second tax.

Key insight: Keep the proving process alive across multiple proofs. A daemon process loads the SRS once into pinned GPU memory and keeps it warm. Subsequent proofs skip the SRS loading step entirely.

Memory impact: The SRS remains in pinned memory (~48 GiB) across proofs, but this memory is already allocated per-proof in the current architecture. The difference is that it is allocated once and reused, rather than allocated, freed, and re-allocated for each proof.

Throughput impact: Eliminates ~60 seconds of overhead per proof. For a single proof, this is a 60-second saving. For batch proofs, the saving compounds.

Implementation approach: The existing cuzk daemon architecture already supports this pattern — the PreloadSRS RPC can pre-warm SRS parameters. The proposal extends this by keeping the daemon process alive and reusing the GPU context across multiple Prove RPC calls, avoiding the process-level teardown and restart.

Proposal 3: Cross-Sector Batching

Cross-Sector Batching exploits the memory freed by Sequential Partition Synthesis to batch multiple sectors' circuits into a single GPU invocation.

Key insight: The GPU has fixed computational capacity. If Sequential Partition Synthesis reduces per-proof memory from ~200 GiB to ~64–103 GiB, the freed memory headroom can be used to load circuits for multiple sectors simultaneously. The GPU can then prove multiple sectors in a single kernel launch, amortizing kernel launch overhead and improving GPU utilization.

Throughput impact: Projected 2–3× throughput per GPU when used alone, and 5–6× reduction in $/proof when combined with the Persistent Prover Daemon.

Implementation approach: Modify the daemon's scheduling logic to batch incoming proof requests. Instead of processing one sector at a time, the daemon accumulates requests and launches a combined GPU kernel that proves multiple sectors' circuits in a single pass. This requires changes to the circuit construction and GPU kernel dispatch code, but the underlying NTT and MSM kernels already support batched operation.

Composability Analysis

The three proposals are designed to work together:

Phase 3: Micro-Optimization Analysis

The final phase of the investigation shifted from architectural proposals to micro-optimization analysis, examining the computational hotpaths at the instruction level.

CPU Synthesis Hotpaths

The CPU synthesis phase was analyzed in detail, revealing three primary hotpaths:

  1. The enforce() loop: This loop is the heart of circuit synthesis, where each constraint is evaluated and the a/b/c vectors are populated. The loop iterates over thousands of constraints, each requiring field element operations and SHA-256 bit manipulation.
  2. SHA-256 bit manipulation: The SUPRASEAL_C2 circuit includes SHA-256 hash operations, which require extensive bit-level manipulation in the circuit's constraint system. Each SHA-256 operation involves XOR, rotation, and selection operations on 32-bit words, translated into field element constraints.
  3. Fr addition overhead: Field element addition in the BLS12-381 scalar field (Fr) is a frequent operation during synthesis. While individual additions are fast, the sheer volume creates measurable overhead. The analysis confirmed that CPU synthesis time (~38 seconds) is the primary bottleneck in the current pipeline, exceeding GPU compute time (~26 seconds) by approximately 12 seconds. This gap means the GPU sits idle for roughly 30% of the proof time, waiting for the CPU to finish synthesizing the next proof's data.

GPU NTT/MSM Characteristics

On the GPU side, the analysis examined the two primary computational kernels:

H-to-D Transfer Patterns

The investigation also analyzed the host-to-device (H-to-D) transfer patterns. The synthesized a/b/c vectors must be transferred from CPU memory to GPU memory over the PCIe bus. This transfer is a bottleneck in its own right, consuming measurable time and competing with other PCIe traffic. The analysis considered:

Recomputing a/b/c Vectors On-the-Fly

One of the more radical ideas examined was whether the a/b/c vectors could be recomputed on the fly rather than materialized in memory. The a/b/c vectors are the three vectors that define each constraint in the QAP (Quadratic Arithmetic Program). In the current architecture, they are fully materialized during synthesis and then transferred to the GPU.

The feasibility analysis considered:

The Broader Shift: From Individual Proofs to Continuous Pipeline

Perhaps the most significant outcome of this investigation is not any single optimization proposal, but the paradigm shift it represents. The current architecture treats each proof as an isolated event: start a process, load SRS, synthesize, prove, exit. This ephemeral model is simple but wasteful.

The optimization proposals collectively point toward a continuous proving pipeline:

Conclusion

The deep-dive investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline has produced a comprehensive understanding of its ~200 GiB peak memory footprint, mapping the full call chain from Curio's Go task layer through Rust FFI into C++ and CUDA kernels. The nine identified structural bottlenecks and three composable optimization proposals provide a clear roadmap for reducing memory, eliminating overhead, and improving throughput.

The work represents a shift from optimizing individual proof generation toward architecting a continuous, memory-efficient proving pipeline. In the context of heterogeneous cloud rental markets where RAM cost dominates, this architectural approach offers the most promising path to making Filecoin proof generation economically viable for storage providers.

The investigation also demonstrated the value of systematic, multi-layered analysis: starting with memory accounting, moving to architectural proposals, and then drilling into micro-optimization at the instruction level. Each layer informed the others, producing a coherent optimization strategy rather than a collection of isolated tweaks. The result is a blueprint for transforming the economics of Filecoin proof generation — one partition at a time.## References

[1] "Probing the Proving Pipeline: A Deep Dive into the cuzk-bench e2e Benchmarking Inquiry" — Analyzes the opening message of a subagent session tasked with exploring the end-to-end benchmarking capabilities of cuzk-bench, revealing the strategic context, assumptions, and thinking process behind the investigation.

[2] "The First Step: How an AI Assistant Began Mapping Filecoin's Benchmarking Infrastructure" — Examines the assistant's opening move in the subagent session, a parallel reconnaissance strike that issued five simultaneous tool calls to read source files and discover proto definitions.

[3] "Mapping the Benchmarking Surface: How a Single Assistant Message Defined the e2e Testing Capabilities of cuzk-bench" — Documents the comprehensive analysis of cuzk-bench's subcommands, concurrency controls, output formats, and throughput measurement capabilities, synthesizing information from multiple source files.

[4] "Repairing the Fragment: How an AI Assistant Handles Tool Output Truncation in a Code Exploration Session" — Explores the assistant's meta-cognitive response to tool output truncation, demonstrating self-monitoring, corrective planning, and transparency in the face of information loss.