The SUPRASEAL_C2 Optimization Pipeline: From 200 GiB Peak Memory to a Continuous Proving Architecture

Introduction

In the world of zero-knowledge proof generation, memory is often the forgotten constraint. Engineers focus on compute throughput—how many multi-scalar multiplications per second a GPU can sustain, how fast the Number Theoretic Transform can run—but the silent killer of proof pipeline economics is peak memory. When a single proof generation requires 200 GiB of host RAM, the hardware rental costs alone can render a protocol economically unviable at scale. This is precisely the challenge facing Filecoin's Proof-of-Replication (PoRep) protocol, where the SUPRASEAL_C2 Groth16 proof generation pipeline routinely consumes approximately 200 GiB of peak host memory for a standard 32 GiB sector with 10 partitions.

This article synthesizes a deep-dive investigation into the SUPRASEAL_C2 pipeline, tracing the full call chain from Curio's Go task orchestration layer through Rust FFI (bellperson) into C++/CUDA kernels (supraseal-c2). The investigation produced four comprehensive documents—a background reference with nine identified structural bottlenecks and three composable optimization proposals—and culminated in a fundamental rethinking of the proving architecture. The work represents a shift from optimizing individual proof generation phases toward architecting a continuous, memory-efficient proving pipeline optimized for heterogeneous cloud rental markets where RAM cost dominates the total cost of ownership.

The Problem: A 200 GiB Memory Wall

The Filecoin PoRep protocol requires storage providers to generate Groth16 proofs that a sector has been correctly sealed. The C2 (Commit 2) proof is the second of two SNARK proofs in the sealing pipeline, and it is by far the most computationally intensive. For a 32 GiB sector, the circuit is divided into 10 partitions, each of which must be synthesized independently before being aggregated into a single proof.

The investigation began with a comprehensive memory accounting exercise that revealed exactly where each gigabyte goes. The peak host memory of ~200 GiB is distributed across three major categories:

A, B, C constraint vectors: Approximately 120 GiB (60% of peak). These are the R1CS constraint matrices that define the circuit's structure. Each partition generates its own set of vectors, and with 10 partitions running in parallel, the memory footprint multiplies accordingly.
Structured Reference String (SRS): Approximately 47 GiB (24% of peak). The SRS is a set of elliptic curve points used as public parameters for the proving system. It must be loaded from disk and deserialized before proof generation can begin, consuming both memory and time (30-90 seconds for deserialization alone).
Auxiliary assignment vectors: Approximately 40 GiB (20% of peak). These vectors carry the witness values—the private inputs to the proof. Remarkably, approximately 99% of these values are boolean (0 or 1), a structural property of the circuit that turns out to be a major optimization opportunity. The GPU, despite being the most expensive component in the system, is active only 30-50% of the total proof time. The rest of the time is spent on CPU-bound phases: circuit synthesis, SRS loading, B_G2 MSM (a specific multi-scalar multiplication on the G2 curve that currently runs on CPU), and proof assembly. This means the expensive GPU sits idle for the majority of each proof cycle—a classic underutilization pattern that signals a pipeline bottlenecked by CPU phases and memory bandwidth rather than GPU compute.

Mapping the Call Chain: From Curio to CUDA

The first major achievement of the investigation was a comprehensive mapping of the entire call chain from Curio's Go task layer down to individual CUDA kernel invocations. This mapping, documented in the background reference, traces the path of a proof request through the software stack:

Curio (Go): The task orchestration layer manages proof generation jobs. When a C2 proof is needed, Curio dispatches a task to a worker process.
Rust FFI (bellperson): The Go worker calls into Rust via a Foreign Function Interface. Bellperson is the Rust implementation of the Groth16 proving system, which handles circuit synthesis and proof generation.
C++/CUDA (supraseal-c2): The Rust layer delegates heavy computation to a C++ library with CUDA kernels. This is where the NTT (Number Theoretic Transform), MSM (Multi-Scalar Multiplication), and other GPU-accelerated operations live. The mapping revealed that the 10 partitions are synthesized in parallel at the Rust level, each producing its own set of constraint vectors. These are then aggregated and passed to the GPU for the proving computation. The parallel synthesis is the primary driver of the 200 GiB peak memory: 10 partitions × ~16 GiB each for the a/b/c vectors, plus the shared SRS and auxiliary data. The background document identified nine structural bottlenecks in this pipeline, ranging from the obvious (parallel partition synthesis) to the subtle (SHA-256 bit manipulation in the enforce() loop dominating CPU synthesis time). Each bottleneck was documented with file:line references, enabling precise targeting of optimization efforts.

Proposal 1: Sequential Partition Synthesis

The first optimization proposal tackles the most obvious problem: the 10 partitions are synthesized in parallel, each consuming ~16 GiB for its a/b/c vectors. If partitions could be synthesized sequentially, the peak memory would drop dramatically—from ~200 GiB to approximately 64-103 GiB, depending on whether the SRS is kept resident or reloaded per partition.

The key insight is that the partitions are independent at the synthesis stage. There is no mathematical reason they must be processed in parallel; the parallelization was an implementation choice driven by the desire to minimize total synthesis time. However, this choice comes at an enormous memory cost. By streaming partitions one-at-a-time through the GPU, the Sequential Partition Synthesis proposal achieves:

Peak memory reduction: From ~200 GiB to ~64-103 GiB (a 48-68% reduction).
GPU utilization improvement: From 30-50% to 70-90%, because the GPU can be kept busy with a steady stream of work rather than waiting for all partitions to finish synthesis.
Minimum machine requirements: The proposal enables C2 proving on machines with 64 GiB of RAM (with SRS reloading) or 128 GiB (with SRS resident), compared to the current requirement of 256 GiB. The tradeoff is a modest increase in total proof time due to sequential processing, but this is offset by the GPU utilization gains and the dramatic reduction in hardware cost. The proposal estimates a net throughput improvement of approximately 1.6× when measured in proofs per hour per dollar.

Proposal 2: Persistent Prover Daemon

The second proposal addresses a different bottleneck: the Structured Reference String (SRS) loading overhead. Currently, each proof generation begins by loading and deserializing the ~47 GiB SRS from disk, a process that takes 30-90 seconds. This is pure overhead—the SRS is identical for every proof, yet it is loaded from scratch each time.

The Persistent Prover Daemon proposal eliminates this overhead by keeping the proving process alive across multiple proof invocations. Instead of spawning a new process for each proof (the current model), a daemon process maintains the SRS in pinned host memory, ready for immediate use. When a new proof request arrives, the daemon can begin computation immediately, skipping the SRS loading phase entirely.

The impact is substantial:

Per-proof time reduction: Eliminating 30-90 seconds of SRS loading reduces total proof time from ~360s to ~270-330s (a 17-25% improvement).
Throughput improvement: From ~12 proofs/hour to ~15 proofs/hour on the same hardware.
Memory efficiency: The SRS remains in pinned memory across proofs, eliminating the allocation and deallocation overhead that currently consumes a significant fraction of each proof cycle. The daemon architecture also enables a cleaner separation of concerns: the proving daemon can be optimized independently of the task orchestration layer, and it can be deployed on dedicated hardware optimized for memory bandwidth and GPU connectivity.

Proposal 3: Cross-Sector Batching

The third proposal exploits the memory headroom freed by Proposals 1 and 2 to achieve a step-change improvement in throughput. The idea is simple: once peak memory has been reduced from ~200 GiB to ~64-103 GiB, the freed memory can be used to batch multiple sectors' circuits into a single GPU invocation.

The Cross-Sector Batching proposal projects:

Throughput multiplier: 2-3× more proofs per GPU compared to the baseline.
Cost reduction: 5-6× reduction in cost per proof when combined with Proposals 1 and 2, dropping from approximately $0.083/proof to $0.013-0.016/proof.
Memory scaling: Batching has almost no impact on peak memory because the partition circuits are processed sequentially within each batch. The key insight is that GPU kernels benefit from larger batch sizes. The NTT and MSM operations that dominate the proving computation are highly parallelizable, and batching multiple sectors' work into a single kernel launch improves GPU utilization and amortizes kernel launch overhead. The proposal identifies the B_G2 MSM (currently running on CPU) as a potential bottleneck at scale: with 30 circuits batched, the B_G2 time grows from ~50s to ~150s, which would need to be addressed by either GPU-accelerating the B_G2 MSM or parallelizing it across multiple CPU cores.

Micro-Optimization Analysis

Beyond the three architectural proposals, the investigation included a detailed micro-optimization analysis of the CPU and GPU hotpaths. This analysis examined:

CPU synthesis hotpaths: The enforce() loop, which constructs the R1CS constraints, was found to be dominated by SHA-256 bit manipulation operations. Approximately 88% of the constraints in the PoRep circuit are SHA-256 related, and the enforce() loop performs extensive bit-level operations to wire the hash function into the constraint system. The analysis identified opportunities for specialized constraint evaluation that could eliminate the 780 million heap allocations per partition that currently occur during synthesis.

GPU NTT/MSM characteristics: The GPU kernels for NTT and MSM were characterized in terms of memory bandwidth utilization, kernel occupancy, and compute-to-memory ratio. The analysis found that the NTT kernels are memory-bandwidth-bound, while the MSM kernels are compute-bound at small batch sizes but become memory-bound as batch size increases. This suggests different optimization strategies for each kernel type.

H-to-D transfer patterns: The host-to-device transfer of constraint vectors was identified as a significant overhead, particularly for the a/b/c vectors which total approximately 120 GiB. The analysis evaluated the feasibility of recomputing the a/b/c vectors on-the-fly on the GPU rather than transferring them from host memory, which could eliminate the transfer entirely at the cost of additional GPU compute.

Recomputing a/b/c vectors: The most radical micro-optimization considered was to avoid materializing the a/b/c vectors altogether. Since the constraint matrices are deterministic and identical for every proof (only the witness values change), the vectors could be recomputed on the GPU from the circuit description and the witness values, eliminating the ~120 GiB of host memory and the associated transfer time. This approach, while technically feasible, would require significant engineering effort to implement the constraint evaluation logic in CUDA kernels.

The Shift in Architecture Philosophy

The most significant outcome of this investigation is not any single optimization proposal but the fundamental rethinking of the proving architecture that underlies all of them. The current pipeline treats each proof as an isolated, batch-processed job: load the SRS, synthesize all partitions in parallel, transfer to GPU, compute the proof, tear everything down, and repeat. This model is simple but wasteful.

The proposed architecture shifts toward a continuous, memory-efficient proving pipeline:

Continuous: The proving process stays alive across proofs, maintaining the SRS and other reusable state in memory. The daemon model (Proposal 2) is the key enabler.
Memory-efficient: Partitions are processed sequentially rather than in parallel, and memory is allocated and freed in a streaming fashion. The Sequential Partition Synthesis (Proposal 1) is the key enabler.
Batch-oriented: Multiple sectors' proofs are batched into single GPU invocations to maximize utilization. Cross-Sector Batching (Proposal 3) is the key enabler. This architectural shift is optimized for the economic reality of cloud rental markets, where RAM cost dominates. A machine with 256 GiB of RAM costs approximately $600/month on cloud rental markets, while a machine with 128 GiB costs approximately $350/month—a 42% reduction. By enabling C2 proving on 128 GiB machines (or even 64 GiB machines with SRS reloading), the proposals unlock access to cheaper hardware tiers that were previously inaccessible.

The Role of the Subagent Extraction

The quantitative foundation for these proposals was established through a structured extraction of timing and memory estimates from the five source documents. This extraction, performed by a subagent as documented in the companion articles [1][2][3], consolidated scattered estimates into a unified framework that enabled cross-proposal comparison and dependency analysis. The subagent's work produced a comprehensive structured summary that included per-phase timing breakdowns, memory accounting tables, throughput projections, and implementation effort estimates—all of which fed into the proposal design and the total impact assessment.

The extraction revealed important nuances that shaped the proposals. For example, the B_G2 CPU bottleneck only becomes visible when the proposals are combined: at the baseline, B_G2 takes ~50s (10 partitions × 5s each), which is manageable. But with Cross-Sector Batching processing 30 circuits, the B_G2 time grows to ~150s, making it the dominant phase. This finding, which emerged from the synthesis rather than from any single document, directly influenced the recommendation to GPU-accelerate the B_G2 MSM as part of the batching proposal.

Conclusion

The SUPRASEAL_C2 optimization investigation represents a comprehensive rethinking of the Groth16 proof generation pipeline for Filecoin PoRep. By mapping the full call chain from Curio to CUDA kernels, accounting for every gigabyte of peak memory, and identifying nine structural bottlenecks, the investigation established a clear understanding of where the pipeline's inefficiencies lie. The three composable optimization proposals—Sequential Partition Synthesis, Persistent Prover Daemon, and Cross-Sector Batching—together project a path from the current baseline of ~360s per proof on 256 GiB machines at ~$0.083/proof to an optimized pipeline delivering ~35-45s per proof on 96 GiB machines at ~$0.004/proof: roughly a 10× throughput improvement and a 20× cost reduction.

The overarching achievement is a shift from optimizing individual proof generation phases toward architecting a continuous, memory-efficient proving pipeline. This architectural shift recognizes that in heterogeneous cloud rental markets, the dominant cost is not GPU compute but host memory, and the most impactful optimizations are those that reduce peak memory and enable continuous operation across proofs. The investigation's findings have direct implications for the economics of Filecoin storage proving, potentially enabling a new generation of cost-effective proving infrastructure that can operate on affordable hardware without sacrificing throughput.## The Subagent Extraction: From Scattered Documents to Unified Framework

The quantitative foundation for the optimization proposals was established through a structured extraction task performed by a subagent. The user's request ([msg 0]) was precise: read five specific files—the background document and Proposals 1 through 4—and extract every timing estimate, memory estimate, speedup percentage, and implementation effort estimate, returning them in a structured format. The assistant's response ([msg 1]) initiated parallel reads of all five files, gathering the raw data before beginning any synthesis work.

The resulting structured extraction ([msg 2]) is a masterwork of analytical synthesis. It transforms scattered, heterogeneous estimates from five independently written documents into a unified framework that enables cross-proposal comparison, dependency analysis, and economic translation. The extraction includes:

A canonical baseline: Per-phase timing breakdowns (SRS loading at 30-90s, circuit synthesis at 60-180s, NTT+H MSM at 30-60s, etc.) and a detailed memory accounting showing exactly where each of the ~200 GiB goes.
Per-proposal comparison tables: Each proposal is presented with current vs. proposed metrics, effort estimates, and dependency information, enabling direct comparison.
Economic translation: The $/proof comparison table translates technical metrics into business value, showing a reduction from $0.083/proof to $0.013-0.016/proof when all proposals are combined.
Implementation roadmap: A phased plan with dependency ordering, showing that Proposals 1 and 2 are independent and can proceed in parallel, Proposal 3 depends on both, and Proposal 4 is fully orthogonal.
Emergent insights: The B_G2 CPU bottleneck only becomes visible when the proposals are combined—at the baseline, B_G2 takes ~50s, but with Cross-Sector Batching processing 30 circuits, it grows to ~150s, making it the dominant phase. This finding emerged from the synthesis rather than from any single document. The extraction also revealed important nuances that shaped the proposal design. For example, the memory scaling table for Proposal 3 shows that batch size has almost no impact on peak memory (from 64 GiB for batch=1 to 66 GiB for batch=5), because the partition circuits are processed sequentially within each batch. This finding was critical for validating the feasibility of batching: if peak memory scaled linearly with batch size, the proposal would be impractical.

The Memory Accounting: Where Every GiB Goes

The background document's memory accounting is a foundational contribution of this investigation. It provides a granular breakdown of the ~200 GiB peak host memory, tracing each component to its source in the code:

A, B, C vectors (~120 GiB, 60%): These are the R1CS constraint matrices, stored as sparse vectors of field elements. Each of the 10 partitions produces its own set of vectors, and they are held in memory simultaneously because the synthesis is parallelized. The vectors are allocated on the Rust heap using Vec<Fr>, where Fr is a 256-bit field element. With approximately 2^23 constraints per partition and 3 vectors per constraint, the memory adds up quickly.
SRS (~47 GiB, 24%): The Structured Reference String is a set of elliptic curve points used as public parameters for the Groth16 proving system. It is stored in pinned host memory (allocated via cudaHostAlloc) to enable direct GPU access via DMA. The SRS is loaded from disk and deserialized for each proof, a process that takes 30-90 seconds and consumes significant I/O bandwidth.
Auxiliary assignment vectors (~40 GiB, 20%): These vectors carry the witness values—the private inputs to the proof. Remarkably, approximately 99% of these values are boolean (0 or 1), a structural property of the circuit that arises from the dominance of SHA-256 constraints. SHA-256 operates on bits, and the circuit encodes each bit as a boolean constraint. This property is a major optimization opportunity: if 99% of the values are 0 or 1, the vectors can be compressed from full field elements to bit flags, reducing memory by approximately 98%.
DensityTrackers (~0.5 GiB): Metadata structures used during synthesis to track which constraints have been assigned.
Split vectors and tail MSM (~12-27 GiB): Intermediate data structures used during the MSM computation on the GPU. The GPU peak memory is much smaller (~12-16 GiB) because the GPU processes data in streaming fashion, transferring only the data needed for the current computation. The GPU memory holds the SRS points (in device memory), the current batch of a/b/c vectors, and the MSM accumulators. This accounting reveals a striking asymmetry: the host memory bottleneck is driven by data structures that are ephemeral (the a/b/c vectors are only needed during synthesis and the initial GPU transfer) but are held for the entire proof duration because of the parallel partition architecture. The Sequential Partition Synthesis proposal directly addresses this by streaming partitions through the pipeline, allocating and freeing memory in a tight window.

The Architectural Shift: From Batch Processing to Continuous Pipeline

The most profound contribution of this investigation is not any single optimization but the fundamental rethinking of the proving architecture. The current pipeline follows a batch-processing model:

Load: Load the SRS from disk (~47 GiB, 30-90s)
Synthesize: Synthesize all 10 partitions in parallel (~60-180s, ~200 GiB peak)
Transfer: Transfer a/b/c vectors to GPU (~120 GiB, ~10-30s)
Compute: Run NTT, MSM, and other GPU kernels (~85s GPU active)
Assemble: Run B_G2 MSM on CPU and assemble the proof (~50s)
Tear down: Free all memory, destroy CUDA context, exit process This model is simple and robust, but it is grossly inefficient. The GPU is active only 30-50% of the time. The SRS is loaded and discarded for every proof. The a/b/c vectors are held in memory for the entire proof duration even though they are only needed for a fraction of that time. The CUDA context is created and destroyed for each proof. The proposed architecture inverts this model:
Initialize once: Start a persistent daemon that loads the SRS once and maintains a CUDA context.
Stream partitions: Process partitions sequentially, allocating a/b/c vectors for one partition at a time, transferring them to the GPU, and freeing them before the next partition begins.
Batch across sectors: Collect circuits from multiple sectors and batch them into a single GPU invocation, amortizing kernel launch overhead and improving SM utilization.
Continuous operation: The daemon remains alive across proofs, accepting new work requests via IPC and returning proofs without process startup or SRS loading overhead. This shift from batch processing to continuous pipeline operation has profound implications for hardware economics. A machine that can sustain a continuous proving load can achieve much higher utilization than a machine that spends half its time loading and tearing down. The proposals project an improvement from ~10 proofs/hour to ~25-30 proofs/hour on the same GPU—a 2.5-3.0× throughput improvement—while simultaneously reducing the memory requirement from 256 GiB to 96-128 GiB.

The Economic Dimension: Cost Per Proof

The investigation consistently emphasizes the economic dimension of the optimization work. The $/proof metric is the ultimate measure of success, and the proposals are evaluated not just on technical merit (faster, smaller) but on economic impact (cheaper, more accessible).

The baseline economics are stark: at $0.083/proof on a $600/month 256 GiB machine, C2 proving is a significant operational cost for storage providers. The combined proposals project a reduction to $0.013-0.016/proof on a $350/month 128 GiB machine—a 5-6× reduction. This is achieved through three mechanisms:

Hardware cost reduction: Moving from 256 GiB to 128 GiB machines reduces the monthly rental cost by 42% ($600 → $350).
Throughput improvement: Increasing proofs/hour from 10 to 25-30 triples the proofs per machine per month.
Amortization of fixed costs: The persistent daemon eliminates per-proof overheads (SRS loading, CUDA context creation) that consume a significant fraction of each proof cycle. The economic analysis also considers the marginal return of each optimization. The Pre-Compiled Constraint Evaluator (PCE) from Proposal 5 is identified as the highest-impact single item, with a 1.00× throughput multiplier per engineering-week. The quick wins from Proposal 4 (Wave 1) deliver the majority of the compute-level gains in just 1-2 weeks of effort. This marginal return analysis enables informed prioritization: implement the highest-impact items first, deliver value early, and continue with diminishing-return optimizations only if the economics justify the engineering investment.

Conclusion

The SUPRASEAL_C2 optimization investigation represents a comprehensive rethinking of the Groth16 proof generation pipeline for Filecoin PoRep. By mapping the full call chain from Curio to CUDA kernels, accounting for every gigabyte of peak memory, and identifying nine structural bottlenecks, the investigation established a clear understanding of where the pipeline's inefficiencies lie. The three composable optimization proposals—Sequential Partition Synthesis, Persistent Prover Daemon, and Cross-Sector Batching—together with the compute-level optimizations and the constraint-shape-aware optimizations, project a path from the current baseline of ~360s per proof on 256 GiB machines at ~$0.083/proof to an optimized pipeline delivering ~35-45s per proof on 96 GiB machines at ~$0.004/proof: roughly a 10× throughput improvement and a 20× cost reduction.

[1] "The Architecture of Delegation: A Close Reading of a Subagent Task Assignment in the SUPRASEAL_C2 Optimization Pipeline" — Examines the subagent task assignment message ([msg 0]) that initiated the structured extraction, analyzing the delegation architecture, assumptions, and reasoning embedded in the request.

[2] "The Data Gathering Prelude: A Subagent Reads Five Files in Parallel" — Analyzes the parallel file read operation ([msg 1]) that gathered the raw data from five optimization proposal documents, examining the methodological pattern of "gather first, synthesize second."

[3] "The Architecture of Optimization: Deconstructing a Structured Extraction of Groth16 Proof Pipeline Estimates" — Deconstructs the structured extraction response ([msg 2]) that consolidated timing, memory, speedup, and effort estimates into a unified analytical framework, examining the decisions, assumptions, and knowledge creation involved.